write data from kafka topic to eventhubs

Find a file

Ed Sweeney 1fdf2ad40b Update README.md		2017-02-11 18:30:44 -08:00
project	init	2017-01-07 21:54:01 -08:00
src	lint	2017-02-11 13:31:35 -08:00
.gitignore	init	2017-01-07 20:25:25 -08:00
build.sbt	init	2017-01-08 11:19:32 -08:00
LICENSE	Initial commit	2017-01-07 19:58:03 -08:00
README.md	Update README.md	2017-02-11 18:30:44 -08:00
scalastyle-config.xml	init	2017-01-07 21:54:01 -08:00

UNDER CONSTRUCTION - NOT WORKING QUITE YET

temporarily inlining the src from the azure event hubs master repo (waiting on a jar for v2.0.3)

using Spark v2.0.2 to write data from Kafka v0.10.x to an Azure Event Hub

    EH_POLICY_NAME=<CHANGE ME>
    EH_POLICY_KEY=<CHANGE ME>
    EH_NAMESPACE=<CHANGE ME>
    EH_HUB_NAME=<CHANGE ME>
    KAFKA_TOPIC=<CHANGE ME>

the defaults work for some during dev but change these too

    KAFKA_BATCH_DUR="2"
    KAFKA_BROKER_LIST="localhost:9092"
    KAFKA_CONSUMER_GROUP=<CHANGE ME>

sbt assembly

sbt assembly && docker build -t myimage .

TODO