Using multiple Avro schema in one Kafka Topic

I tried Kafka with Avro on Spring Boot in the previous blog post(Japanese).

bufferings.hatenablog.com

Then this time, I tried to use multiple Avro schema in one topic.

The source code

https://github.com/bufferings/kafka-streams-demo/tree/v0.3

Schema

"tweet.avsc" is the schema file which I used in the previous post.

{
  "name": "Tweet",
  "type": "record",
  "fields": [
    {"name": "id",  "type": "long"},
    {"name": "text", "type": "string"}
  ]
}

Then, I added another schema file "different-schema.avsc"

{
  "name": "DifferentSchema",
  "type": "record",
  "fields": [
    {"name": "id2",  "type": "long"},
    {"name": "createdAt", "type": "string"},
    {"name": "text2", "type": "string"}
  ]
}

(Please note that I don't think about Schema Evolution yet, because it's still early for me to think about it.)

Modify Serializer

I found that KafkaAvroSerializer uses the topic name for the schema registration key. Because I want to try multiple schema in one topic, I created KafkaAvroSerializerWithSchemaName to use the schema fullname for registering schema.

https://github.com/bufferings/kafka-streams-demo/blob/v0.3/src/main/java/com/example/demo/KafkaAvroSerializerWithSchemaName.java#L36

  @Override
  public byte[] serialize(String topic, Object record) {
    return serializeImpl(getSchema(record).getFullName(), record);
  }

In addition, I changed Serde for KStream to use KafkaAvroSerializerWithSchemaName for serialization.

https://github.com/bufferings/kafka-streams-demo/blob/v0.3/src/main/java/com/example/demo/GenericAvroSerializerWithSchemaName.java

Finally, I have multiple schemas in one topic :)

Addition

This time I tried "schema-registry-ui" and "kafka-topics-ui" which are pretty nice web UI provided by Landoop( http://www.landoop.com/kafka/kafka-tools/ ) under BSL license( http://www.landoop.com/bsl/ ).

Schema Registry UI

To enable CORS accessing from other docker instance, I added "access.control.allow.methods" and "access.control.allow.origin" to the schema registry.

docker run -d --net=host --name=schema-registry \
    -e SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=localhost:2181 \
    -e SCHEMA_REGISTRY_HOST_NAME=localhost \
    -e SCHEMA_REGISTRY_DEBUG=true \
    -e SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_METHODS=GET,POST,PUT,OPTIONS \
    -e SCHEMA_REGISTRY_ACCESS_CONTROL_ALLOW_ORIGIN=* \
    confluentinc/cp-schema-registry:3.2.1

Then run schema-regisry-ui.

docker run -d --rm -p 7000:8000 \
    -e "SCHEMAREGISTRY_URL=http://172.17.0.1:8081" \
    landoop/schema-registry-ui

Now I can see schemas like this:

Kafka Topic UI

To use Kafka Topic UI, I need to run Kafka Rest Proxy.

docker run -d \
  --net=host \
  --name=kafka-rest \
  -e KAFKA_REST_ZOOKEEPER_CONNECT=localhost:2181 \
  -e KAFKA_REST_SCHEMA_REGISTRY_URL=http://localhost:8081 \
  -e KAFKA_REST_HOST_NAME=localhost \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_ORIGIN=* \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_METHODS=GET,POST,PUT,DELETE,OPTIONS \
  confluentinc/cp-kafka-rest:3.2.1

Then run kafka-topics-ui.

docker run --rm -it -p 7001:8000 \
  -e "KAFKA_REST_PROXY_URL=http://172.17.0.1:8082" \
  landoop/kafka-topics-ui

It's fun!

Reference

http://docs.confluent.io/current/cp-docker-images/docs/quickstart.html