Partitioning

Topic Replica and Partition Count Manipulation

KSQL provides a powerful way for you to change the properties of Kafka topics by defining new streams with the desired properties of the new topic, populated by the streaming events of the original topic.

Because these are streaming transformations, the existing Kafka topic gets streamed to the new topic with the desired properties, and the target topic is kept in sync with every single message that subsequently arrives on the source topic.

Directions

Follow these steps to change a topic’s replication factor and number of partitions.

1. Inspect the source topic’s properties:

[source,bash]
----
$ kafka-topics --zookeeper localhost:2181 --describe --topic mysql_users_avro
Topic:mysql_users_avro  PartitionCount:4        ReplicationFactor:1     Configs:
        Topic: mysql_users_avro Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: mysql_users_avro Partition: 3    Leader: 0       Replicas: 0     Isr: 0
----

2. Define the source topic stream:

[source,sql]
----
CREATE STREAM SOURCE_TOPIC WITH (KAFKA_TOPIC='mysql_users_avro', VALUE_FORMAT='AVRO');
----

3. Create a derived topic with new properties:

[source,sql]
----
CREATE STREAM TARGET_TOPIC WITH (KAFKA_TOPIC='mysql_users_avro_v2', \
                                 PARTITIONS=1,                      \
                                 REPLICAS=3) AS                     \
SELECT * FROM SOURCE_TOPIC;
----

4. Inspect the resulting topic:

[source,bash]
----
$ kafka-topics --zookeeper localhost:2181 --describe --topic mysql_users_avro_v2
Topic:mysql_users_avro_v2       PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: mysql_users_avro_v2      Partition: 0    Leader: 1       Replicas: 1,2,0     Isr: 1,2,0
----
< Back to the Stream Processing Cookbook

Wir verwenden Cookies, damit wir nachvollziehen können, wie Sie unsere Website verwenden, und um Ihr Erlebnis zu optimieren. Klicken Sie hier, wenn Sie mehr erfahren oder Ihre Cookie-Einstellungen ändern möchten. Wenn Sie weiter auf dieser Website surfen, stimmen Sie unserer Nutzung von Cookies zu.