Streaming-ETL

Inspecting and Changing Topic Keys

KSQL provides a powerful way for you to change the properties of Kafka topics by defining new streams with the desired properties of the new topic, populated by the streaming events of the original topic.

Because these are streaming transformations, the existing topic gets streamed to the new topic with the desired properties, and the target topic is kept in sync with every single message that subsequently arrives on the source topic.

Environment4.1 or higher
Robin Moffatt

Directions

1. You can use the ROWKEY system column to inspect the key on a KSQL stream or table:

ksql> SELECT ROWKEY, ID, FIRST_NAME, LAST_NAME FROM CUSTOMERS LIMIT 5;
1 | 1 | Bibby | Argabrite
3 | 3 | Marv | Dalrymple
7 | 7 | Marigold | Veld
8 | 8 | Ruperto | Matteotti
5 | 5 | Modestia | Coltart

Here you can see that the key of the message (ROWKEY) being read from the Kafka topic is the same as the ID column.

2. To change the key used for a topic, use the PARTITION BY clause. This can be especially useful for ensuring that topics which are used for KSQL tables are correctly keyed. In this example, you repartition the topic from ID to LAST_NAME:

CREATE STREAM CUSTOMERS_REKEY AS SELECT * FROM CUSTOMERS PARTITION BY LAST_NAME;

3. Use DESCRIBE EXTENDED to confirm that the key has been set:

ksql> DESCRIBE EXTENDED CUSTOMERS_REKEY;

Type                 : STREAM
Key field            : LAST_NAME
[...]

4. Validate on a per-message basis that the key is now as required:

ksql> SELECT ROWKEY, ID, FIRST_NAME, LAST_NAME FROM CUSTOMERS_REKEY LIMIT 5;
Yeeles | 4 | Nolana | Yeeles
Dalrymple | 3 | Marv | Dalrymple
Coltart | 5 | Modestia | Coltart
Acaster | 6 | Bram | Acaster
Argabrite | 1 | Bibby | Argabrite
< Back to the Stream Processing Cookbook