Project Metamorphosis: Wir präsentieren die Event-Streaming-Plattform der nächsten GenerationMehr Erfahren

Data Serialization

Changing Data Serialization Format from Avro to CSV

KSQL provides a powerful way for developers to reserialize the data in a Kafka topic by defining new streams with the desired serialization of the new topic, populated by the streaming events of the original topic. Here, we convert a stream of Avro-formatted data into CSV.

The fantastic thing here is that these are streaming transformations, so not only does all existing data on the topic get converted, but so do every single message that subsequently arrives on the source topic.

Directions

1. Define the source topic’s schema:

CREATE STREAM source_avro 
WITH (KAFKA_TOPIC='mysql_users_avro', VALUE_FORMAT='AVRO');

2. Create a derived topic in delimited (CSV) format:

CREATE STREAM target_delim WITH (KAFKA_TOPIC='mysql_users_delim',VALUE_FORMAT='DELIMITED') AS 
SELECT * FROM source_avro;
You will get the resulting data:
$ kafkacat -C -b localhost:9092 -t mysql_users_delim
1,Cliff,en_US,St Louis,P
< Back to the Stream Processing Cookbook

Diese Website verwendet Cookies zwecks Verbesserung der Benutzererfahrung sowie zur Analyse der Leistung und des Datenverkehrs auf unserer Website. Des Weiteren teilen wir Informationen über Ihre Nutzung unserer Website mit unseren Social-Media-, Werbe- und Analytics-Partnern.