Confluent
Log Compaction | Highlights in the Kafka and Stream Processing Community | January 2016
Log Compaction

Log Compaction | Highlights in the Kafka and Stream Processing Community | January 2016

Gwen Shapira

Happy 2016! Wishing you a wonderful, highly scalable, and very reliable year. Log Compaction is a monthly digest of highlights in the Apache Kafka and stream processing community. Got a newsworthy item? Let us know.

Many things have happened since we last shared the state of Apache Kafka and the streams ecosystem. Lets take a look!

  • Kafka 0.9 was released and so was Confluent Platform 2.0
  • The call for proposals for Kafka Summit is closing soon – submit your abstract by Monday, January 11! Make it your new year’s resolution to participate more in the Kafka community and you can start by registering for the conference. Catch the Early Bird price (save $100) before it expires on January 15. 
  • Congratulations to Ewen Cheslack-Postava who joined Apache Kafka as a committer! We wish him much success and many patch reviews.
  • Kafka 0.9 added protocol support for managing groups so that clients no longer need to interact with ZooKeeper directly. KIP-40 gives a design for adding protocol support for clients to list available groups and to show the group members and their lag. The protocol is implemented in the new ConsumerGroupCommand (kafka-consumer-groups.sh), and is also available for use by 3rd party clients.
  • Kafka currently only supports a “logical” notion of time – the message offset, which indicates a relative order of messages. Many users want to be able to know the physical time that a message was produced. KIP-32, which is in active discussions and voting, will add a timestamp field to each Kafka message, indicating the time the client created the message. This would eventually allow adding special indexes and also support consuming messages based on their timestamp.
  • Kafka Connect is a new feature introduced in 0.9 that makes it really easy to directly integrate Kafka with external data systems like RDBMS’s or Hadoop. This tutorial shows how to use Kafka Connect to get events from a relational database to Kafka, and from there to HDFS and Hive, including automated partitioning of the data and updates to the Hive schema.
  • Looking to use Kafka with Spring for stream processing? There’s a 5-part blog post on how to do just that.
  • Yahoo published a benchmark comparing popular stream processing frameworks – Storm, SparkStreaming, and Flink.
  • At the first Seattle Kafka Meetup this past November, Microsoft shared how they use Kafka in Bing. One trillion messages per day!
  • Kafka, The Definitive Guide is now available for pre-order from O’Reilly.

Subscribe to the Confluent Blog

Subscribe

More Articles Like This

Confluent + Scylla
Maheedhar Gunturu

Scylla and Confluent Integration for IoT Deployments

Maheedhar Gunturu .

The internet is not just connecting people around the world. Through the Internet of Things (IoT), it is also connecting humans to the machines all around us and directly connecting ...

Figure 4. The packaging of payloads for OWMC
Stewart Bryson

Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation

Stewart Bryson .

Red Pill Analytics was recently engaged by a Fortune 500 e-commerce and wholesale company that is transforming the way they manage inventory. Traditionally, this company has used only a few ...

Multi-Cluster: Kafka Connect, KSQL and Kafka Brokers
Robin Moffatt

Dawn of Kafka DevOps: Managing Multi-Cluster Kafka Connect and KSQL with Confluent Control Center

Robin Moffatt .

In anything but the smallest deployment of Apache Kafka®, there are often going to be multiple clusters of Kafka Connect and KSQL. Kafka Connect is used for building event streaming ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.