Confluent
Log Compaction | Highlights in the Kafka and Stream Processing Community | February 2016
Log Compaction

Log Compaction | Highlights in the Kafka and Stream Processing Community | February 2016

Gwen Shapira

Welcome to the February 2016 edition of Log Compaction, a monthly digest of highlights in the Apache Kafka and stream processing community. Got a newsworthy item? Let us know.

  • We’ve been discussing many improvement proposals this month
    • KIP-41 – a proposal to limit the number of records returned by KafkaConsumer.poll method has been accepted.
    • KIP-42 – a proposal to add interceptors to producers and consumers has been accepted. This improvement creates interesting new monitoring options and once this is implemented, it will be interesting to hear how to community is using the new APIs.
    • KIP-43 and KIP-44 propose improvements and extensions to Kafka’s authentication protocols. These are still under active discussion, and if you are interested in security in Kafka, I suggest reading the wiki and the discussion to see where we are heading.             
    • KIP-45 – a proposal to standardize the various collections that the KafkaConsumer API expects is still under discussion, with the benefits of more standardized approach being weighed against the desire to maintain backward compatibility for this new API.
  • Many of us are just learning the ins and outs of the new consumer. This recently published blog post, with a complete end-to-end example proves very useful.
  • A passionate developer wrote very detailed blog posts on Kafka integration with Spark Streaming. This includes the little-discussed question of how to write the results of the stream processing job back into Kafka.
  • LinkedIn wrote about new features in Samza. The blog post also includes sexy throughput numbers, description of their use-case and description of how Samza is used in their data products. Really cool stuff.
  • Google contributed their Dataflow API (but not implementation) to the Apache Software Foundation and are inviting other stream processing projects to implement their SDK. We are watching to see where this will take the active stream processing community.

Subscribe to the Confluent Blog

Subscribe

More Articles Like This

Confluent + Scylla
Maheedhar Gunturu

Scylla and Confluent Integration for IoT Deployments

Maheedhar Gunturu .

The internet is not just connecting people around the world. Through the Internet of Things (IoT), it is also connecting humans to the machines all around us and directly connecting ...

Figure 4. The packaging of payloads for OWMC
Stewart Bryson

Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation

Stewart Bryson .

Red Pill Analytics was recently engaged by a Fortune 500 e-commerce and wholesale company that is transforming the way they manage inventory. Traditionally, this company has used only a few ...

User DSL Code → Logical Plan: Operator Graph → Physical Plan: Processor Topology
Bill Bejeck

Optimizing Kafka Streams Applications

Bill Bejeck .

With the release of Apache Kafka® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. This framework opens the door for various optimization techniques ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Try Confluent Platform

Download Now

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.