Project Metamorphosis: Unveiling the next-gen event streaming platformLearn More

Log Compaction | Highlights in the Apache Kafka and Stream Processing Community | September 2016

It is September and it’s evident that everyone is back from their summer vacation! We released Apache Kafka which includes fixes of the bugs in the 0.10.0 release. In our last meeting we agreed to give time-based releases a try and immediately started planning Apache Kafka

  • Confluent Platform 3.0.1 and Apache Kafka were released. Lots of important bug fixes! If you are on Apache Kafka 0.10.0 or Confluent Platform 3.0.0, we recommend upgrading. If you are on an older release, please make sure you upgrade directly to the bugfix version.
  • We agreed to try Time-Based Release Plan. Aiming for 3 Apache Kafka releases a year (one every 4 months) and guaranteeing rolling upgrades for a duration of two years.
  • We started planning the next Apache Kafka release, which will have the version Much thanks to Jason Gustafson, Kafka’s newest committer for volunteering to drive the release. As usual, the community is encouraged to participate. Take a look at the release plan to learn how.
  • KIP-62 has been merged and will be included in Apache Kafka and Confluent Plafrom 3.10. This KIP adds a background thread to the Kafka Consumer, allowing background heartbeats which will keep alive Consumers that stop polling. This should make it much easier to write consumers, especially consumers that need to process large amounts of data between iterations.
  • KIP-63, a proposal for improving caching in the Streams API in Kafka, was approved. This is a significant performance optimization that coalesces processing updates before sending them downstream, which reduces the load on Kafka clusters and on downstream external systems. It also paves the way for implementing new “trigger” behaviors.
  • KIP-71 was approved, allowing messages in topics to be both compacted and deleted. This will allow admins to impose disk constraints on compacted topics, by removing compacted keys which are older than the time limit or exceed disk space limits.
  • KIP-73 was approved, adding replication quotas or throttling to Apache Kafka. This feature is especially useful when reassigning replicas to brokers, allowing admins to limit the resources used by the reassignment process and therefore reducing the risk in reassignment. Replica reassignment has long been a difficult process in Apache Kafka, and we are excited about this improvement.
  • KIP-79, a proposal to evolve the Apache Kafka protocol to allow for requesting offsets according to timestamps (using the new timestamp indexes) is under active discussion. You are invited to take a look and share your feedback with the Kafka community.
  • Ben Stopford gave a very popular presentation on how Microservices and Apache Kafka fit together.
  • If you are curious to learn about the internals of the new Kafka Consumer Groups, you can watch this presentation from Kafka meetup at LinkedIn.
  • Want to learn how to choose a stream processing framework? Neha Narkhede and Stephan Ewen the Streams API in Kafka and Flink, providing good decision guidelines in the process.
  • Are Kafka Connect and Kafka Streams  ready for production? The Kafka community says yes! LINE Corp. explain how they are using Kafka Streams in large-scale production, and WePay talk about their use of Kafka Connect in large-scale production.
  • Grant Henke explains the architectural benefits of Apache Kafka for decoupling.
  • Confluent has updated the schedule of training classes for developers and operators of Kafka. Online courses are also available.

Did you like this blog post? Share it now

Subscribe to the Confluent blog

More Articles Like This

Kafka Streams Interactive Queries Go Prime Time

What is stopping you from using Kafka Streams as your data layer for building applications? After all, it comes with fast, embedded RocksDB storage, takes care of redundancy for you, […]

Apache Kafka Needs No Keeper: Removing the Apache ZooKeeper Dependency

Currently, Apache Kafka® uses Apache ZooKeeper™ to store its metadata. Data such as the location of partitions and the configuration of topics are stored outside of Kafka itself, in a […]

From Eager to Smarter in Apache Kafka Consumer Rebalances

Everyone wants their infrastructure to be highly available, and ksqlDB is no different. But crucial properties like high availability don’t come without a thoughtful, rigorous design. We thought hard about […]

Jetzt registrieren

Start your 3-month trial. Get up to $200 off on each of your first 3 Confluent Cloud monthly bills

Nur neue Registrierungen.

Wenn Sie oben auf „registrieren“ klicken, erklären Sie sich damit einverstanden, dass wir Ihre personenbezogenen Daten verarbeiten – gemäß unserer und bin damit einverstanden.

Indem Sie oben auf „Registrieren“ klicken, akzeptieren Sie die Nutzungsbedingungen und den gelegentlichen Erhalt von Marketing-E-Mails von Confluent. Zudem ist Ihnen bekannt, dass wir Ihre personenbezogenen Daten gemäß unserer und bin damit einverstanden.

Auf einem einzigen Kafka Broker unbegrenzt kostenlos verfügbar

Die Software ermöglicht die unbegrenzte Nutzung der kommerziellen Funktionen auf einem einzelnen Kafka Broker. Nach dem Hinzufügen eines zweiten Brokers startet automatisch ein 30-tägiger Timer für die kommerziellen Funktionen, der auch durch ein erneutes Herunterstufen auf einen einzigen Broker nicht zurückgesetzt werden kann.

Wählen Sie den Implementierungstyp aus
Manuelle Implementierung
  • tar
  • zip
  • deb
  • rpm
  • docker
Automatische Implementierung
  • kubernetes
  • ansible

Wenn Sie oben auf „kostenlos herunterladen“ klicken, erklären Sie sich damit einverstanden, dass wir Ihre personenbezogenen Daten verarbeiten – gemäß unserer Datenschutzerklärung zu.

Indem Sie oben auf „kostenlos herunterladen“ klicken, akzeptieren Sie die Confluent-Lizenzvertrag und den gelegentlichen Erhalt von Marketing-E-Mails von Confluent. Zudem erklären Sie sich damit einverstanden, dass wir Ihre personenbezogenen Daten gemäß unserer Datenschutzerklärung zu.

Diese Website verwendet Cookies zwecks Verbesserung der Benutzererfahrung sowie zur Analyse der Leistung und des Datenverkehrs auf unserer Website. Des Weiteren teilen wir Informationen über Ihre Nutzung unserer Website mit unseren Social-Media-, Werbe- und Analytics-Partnern.