Audio version of the article
Kafka has become a de facto standard interface for building decoupled systems and working with streaming data. Despite its widespread popularity, there are numerous accounts of the difficulty that operators face in keeping it reliable and performant, or trying to scale an installation. To make the benefits of the Kafka ecosystem more accessible and reduce the operational burden, Alexander Gallego and his team at Vectorized created the Red Panda engine. In this episode he explains how they engineered a drop-in replacement for Kafka, replicating the numerous APIs, that can scale more easily and deliver consistently low latencies with a much lower hardware footprint. He also shares some of the areas of innovation that they have found to help foster the next wave of streaming applications while working within the constraints of the existing Kafka interfaces. This was a fascinating conversation with an energetic and enthusiastic engineer and founder about the challenges and opportunities in the realm of streaming data.
- How did you get involved in the area of data management?
- Can you start by describing what Red Panda is and what motivated you to create it?
- What are the limitations of Kafka that make something like Red Panda necessary?
- What are the current strengths of the Kafka ecosystem that make it a reasonable implementation target for Red Panda?
- How is Red Panda architected?
- How has the design or direction changed or evolved since you first began working on it?
- What are the challenges that you face in automatically optimizing the runtime to take advantage of the hardware that it is deployed on?
- How do cloud environments contribute to that complexity?
- How are you handling the compatibility layer for the Kafka API?
- What is your approach for managing versioning and ensuring that you maintain bug compatibility?
- Beyond performance, what other areas of innovation or improvement in the capabilities and experience do you see while adhering to the Kafka protocol?
- What are the opportunities for innovation in the streaming space that aren’t being explored yet?
- What are some of the most interesting, innovative, or unexpected ways that you have seen Redpanda being used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while building Red Panda and Vectorized?
- When is Red Panda the wrong choice?
- What do you have planned for the future of the product and business?
- What is your Hack The Planet diversity scholarship?
This article has been published from the source link without modifications to the text. Only the headline has been changed.