1. 红孩笔记首页
  2. 大数据

How Does Kafka Perform When You Need Low Latency?

• 大数据, 计算机科学 • 阅读 4710

Most Kafka benchmarks appear to test high throughput but not low latency. Apache Kafka was traditionally used for high throughput rather than latency-sensitive messaging, but it does have a low-latency configuration. (Mostly setting linger.ms=0 and reducing buffer sizes). In this configuration, you can get below 1-millisecond latency a good percentage of the time for modest throughputs.

Benchmarks tend to focus on clustering Kafka in a high-throughput configuration. While this is perhaps the most common use case, how does it perform if you need lower latencies?

Where are Some Latency Benchmarks Available?

These are various benchmarks testing higher throughputs of 200kmsg/s to 800kmsg/s, with end-to-end latencies between 2.5 and 30 milliseconds.

  • Confluent benchmark, looking at the 99 percentile latency compared with Apache Pulsar and Rabbit MQ (pro Kafka). “Kafka provides the lowest latency at higher throughputs, while also providing strong durability and high availability.”

  • NativeStream benchmark comparing Pulsar to Kafka (pro Pulsar). “Pulsar’s 99th percentile latency is within the range of 5 and 15 milliseconds.”

  • Instacluster performance, looking at average latencies with varying number of producers, with different configurations.

  • Datastax latency benchmark using the same benchmark as Confluent. Their conclusion appears to be that, when flushing every message to disk, Pulsar is better.

  • Using Confluent Cloud from AWS: “With my specific test parameters, Kafka p99 latencies are 100-200 ms and much lower than Pub/Sub latencies.”

My impression is that these benchmarks aren’t so much an attempt to show low latency, but rather show what the authors consider good latency under high load.

Benchmarking Kafka for Low Latency

For a low-latency system, you want the hardware which will best support your requirements. This is often plenty of the fastest CPUs you can afford and more than enough IO bandwidth as well.

The best way to go fast is often to do as little as possible, and keep the solution simple. In my case, I am starting with just one PC, a Ryzen 9 5950X with 64 GB memory and a Corsair MP600 PRO XT M.2 drive.

Obviously cluster support is an important use case for Kafka, but let’s start with a really simple end-to-end use case: one machine, two message hops and a trivial microservice in between.

One Machine, One Trivial Microservice, End-to-End Latency

This benchmark is similar to a previous one found here. However, Kafka is configured for lower latencies and multiple producers are used to support a significant, but lower, message throughput.

In this configuration, Kafka has a fraction of the latencies reported in the benchmarks above.

One producer doesn’t handle this throughput well, but two and above producers (I tested up to 10) produce good results. Increasing the number of partitions only increased the overhead (albeit slightly). Increasing the number of consumers saw a small variation in latencies.

To put this in perspective, I added the results for a single producer using Chronicle Queue Enterprise which you might expect has far, far less jitter. (See the almost invisible blue line at the bottom of the graph above. The line runs just above the X-Axis; the reason this line can’t be seen is that Chronicle Queue is performing significantly better than Kafka.) This indicates the performance between processes on the same machine.

No Conclusion

I like to finish with a conclusion, but this leaves me with more questions than answers. The benchmarks linked at the start of the post aim to discuss the low latency characteristics of Kafka. However, in actual fact, these tests appear to have instead configured Kafka to maximize throughput rather than for low latency. Kafka can produce better benchmark numbers when suitably configured for low latency, but even in this setup, other options can perform two or more orders of magnitude better.

文章来源于互联网:How Does Kafka Perform When You Need Low Latency?

发布者：小站，转转请注明出处：http://blog.gzcity.top/4198.html

赞 (0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
1140
上一篇 2022年5月3日 02:49
下一篇 2022年5月3日 18:08

相关推荐

  • React 18 - 新功能和改进策略 计算机科学

    React 18 – 新功能和改进策略

    3 月 8 日，React 团队发布了 React 18 RC（Release Candidate ）。最新版本带来了许多新功能，这些功能将改变许多应用程序中的编码模式。它带来了一些性能改进，我将在本博客中介绍这些改进。 并发 并发性是多个进程同时执行，并且可能相互交互也可能不交互的系统的属性。太复杂了？让我们分解一下。假设有一场比赛正在进行中。现在，并发性…

    2022年4月23日
    47820
  • WIN10 控制台cmd乱码及永久修改编码的解决办法 学习笔记

    WIN10 控制台cmd乱码及永久修改编码的解决办法

    WIN10 cmd控制台本来的编码是ANSI的,所以要求执行的批处理脚本编码格式也是ANSI才行,要不就两边统一都改成UTF-8处理(作为一个程序员,一般我都会统一成UTF-8编码格式),下面就介绍如何统一改成UTF-8编码格式   一、修改控制台CMD编码格式为UTF-8 我本机的系统环境： OS Name: Microsoft Windows 10 企业…

    2022年12月6日
    626340
  • What Is a Data Reliability Engineer, and Do You Really Need One? 大数据

    What Is a Data Reliability Engineer, and Do You Really Need One?

    As software systems became increasingly complex in the late 2000s, merging development and operations (DevOps) was a no-brainer.  One-half software engineer, one-half operations ad…

    2022年5月3日
    9.9K22510
  • Intro to Yelp Web Scraping Using Python 大数据

    Intro to Yelp Web Scraping Using Python

    Originally published June 17, 2020 Like many programmers who hold degrees that are not even relevant to computer programming, I was struggling to learn coding by myself since 2019 …

    2022年5月3日
    5.1K12340
  • The Lakehouse: An Uplift of Data Warehouse Architecture 大数据

    The Lakehouse: An Uplift of Data Warehouse Architecture

    In short, the initial architecture of the data warehouse was designed to provide analytical insights by collecting data from various heterogeneous data sources into the centralized…

    2022年5月3日
    593120

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注

评论列表（1,140条）