Apache Kafka vs Redpanda Sizing Calculator
Calculate exact broker node counts, NVMe disk throughput (MB/s), EBS IOPS, and network egress requirements for Apache Kafka (JVM/KRaft) versus Redpanda (C++ Seastar).
Throughput & Ingestion Profile
MB/s
e.g., 50 MB/sec = ~50,000 msgs/sec @ 1KB
groups
Cluster Sizing Recommendations
12.1 TB / node
25% Unpartitioned Reserve
Embed this Apache Kafka vs Redpanda Hardware & Disk IOPS Sizing Calculator on Your Website
Add this interactive, privacy-first calculator to your blog or web application. 100% free and mobile-responsive.
JVM Garbage Collection vs Seastar Direct I/O: The Architectural Divergence
Apache Kafka and Redpanda share identical client API protocols (Kafka wire protocol compatible), but their underlying operating system interaction models represent two opposite software paradigms:
- Apache Kafka (Java / JVM + Linux PageCache): Kafka runs on the Java Virtual Machine. To avoid JVM Garbage Collection pauses on gigabyte-sized heaps, Kafka keeps its JVM heap modest (typically 6 GB to 10 GB) and delegates all message caching directly to the Linux Operating System PageCache. While efficient, when multiple consumer groups read historical data out of cache, the Linux kernel must swap memory pages, inducing PageCache eviction thrashing that degrades active producer write latencies.
- Redpanda (C++ / Seastar Thread-Per-Core): Redpanda bypasses the Linux PageCache entirely using direct I/O (
O_DIRECT). Redpanda pins one C++ fiber thread to each hardware CPU core. Memory buffers are allocated directly within user space, guaranteeing zero Garbage Collection pauses, deterministic sub-5ms P99 latencies, and 3x to 6x lower hardware compute footprint on identical event loads.
High-Availability Partition Sizing & In-Sync Replicas (ISR)
To prevent data loss while maximizing streaming parallelism, engineers must size topic partitions according to cluster concurrency:
- The Rule of Partition Throughput: A single Kafka partition can reliably sustain 10 MB/sec to 25 MB/sec of write throughput. If an application streams 100 MB/sec, the topic must be provisioned with at least
100 / 15 = 7 to 10 partitionsdistributed across broker nodes. - In-Sync Replicas (min.insync.replicas): For financial transactions, order processing, and audit trails, always configure
replication.factor = 3andmin.insync.replicas = 2combined with producer settingacks = all. If one broker crashes during a partition rebalance, the cluster continues acknowledging writes safely without data loss.
Production Broker Configuration Directives
# server.properties - Production High-Throughput Kafka Broker num.partitions=12 default.replication.factor=3 min.insync.replicas=2 # Memory & Buffer Directives socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 # Log Flush & Retention Sizing (Let OS manage PageCache flushing) log.flush.interval.messages=9223372036854775807 log.flush.interval.ms=9223372036854775807 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000
Frequently Asked Questions: Streaming Broker Sizing
Why is ZooKeeper being phased out in Apache Kafka (KRaft)?
In legacy Kafka deployments, cluster metadata and partition state were managed by an external Apache ZooKeeper quorum. ZooKeeper introduced severe scaling bottlenecks when cluster partition counts exceeded 100,000. Modern Kafka operates on KRaft (Kafka Raft Metadata mode), where broker nodes elect a metadata controller internally using the Raft consensus protocol, accelerating partition rebalances by over 10x.
Does Redpanda support tiered storage to S3 object storage?
Yes. Redpanda features native Tiered Storage (Shadow Indexing). Hot, real-time message streams are stored locally on high-speed NVMe SSDs, while older historical log segments are asynchronously uploaded to AWS S3 or Google Cloud Storage. Consumers reading 30-day-old logs fetch data directly from S3 without consuming local broker disk IOPS.
Production Incident Runbook: Mitigating Kafka PageCache Thrashing & Consumer Lag Storms
In high-throughput event streaming architectures, Apache Kafka brokers rely on Linux PageCache to serve consumer read requests directly from RAM using zero-copy OS calls (sendfile). However, when a downstream consumer crashes, falls hours behind, or an offline batch analytics pipeline spins up, these slow consumers request historical log segments evicted from PageCache. This forces the Kafka broker to read directly from physical disk, creating catastrophic PageCache thrashing, driving disk IOPS to 100% saturation, and degrading real-time produce latency from 3ms to over 2,000ms. Follow this emergency triage runbook:
- Identify the Lagging Consumer Group & Offending Partitions: Immediately execute the Kafka consumer group CLI tool to detect which partition offsets are lagging:
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group analytics-batch-pipeline
Isolate any consumer with lag exceeding 1,000,000 offsets.
- Throttle or Quarantine Historical Batch Consumers: Do not allow cold-data consumers to exhaust broker disk bandwidth. Apply dynamic broker client quotas to throttle the offending consumer’s byte fetch rate:
kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'consumer_byte_rate=10485760' --entity-type clients --entity-name analytics-batch-pipeline
This restricts the lagging consumer to 10 MB/s, protecting real-time producer write bandwidth.
- Inspect Broker Disk Read Saturation (iostat): On the Kafka broker nodes, execute
iostat -xz 1 10. If%utilis consistently above 95% andr/s(read IOPS) exceeds provisioned EBS limits, the broker is starved of I/O. - Dynamic Topic Segment Retention Truncation: If disk capacity drops below 15% free space, prevent read-only broker shutdown by temporarily reducing retention hours on high-volume debug topics:
kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name telemetry-raw --alter --add-config retention.ms=86400000
- Implement Redpanda Tiered Storage for Zero-Impact Historical Reads: Modern streaming architectures migrate cold reads off broker disks entirely. Platforms like Redpanda utilize Tiered Storage to stream historical segments directly from AWS S3, guaranteeing that cold consumer replays never touch local NVMe caches or degrade live producer throughput.
Streaming Architecture TCO Comparison: Apache Kafka vs Redpanda vs Managed Confluent Cloud
Evaluating the operational cost of message streaming requires analyzing broker count, JVM memory allocation, storage replication overhead, and multi-AZ egress networking. The benchmark table below compares infrastructure and maintenance costs for an enterprise cluster processing 100 MB/sec ingestion (8.64 TB/day raw) with 3-way replication (RF=3):
| Architecture Dimension | Apache Kafka (JVM + KRaft) | Redpanda (C++ Seastar) | Confluent Cloud (Enterprise SaaS) |
|---|---|---|---|
| Minimum Broker Count | 6 to 8 nodes (r6i.2xlarge) | 3 nodes (i3en.2xlarge NVMe) | Serverless CKU capacity units (~4 CKUs) |
| Thread & Memory Model | JVM Garbage Collection + OS PageCache | Thread-per-core (No GC, O_DIRECT NVMe) | Managed Multi-tenant Infrastructure |
| p99.9 Produce Latency | 18ms – 45ms (GC pauses & disk syncs) | 2.5ms – 5.0ms (Deterministic) | 8ms – 25ms |
| Storage Layer Architecture | EBS gp3 / io2 volumes (Costly disk expansion) | Local NVMe + S3 Tiered Storage | Confluent Infinite Storage (Object Tier) |
| Monthly Cloud Compute & Disk Bill | $3,450.00 / mo (EC2 + EBS gp3 provisioned) | $1,480.00 / mo (EC2 i3en + S3 cold tier) | Included in CKU base fees |
| Total Monthly Running Cost | $3,450 + $2,000 Ops engineering = $5,450/mo | $1,480 + $500 Ops engineering = $1,980/mo | $4,800 – $6,200 / month |
Production-Hardened Redpanda Docker Compose with Prometheus Telemetry
Deploying Redpanda provides complete Kafka API compatibility without JVM overhead, ZooKeeper, or KRaft controller orchestration complexity. Below is a production-hardened Docker Compose specification configuring thread-per-core pinning, memory lock, and tiered storage to AWS S3:
version: '3.8'
services:
redpanda-broker:
image: docker.redpanda.com/redpandadata/redpanda:v24.2.5
container_name: redpanda-prod-node1
network_mode: host
restart: always
command:
- redpanda
- start
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
- --advertise-kafka-addr internal://10.0.1.15:9092,external://lb.zelobiz.com:19092
- --pandaproxy-addr 0.0.0.0:8082
- --schema-registry-addr 0.0.0.0:8081
- --rpc-addr 0.0.0.0:33145
- --advertise-rpc-addr 10.0.1.15:33145
- --mode production
- --default-log-segment-size 134217728 # 128 MB segments
- --smp 4 # Pin to 4 CPU cores
- --memory 16G # Allocate 16 GB dedicated RAM
- --reserve-memory 2G # OS kernel reserve
- --overprovisioned 0
- --lock-memory true
environment:
- REDPANDA_ENVIRONMENT=production
volumes:
- /mnt/nvme0n1/redpanda:/var/lib/redpanda/data
ulimits:
nofile:
soft: 65535
hard: 65535
memlock:
soft: -1
hard: -1
redpanda-console:
image: docker.redpanda.com/redpandadata/console:v2.7.2
container_name: redpanda-web-console
restart: always
ports:
- "8080:8080"
environment:
- KAFKA_BROKERS=127.0.0.1:9092
- KAFKA_SCHEMAREGISTRY_ENABLED=true
- KAFKA_SCHEMAREGISTRY_URLS=http://127.0.0.1:8081
Kafka OS Kernel & Disk IOPS Tuning Cheatsheet
To prevent broker disk write stalls when running Apache Kafka on Linux, tune virtual memory dirty page ratios and I/O schedulers in /etc/sysctl.conf:
- Lower Dirty Background Ratio: Set
vm.dirty_background_ratio = 5andvm.dirty_ratio = 10. This forces the Linux kernel flusher threads to continuously write dirty pages to NVMe/SSD storage rather than buffering large bursts that freeze I/O for seconds. - Disable OS Swapping: Set
vm.swappiness = 1. Paging JVM heap memory or broker caches to swap disk induces instantaneous latency spikes exceeding 1,000ms. - Mount Filesystems with noatime: When mounting broker data directories on XFS or ext4, always specify
noatime,nodiratime,barrier=0to eliminate disk write metadata overhead on every read operation. - Enforce Disk Queue Depth with NVMe: For AWS EBS gp3 volumes, provision a minimum of 3,000 IOPS and 250 MB/s throughput per broker to survive simultaneous partition rebalances and peak ingestion bursts.
Frequently Asked Questions: Kafka & Redpanda Hardware Sizing
Why does Apache Kafka recommend 3-way replication (RF=3) as the production standard?
A replication factor of 3 paired with min.insync.replicas=2 guarantees zero data loss (RPO=0) and high availability even when one broker suffers hardware failure while a second undergoes rolling upgrades. If RF=2 were used, a single node loss leaves the partition with zero redundancy, risking immediate downtime.
How does Redpanda achieve 10x lower latency than Apache Kafka?
Redpanda is written in C++ using the Seastar asynchronous framework. It implements a thread-per-core shared-nothing architecture that bypasses the Linux PageCache using direct I/O (O_DIRECT) to write directly to NVMe storage. This completely eliminates JVM garbage collection pauses, memory cache context switching, and thread locking contention.
How much RAM does a Kafka broker actually require?
Kafka brokers typically need only 6 GB to 8 GB of JVM heap memory (for internal state and metadata). However, the underlying physical node should have 32 GB to 64 GB of RAM because Kafka utilizes all remaining unallocated memory as OS PageCache to serve consumers with zero-copy disk reads.
What is Kafka Tiered Storage and when should it be activated?
Tiered Storage separates streaming compute from long-term data retention. Active, warm partitions remain on fast local NVMe/EBS disks, while completed log segments are automatically offloaded to inexpensive object storage (Amazon S3, Google Cloud Storage). This allows organizations to retain event streams for months or years without provisioning petabytes of expensive block storage.