Apache Kafka vs Redpanda Hardware & Disk IOPS Sizing Calculator

Distributed Streaming & Message Brokers

Apache Kafka vs Redpanda Sizing Calculator

Calculate exact broker node counts, NVMe disk throughput (MB/s), EBS IOPS, and network egress requirements for Apache Kafka (JVM/KRaft) versus Redpanda (C++ Seastar).

Throughput & Ingestion Profile


MB/s

e.g., 50 MB/sec = ~50,000 msgs/sec @ 1KB




groups


Cluster Sizing Recommendations

Recommended Broker Topology
3 Nodes × i3en.xlarge
4 vCPU / 32 GB RAM per node (12 vCPU / 96 GB Total)

Total Disk Required
36.2 TB

Write Disk Throughput
150 MB/s

Egress Network Bandwidth
1.60 Gbps (Consumers + Replication)

Storage & Reliability Telemetry
Disk per Broker Node:
12.1 TB / node
Headroom Safety Multiplier:
25% Unpartitioned Reserve

Free Embeddable Widget

Embed this Apache Kafka vs Redpanda Hardware & Disk IOPS Sizing Calculator on Your Website

Add this interactive, privacy-first calculator to your blog or web application. 100% free and mobile-responsive.

JVM Garbage Collection vs Seastar Direct I/O: The Architectural Divergence

Apache Kafka and Redpanda share identical client API protocols (Kafka wire protocol compatible), but their underlying operating system interaction models represent two opposite software paradigms:

  • Apache Kafka (Java / JVM + Linux PageCache): Kafka runs on the Java Virtual Machine. To avoid JVM Garbage Collection pauses on gigabyte-sized heaps, Kafka keeps its JVM heap modest (typically 6 GB to 10 GB) and delegates all message caching directly to the Linux Operating System PageCache. While efficient, when multiple consumer groups read historical data out of cache, the Linux kernel must swap memory pages, inducing PageCache eviction thrashing that degrades active producer write latencies.
  • Redpanda (C++ / Seastar Thread-Per-Core): Redpanda bypasses the Linux PageCache entirely using direct I/O (O_DIRECT). Redpanda pins one C++ fiber thread to each hardware CPU core. Memory buffers are allocated directly within user space, guaranteeing zero Garbage Collection pauses, deterministic sub-5ms P99 latencies, and 3x to 6x lower hardware compute footprint on identical event loads.

High-Availability Partition Sizing & In-Sync Replicas (ISR)

To prevent data loss while maximizing streaming parallelism, engineers must size topic partitions according to cluster concurrency:

  1. The Rule of Partition Throughput: A single Kafka partition can reliably sustain 10 MB/sec to 25 MB/sec of write throughput. If an application streams 100 MB/sec, the topic must be provisioned with at least 100 / 15 = 7 to 10 partitions distributed across broker nodes.
  2. In-Sync Replicas (min.insync.replicas): For financial transactions, order processing, and audit trails, always configure replication.factor = 3 and min.insync.replicas = 2 combined with producer setting acks = all. If one broker crashes during a partition rebalance, the cluster continues acknowledging writes safely without data loss.

Production Broker Configuration Directives

# server.properties - Production High-Throughput Kafka Broker
num.partitions=12
default.replication.factor=3
min.insync.replicas=2

# Memory & Buffer Directives
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600

# Log Flush & Retention Sizing (Let OS manage PageCache flushing)
log.flush.interval.messages=9223372036854775807
log.flush.interval.ms=9223372036854775807
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Frequently Asked Questions: Streaming Broker Sizing

Why is ZooKeeper being phased out in Apache Kafka (KRaft)?

In legacy Kafka deployments, cluster metadata and partition state were managed by an external Apache ZooKeeper quorum. ZooKeeper introduced severe scaling bottlenecks when cluster partition counts exceeded 100,000. Modern Kafka operates on KRaft (Kafka Raft Metadata mode), where broker nodes elect a metadata controller internally using the Raft consensus protocol, accelerating partition rebalances by over 10x.

Does Redpanda support tiered storage to S3 object storage?

Yes. Redpanda features native Tiered Storage (Shadow Indexing). Hot, real-time message streams are stored locally on high-speed NVMe SSDs, while older historical log segments are asynchronously uploaded to AWS S3 or Google Cloud Storage. Consumers reading 30-day-old logs fetch data directly from S3 without consuming local broker disk IOPS.

Production Incident Runbook: Mitigating Kafka PageCache Thrashing & Consumer Lag Storms

In high-throughput event streaming architectures, Apache Kafka brokers rely on Linux PageCache to serve consumer read requests directly from RAM using zero-copy OS calls (sendfile). However, when a downstream consumer crashes, falls hours behind, or an offline batch analytics pipeline spins up, these slow consumers request historical log segments evicted from PageCache. This forces the Kafka broker to read directly from physical disk, creating catastrophic PageCache thrashing, driving disk IOPS to 100% saturation, and degrading real-time produce latency from 3ms to over 2,000ms. Follow this emergency triage runbook:

  1. Identify the Lagging Consumer Group & Offending Partitions: Immediately execute the Kafka consumer group CLI tool to detect which partition offsets are lagging:
    kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group analytics-batch-pipeline

    Isolate any consumer with lag exceeding 1,000,000 offsets.

  2. Throttle or Quarantine Historical Batch Consumers: Do not allow cold-data consumers to exhaust broker disk bandwidth. Apply dynamic broker client quotas to throttle the offending consumer’s byte fetch rate:
    kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config 'consumer_byte_rate=10485760' --entity-type clients --entity-name analytics-batch-pipeline

    This restricts the lagging consumer to 10 MB/s, protecting real-time producer write bandwidth.

  3. Inspect Broker Disk Read Saturation (iostat): On the Kafka broker nodes, execute iostat -xz 1 10. If %util is consistently above 95% and r/s (read IOPS) exceeds provisioned EBS limits, the broker is starved of I/O.
  4. Dynamic Topic Segment Retention Truncation: If disk capacity drops below 15% free space, prevent read-only broker shutdown by temporarily reducing retention hours on high-volume debug topics:
    kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name telemetry-raw --alter --add-config retention.ms=86400000
  5. Implement Redpanda Tiered Storage for Zero-Impact Historical Reads: Modern streaming architectures migrate cold reads off broker disks entirely. Platforms like Redpanda utilize Tiered Storage to stream historical segments directly from AWS S3, guaranteeing that cold consumer replays never touch local NVMe caches or degrade live producer throughput.

Streaming Architecture TCO Comparison: Apache Kafka vs Redpanda vs Managed Confluent Cloud

Evaluating the operational cost of message streaming requires analyzing broker count, JVM memory allocation, storage replication overhead, and multi-AZ egress networking. The benchmark table below compares infrastructure and maintenance costs for an enterprise cluster processing 100 MB/sec ingestion (8.64 TB/day raw) with 3-way replication (RF=3):

Architecture Dimension Apache Kafka (JVM + KRaft) Redpanda (C++ Seastar) Confluent Cloud (Enterprise SaaS)
Minimum Broker Count 6 to 8 nodes (r6i.2xlarge) 3 nodes (i3en.2xlarge NVMe) Serverless CKU capacity units (~4 CKUs)
Thread & Memory Model JVM Garbage Collection + OS PageCache Thread-per-core (No GC, O_DIRECT NVMe) Managed Multi-tenant Infrastructure
p99.9 Produce Latency 18ms – 45ms (GC pauses & disk syncs) 2.5ms – 5.0ms (Deterministic) 8ms – 25ms
Storage Layer Architecture EBS gp3 / io2 volumes (Costly disk expansion) Local NVMe + S3 Tiered Storage Confluent Infinite Storage (Object Tier)
Monthly Cloud Compute & Disk Bill $3,450.00 / mo (EC2 + EBS gp3 provisioned) $1,480.00 / mo (EC2 i3en + S3 cold tier) Included in CKU base fees
Total Monthly Running Cost $3,450 + $2,000 Ops engineering = $5,450/mo $1,480 + $500 Ops engineering = $1,980/mo $4,800 – $6,200 / month

Production-Hardened Redpanda Docker Compose with Prometheus Telemetry

Deploying Redpanda provides complete Kafka API compatibility without JVM overhead, ZooKeeper, or KRaft controller orchestration complexity. Below is a production-hardened Docker Compose specification configuring thread-per-core pinning, memory lock, and tiered storage to AWS S3:

version: '3.8'

services:
  redpanda-broker:
    image: docker.redpanda.com/redpandadata/redpanda:v24.2.5
    container_name: redpanda-prod-node1
    network_mode: host
    restart: always
    command:
      - redpanda
      - start
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr internal://10.0.1.15:9092,external://lb.zelobiz.com:19092
      - --pandaproxy-addr 0.0.0.0:8082
      - --schema-registry-addr 0.0.0.0:8081
      - --rpc-addr 0.0.0.0:33145
      - --advertise-rpc-addr 10.0.1.15:33145
      - --mode production
      - --default-log-segment-size 134217728 # 128 MB segments
      - --smp 4                              # Pin to 4 CPU cores
      - --memory 16G                         # Allocate 16 GB dedicated RAM
      - --reserve-memory 2G                  # OS kernel reserve
      - --overprovisioned 0
      - --lock-memory true
    environment:
      - REDPANDA_ENVIRONMENT=production
    volumes:
      - /mnt/nvme0n1/redpanda:/var/lib/redpanda/data
    ulimits:
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1

  redpanda-console:
    image: docker.redpanda.com/redpandadata/console:v2.7.2
    container_name: redpanda-web-console
    restart: always
    ports:
      - "8080:8080"
    environment:
      - KAFKA_BROKERS=127.0.0.1:9092
      - KAFKA_SCHEMAREGISTRY_ENABLED=true
      - KAFKA_SCHEMAREGISTRY_URLS=http://127.0.0.1:8081

Kafka OS Kernel & Disk IOPS Tuning Cheatsheet

To prevent broker disk write stalls when running Apache Kafka on Linux, tune virtual memory dirty page ratios and I/O schedulers in /etc/sysctl.conf:

  • Lower Dirty Background Ratio: Set vm.dirty_background_ratio = 5 and vm.dirty_ratio = 10. This forces the Linux kernel flusher threads to continuously write dirty pages to NVMe/SSD storage rather than buffering large bursts that freeze I/O for seconds.
  • Disable OS Swapping: Set vm.swappiness = 1. Paging JVM heap memory or broker caches to swap disk induces instantaneous latency spikes exceeding 1,000ms.
  • Mount Filesystems with noatime: When mounting broker data directories on XFS or ext4, always specify noatime,nodiratime,barrier=0 to eliminate disk write metadata overhead on every read operation.
  • Enforce Disk Queue Depth with NVMe: For AWS EBS gp3 volumes, provision a minimum of 3,000 IOPS and 250 MB/s throughput per broker to survive simultaneous partition rebalances and peak ingestion bursts.

Frequently Asked Questions: Kafka & Redpanda Hardware Sizing

Why does Apache Kafka recommend 3-way replication (RF=3) as the production standard?

A replication factor of 3 paired with min.insync.replicas=2 guarantees zero data loss (RPO=0) and high availability even when one broker suffers hardware failure while a second undergoes rolling upgrades. If RF=2 were used, a single node loss leaves the partition with zero redundancy, risking immediate downtime.

How does Redpanda achieve 10x lower latency than Apache Kafka?

Redpanda is written in C++ using the Seastar asynchronous framework. It implements a thread-per-core shared-nothing architecture that bypasses the Linux PageCache using direct I/O (O_DIRECT) to write directly to NVMe storage. This completely eliminates JVM garbage collection pauses, memory cache context switching, and thread locking contention.

How much RAM does a Kafka broker actually require?

Kafka brokers typically need only 6 GB to 8 GB of JVM heap memory (for internal state and metadata). However, the underlying physical node should have 32 GB to 64 GB of RAM because Kafka utilizes all remaining unallocated memory as OS PageCache to serve consumers with zero-copy disk reads.

What is Kafka Tiered Storage and when should it be activated?

Tiered Storage separates streaming compute from long-term data retention. Active, warm partitions remain on fast local NVMe/EBS disks, while completed log segments are automatically offloaded to inexpensive object storage (Amazon S3, Google Cloud Storage). This allows organizations to retain event streams for months or years without provisioning petabytes of expensive block storage.