Apache Kafka Tutorial for Beginners: Topics, Producers, and Consumers

Overview

Apache Kafka is a distributed event streaming platform used for real-time data pipelines, log aggregation, and event-driven microservices. This tutorial explains the core concepts and walks through producing and consuming messages in practice.

What Kafka Is and Is Not

Kafka is	Kafka is not
A distributed commit log	A traditional message queue with per-message acknowledgment
Built for high throughput	A database (though it can retain data for a long time)
Persistent by default	An in-memory-only cache like Redis

Core Concepts

Term	Description
Topic	A named stream of records, like a table in a database
Partition	An ordered, append-only log; topics are split into partitions for parallelism
Offset	A unique sequential ID for each record within a partition
Producer	Client that publishes records to topics
Consumer	Client that reads records from topics
Consumer group	Set of consumers that share the work of reading partitions
Broker	A single Kafka server

Running Kafka Locally

The fastest way to start is with Docker Compose using the KRaft mode (no ZooKeeper required):

services:
  kafka:
    image: apache/kafka:3.8.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker compose up -d

The official Kafka distribution is available from the Apache Kafka downloads page if you prefer a native install.

Topic Management

# Create a topic with 3 partitions and replication factor 1
kafka-topics.sh --create \
  --topic orders \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

# List topics
kafka-topics.sh --list --bootstrap-server localhost:9092

# Describe a topic
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Producing Messages

kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

# Then type messages, one per line:
{"orderId":1,"amount":100}
{"orderId":2,"amount":250}

Send messages with keys to control which partition they land in:

kafka-console-producer.sh \
  --topic orders \
  --property "parse.key=true" \
  --property "key.separator=:" \
  --bootstrap-server localhost:9092

# customer1:{"orderId":1}
# customer2:{"orderId":2}

Records with the same key always go to the same partition, preserving order for that key.

Consuming Messages

kafka-console-consumer.sh \
  --topic orders \
  --from-beginning \
  --bootstrap-server localhost:9092

To consume as part of a group (so offsets are committed):

kafka-console-consumer.sh \
  --topic orders \
  --group order-processors \
  --bootstrap-server localhost:9092

Producer in Python

pip install kafka-python

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers="localhost:9092",
    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
    key_serializer=lambda k: k.encode("utf-8"),
)

for i in range(5):
    producer.send(
        "orders",
        key=f"customer-{i % 2}",
        value={"orderId": i, "amount": 100 + i * 10},
    )

producer.flush()
producer.close()

Consumer in Python

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    "orders",
    bootstrap_servers="localhost:9092",
    group_id="order-processors",
    auto_offset_reset="earliest",
    value_deserializer=lambda v: json.loads(v.decode("utf-8")),
)

for message in consumer:
    print(message.partition, message.offset, message.value)

Consumer Groups and Rebalancing

A consumer group allows multiple consumer instances to share the load. Each partition is assigned to exactly one consumer in the group. If a consumer joins or leaves, Kafka triggers a rebalance.

Partitions	Consumers	Result
3	1	One consumer reads all 3 partitions
3	3	One partition each
3	5	Two consumers idle

Retention and Durability

# Keep messages for 7 days
retention.ms=604800000

# Compact the topic so only the latest value per key is kept
cleanup.policy=compact

Log compaction is useful for topics that store the latest state of an entity, such as user profile updates.

When to Use Kafka

Event sourcing and event-driven microservices
Real-time analytics and stream processing
Log and metrics aggregation
Change data capture (CDC) from databases

For simple task queues, RabbitMQ or Redis Streams may be a better fit. Kafka shines when you need high throughput and durable replay.