Kafka Consumer Groups, Explained the Way I Wish I'd Been Taught

Partitions are just ordered logs

Once I stopped picturing Kafka as some abstract event bus and started picturing each partition as a plain append-only log with an offset pointer, the rest of the model clicked into place.

A consumer group is just a set of consumers agreeing on who reads which partitions. Rebalancing is what happens when that agreement needs to change — a consumer joins, leaves, or dies. It’s coordination, not magic.

Where teams get burned

Most incidents I’ve seen trace back to under-provisioned partitions or consumers that take too long per message, triggering rebalances mid-processing. Both are solvable with a bit of headroom and honest capacity planning.