Ideas
One Kafka platform, not one cluster per project
A Kafka cluster per project looks like autonomy and costs like a debt: one shared offer, run by one central team, with every topic declared as code
When a project needs Kafka, the fastest route is to give it a cluster. Then the next project gets its own, and the organisation ends up with as many clusters as projects, each configured by whoever was there when it was installed.
The assessment I delivered this year began with two sentences.
A Kafka cluster for every project is not an affordable solution, and administering a cluster takes a very high level of expertise
A rare skill spread across every project ends up belonging to no one. What replaces those clusters is an offer. One central team runs them: it keeps the rules up to date, shares them and maintains the platform. Projects no longer worry about infrastructure, and they get standard-compliant security and high availability they would not have built alone. Non-production, performance and production are separate environments, and adding or removing a broker affects no project.
Topics live as code. One git repository holds every topic of every environment, with its partitions, retention, access rights and schema; a CI/CD pipeline validates and applies each request, and the same repository serves as the central catalogue of what exists. Any other change to a cluster goes through a support ticket. Teams can see the monitoring of every environment, and starter projects and upskilling sessions bring good practices into the projects.
The objection is autonomy. But a project rarely chose its replication factor for good reasons: it took a default. What it truly has to decide, its topics, its keys, its retention, it still decides, in a file it can read and have reviewed.
Centralise what takes expertise, leave to the projects what takes business knowledge