Summary
When one Jenkins controller is no longer enough, the first step is often not another controller but dedicated agents: a controller that only orchestrates frees its resources for scheduling and stops builds from changing potentially sensitive data in its home directory. Once more controllers are needed, four criteria judge any split: what a single failure stops, whether a change can be followed from commit to production, how access is partitioned, and what each extra controller costs in plugin updates, core upgrades, backups and per-controller roles.
The article compares four splits. By environment gives each environment its own plugins and access, but breaks end-to-end pipelines, duplicates jobs and lets one outage stop every product. By organisation keeps plugins and access per division, but hides cross-division flows and makes security complex. By product line, the favoured option, shows the whole flow on one controller and limits a failure to one product, at the price of a permission scheme that has to be designed. The hybrid groups agents by technology (Java, .NET or Node) under controllers split by one of the other three. An active/passive setup, a load balancer with health checks in front of two controllers sharing the jobs directory, shortens an interruption but does not spread the load.
The leadership point follows Conway’s observation: a CI topology reproduces the organisation that owns it. Splitting by environment mirrors hand-offs between teams, splitting by organisation mirrors the org chart, splitting by product assumes a team owns its product end to end. If nobody owns a product’s whole flow, no Jenkins topology will create that ownership, so the question to settle first is who owns what; the number of controllers follows.
Key ideas
- Many installations that seem short of controllers are short of agents: a controller that builds nothing schedules better and keeps builds away from its own data.
- Every split is judged on four criteria: what one failure stops, whether the whole flow is visible, how access is partitioned, and what each controller costs to run.
- Splitting by product line keeps the flow visible on one controller and limits an outage to one product, at the price of a permission scheme that has to be designed.
- An active/passive setup shortens an interruption but does not spread the load, and failover takes longer the more jobs a controller carries.
- The CI topology mirrors who owns what; deciding who owns each product’s flow comes before deciding how many controllers to run.