You may be thinking of sharing a Jenkins server, perhaps you’re worried about:
- Who has the time to do admin on multiple servers?
- How can we be sure that we don’t open a gaping chasm (it happens) on our network making our code and test environment public?
- Ensuring people use a standard toolset
- Avoiding the cost of multiple copies of the same services
Why would you not want to have a shared Jenkins server, then…?
Sharing Jenkins Between Different Development Groups
Don’t do it!
Seriously!
Aaaaagh!
Each team has its own needs and it’s hard to get a set up that’s all things to all men.
Can you Share Jenkins?
Yes, you can, and it’s possible to do it across diverse groups, but it gets harder the more diverse they are, and it gets self-defeating if you do not provide the resource for central admins to keep the server working.
The Irony of the Shared Jenkins
The inclination with a big shared box is to take a conservative approach to installations, updates and access rights. Trying to reduce the amount of change on the box will eventually make it useless.
You need instead to embrace change. Accept that all but the craziest of incremental updates, patches, and plugins SHOULD be installed by default, and that any fallout from incremental change can be managed trivially.
If you only make huge changes after deep testing, the test cycles and resistance to change grow nonlinearly. If you limit everyone’s access to things, then you create a centralised demand on the people who do have that access and they get sucked into everyone’s projects.
How to Manage a Shared Jenkins
- Never say No always say How?
- Run patches and updates weekly – a small impact on a couple of builds is easier to resolve, than risking breaking tons of things with an occasional big bang update
- Provide delegated admin rights for as much as you can so teams can sort themselves out
- Treat any problems with the box as an emergency – break the builds and you break the world
- Provide enough centralised support with a very lightweight process to enable problems to be resolved
- Accept that builds can be messy and that there should be regular clean-up jobs to keep the server working – diskspace, docker images, restarts even
- Ensure the very best connectivity to external services – i.e. getting GitHub integration working properly is better than working around the lack of webhooks with polling
- Document things that can’t be discovered – the Jenkins ecosystem is oblique enough… if you’ve got things installed that people can use, then have a page where you advertise them for those who don’t have admin rights to go looking for them, for example, which tool installations are available
- Share tips and tricks – while people can work out how to use the particular breed of plugins and services surrounding your Jenkins, why not share the HOWTOs around? Encourage sharing.
- Back the whole thing up – be protected against both outages and fatal outbreaks of stupidity where someone wishes they could go back to how their build was a few days ago.
Of Course You Could Just Not Share
Most of the above techniques would be required by the seniors on a team that owned its own Jenkins… just sayin… and if you, as a central DevOps type, decided you didn’t want the headache of doing this things for a multitude of different teams, then you could always:
- Set the standard for how Jenkins is built
- Perhaps provide a Docker template or some other deployment script to create Jenkins instances
- Create security scanning to ensure all Jenkins instances are safe and updated
- Encourage cross-team sharing of HOW-TOs and tips
- Sit back and let everyone do their own projects
Centralisation vs Federation
Two dimensions:
- How impactful are the decisions – company wide or local?
- How frequently are the decisions made – several times a day or on a longer timescale?
Example 1: a team decides which meeting room to use for an ad-hoc meeting – impact to company almost zero, frequency of decision, very often – delegate the admin of that to the team.
Example 2: a team decides which Software Vendor to use for a new Global IT System – impact to company – potentially a huge cost and a huge impact, frequency of decision – very infrequent – should be a decision made at high level.
Example 3: a team decides to buy more staples – impact to company, small fry, frequency of decision, ad-hoc… probably delegate… except… if EVERYONE’S ALWAYS BUYING STAPLES… then it turns out to have a potential economy of scale IF IT’S A BIG ENOUGH PERCENTAGE OF THE COMPANY’S OVERHEADS. In other words, in this middle-ground situations you have to do a global optimisation.
As far as the Jenkins discussion is concerned, the regulation of security and safety is probably a central piece of accountability, but all the day to day use of the server is really ripe for federation/delegation. Finding a hybrid strategy for this is hard for these reasons…
However, if work grinds to a halt because of a lack of centralised decision making/execution, then the operating model is hugely wasteful and needs reconsideration.
Published on Java Code Geeks with permission by Ashley Frieze, partner at our JCG program. See the original article here: The Perils of the Shared Jenkins and Making Decisions Opinions expressed by Java Code Geeks contributors are their own. |
