Delivering at Pace: CentOS Upgrade

Question 1

Situation

Answer 1

• We needed to upgrade the operating system our instances run on.
• This had been attempted previously, but without clear communication channels between stakeholders and the team, resulting in a loss of trust and disagreement about technical approach.
• Work was stopped due to other priorities.

Question 2

Task

Answer 2

• Create an shared infrastructure instance running on newer version of CentOS
• Learn from previous experience to improve stakeholder relationship

Question 3

Action - first steps?

Answer 3

• First, I spoke with developers, users, and stakeholders involved in the first piece of work to better understand the context.

Question 4

Action - how did you begin your approach?

Answer 4

• I advocated to reduce the scope: identify the core libraries required to build the foundation of a system.
• Pick one of our simplest services to run as an experiment.

Question 5

Action - what was the second step in your approach?

Answer 5

• I communicated with stakeholders early and often.
• I created a way to communicate our approach and progress to users. Created CentOS 7 Line on externally facing whiteboard.
• Invited or pulled stakeholders in to relevant meetings so they could feel like they were part of the process. • Encouraged stakeholders to pair or mob with us when they could.

Question 6

Action - what was the third step in your approach?

Answer 6

• Used what I’d read about checklists in surgeries in The Power of Habit and roadmaps in Switch to developed standardised approach for upgrading the packages.
• Roadmap optimised for pairing continuity.
• Each stop on the CentOS line was a checklist

Question 7

Action - what was the final step in your approach?

Answer 7

• I worked with my team to build this in the open, following GDS’ example.
• I wanted to decouple our code from the data behind it and use developing in the open as an inspiration to better inform our design decisions

Question 8

Result

Answer 8

• An open-sourced base centos puppet configuration built and TDD’d using PDK.
• Healthier stakeholder relationship.
• Confident team who had the tools to approach more OS upgrades

Question 9

Reflection

Answer 9

• Feedback loop between library upgrades was slow.

* I worked on developing a docker container to smoke test any future upgrades in the open source project.