DOH

Question 1

What are “horse” organisations?

Answer 1

Highly dangerous code releases that are prone to catastrophic failure.

Question 2

Why are lean and devops practices important?

Answer 2

The time required to develop new functionality with lean drop from years to weeks or months. However, without devops, deployment into production would still represent a risk factor and could take a long time.

Organisations able to utilize practices from both areas are able to perform experiments to test business ideas, discovering which ideas create the most value, which are further developed and safely deployed into production.

Competitive advantage relies on fast time to market and relentless experimentation.

Question 3

What are two important goals of IT organisations?

Answer 3

1. Respond to the rapidly changing competitive landscape.
2. Provide stable, reliable and secure services to the customer.

Question 4

What is manufacturing lead time and why is it important in lean?

Answer 4

Lead time is the time required to convert raw materials into finished goods. And it was the best predictor of quality, customer satisfaction, and employee happiness.

One of the best indicators of short lead times is small batch sizes of work.

Question 5

What is dark launching?

Answer 5

A process that allows us to release production-ready software features to a small group of users while hiding them from the rest of the user base prior to a full release.

A dark launch can also include deploying code to a production environment but without exposing it to any production traffic.

Question 6

How is a dark launch implemented?

Answer 6

We start by wrapping the new feature in a feature flag. This decouples the deployment from the release. However, this also increases technical dept if the feature flags are not managed properly.

Question 7

What are feature flags?

Answer 7

Feature flags can turn functionalities on or off in order to safely test in production by separating code deployment from feature release.

A feature flag ranges from a simple IF statement to more complex decision trees, which act upon different variables.

Question 8

What are the difference between dynamic and static feature flags?

Answer 8

Dynamic feature flags can be changed at runtime.
Static feature flags can only be changed with a code change or updated configuration file.

Dynamism refers to how much we need to modify a flag in its lifetime.

Question 9

What are the benefits of feature flags?

Answer 9

1. Feature flags enable continuous delivery - the risk of long-lived branches and dreaded merge hell is reduced.
2. Increase rate of product releases and engineering velocity - Test in production without risks and less buggy releases.
3. Continuous experimentation - frequently validate changes and measure performance based on chosen KPIs.
4. Risk mitigation - releases to a subset of users instead of big-bang releases.

Question 10

What are the challenges of feature flags?

Answer 10

1. Using feature flags can become complex especially with in-house solutions.
2. Messy code and coordination - feature flags need to be cleaned up when they have served their purpose but when and by who?
3. Technical dept - unused feature flags

Question 11

What are the pros and cons of storing feature flags in configuration files?

Answer 11

Its a simple implementation method, a configuration file is usually used to store application settings and some feature flags fall into this category.

However, it requires a redeploy each time and feature flag is activated or deactivated. Additionally, flags cannot be context specific they are either on or off for all users.

The distinction between short- and long-lived flags influence how we go about implementing its toggle point.

Question 12

What are the pros and cons of storing feature flags in a database?

Answer 12

Its convenient since the settings can be updated often and easily. This also means that product teams can change flags and not only developers.

However, when we scale and use more dynamic feature flags then quick solutions fall short, since it requires more complex segmenting and controlled rollouts logic to be implemented in-house.

Question 13

Why is it a good idea to scope each feature flag to a single team?

Answer 13

To avoid overlap in accountability and confusion in ownership. The team is responsible for rollout, monitoring and retiring/cleanup of the flag.

Question 14

Why should all feature flags not be treated the same?

Answer 14

Feature flags can generally have many purposes.
1. Release toggles should be managed by developers
2. A/B testing flags should be managed by product teams.
3. Kill switches should be managed by the ops team.
Etc.

Question 15

What is important to remember when doing percentage rollout?

Answer 15

1. Maintain consistency - a specific user should see the same variation of the feature regardless of the number of times it is evaluated.
2. Do not reach a point of no return - if the flag is still live then do not completely remove the option to go back.

Question 16

What is important to remember when implementing our own feature flag system?

Answer 16

1. Do not make it a single point of failure - the flagging system should not be on the critical path and needs to be graceful degradation path.
2. Feature flag changes should be backward compatible - we should be able to go back to a previous flagging state.
3. Treat feature flagging code as production code - should be clean and tested.
4. Have a plan to remove old feature flags - but keep in mind that different flags have different sunsetting timelines depending on their purpose.

Question 17

What is the definition of a technology value stream?

Answer 17

Typically defined as the process required to convert a business hypothesis into a technology enabled service that delivers value to the customer.

Question 18

What is development lead time?

Answer 18

The time from when the ticket was created to when the solution is in production. This is what the customer is experiencing.

Question 19

Why is a problem to have long deployment lead times?

Answer 19

Heroics are required at almost every stage of the value stream. Everything need to be merged together and work together which can result in bug fixes and investigations that can take days and then have to be tested again.

Question 20

What are the three principles underpinning devops?

Answer 20

1. Flow: fast flow from dev to ops to customer
2. Feedback: fast feedback from customer to devs.
3. Continual learning: high-trust culture that supports a dynamic, disciplined and scientific approach to experimentation and risk-taking.

Question 21

Why is it important for work to be visible?

Answer 21

In order for us to have an overview of where the downstream flow is good and where it is stalled. Otherwise, work might be stalled or bounce between departments without/few people knowing it.

It is also easier to prioritize the work that are available.

Question 22

Why are small batch sizes important for flow?

Answer 22

Large batch sizes can result in a lot of WIP and larger variability in flow.

Question 23

What are the typical system constraints preventing short development lead time?

Answer 23

• Environment creation - creating production and testing environment each time we develop something.
• Code deployment - long deploys
• Test setup - long tests and many required manual tests without automation
• Tight architecture - require many meetings in order to change code because the architecture is non extensible hence teams do not have autonomy to change what they need

Question 24

What describes the second way?

Answer 24

The second way enable fast and constant feedback from right to left at all stages of the value stream.

We make our system of work safer by creating fast, frequent and high-quality information flow throughout the value stream.

Question 25

What is the defining characteristics of a complex system?

Answer 25

It defies any single person’s ability to see the system as whole and understand how all the pieces fit together.

Complex systems typically have a high degree of interconnectedness of tightly coupled components, and system-level behavior cannot be explained in terms of the behavior of the system components.

Another observed characteristic: doing the same thing twice does not necessary result in the same outcome.

Question 26

What conditions should be met in order to safely work on complex systems?

Answer 26

• Complex work is managed so that problems in design and operations are revealed.
• Problems are swarmed and solved, resulting in quick construction of new knowledge.
• New local knowledge is exploited globally throughout the organization.
• Leaders create other leaders who continually grow these types of capabilities.

Question 27

What effect do feedforward and feedback loops have on system components?

Answer 27

Cause components within a system to reinforce or counteract each other.

Question 28

What is the difference between feedback and feedforward loops?

Answer 28

Feedback is in relation to the past. We receive feed that customers use our service less efficiently because of xyz. Feedforward is looking into the future with suggestions on how customers can use the service more efficiently.

Question 29

Why is feedback critical?

Answer 29

It allows us to steer. We must constantly validate between customer needs, out intentions and our implementations.

Question 30

What does it mean to “swarm” problems?

Answer 30

Contain the porblems before they can spread and to diagnose and treat the problems so that they cannot recur. By doing this we build knowledge about how to manage the systems for doing our work, converting ignorance into knowledge.

Question 31

Why is swarming necessary?

Answer 31

• It prevents the problem from progressing downstream, where the cost and effort to repair increases exponentially and technical dept is allowed to accumulate.
• It prevents the work center affected from starting new work, which can introduce new errors in the system.
• If the problem is not addressed it can recur, requiring more fixes and work.

Question 32

Why does adding more inspection steps increase the likelihood of future failures?

Answer 32

The effectiveness of approval processes decreases as we push decision making further away from where the work is performed.

We need everyone in our value stream to find and fix problems in their area of control. We push quality and safety responsibilities and decision-making to where the work is performed.

Question 33

What two types of customers does “lean” define?

Answer 33

• The external customer that pays for the product
• The internal customer who receives and processes the work immediately after us.

Question 34

What is the focus of the thrid way?

Answer 34

Creating a culture of continual learning and experimentation. These are the principles that enable constant creation of individual knowledge, which in then turned into team and organisational knowledge.

Question 35

What three types of organization cultures can we summarize?

Answer 35

• Pathological: large amounts of fear and threat. People hoard information. Failure is hidden.
• Bureaucratic: rules and processes, often to help individual departments maintain their turf. Failure is processed through a system of judgement.
• Generative: actively seeking and sharing information to better enable the achieve its mission. Failure results in reflection and genuine inquiry.

Question 36

How do generative organisations respond to failures?

Answer 36

When accidents and failures occur, instead of looking for human error they look for how we can redesign the system to prevent the accident for happening again.

Question 37

What happens to processes over time in the absence of improvements?

Answer 37

Due to chaos and lack of order processes degrade over time.

“Even more important than daily work is the improvement of daily work” - Mike Orzen, Lean IT.

Question 38

Why is tension important?

Answer 38

Lower-performing organizations buffer themselves from disruptions and might even increase WIP in order to ensure that work centers do not go idle.

High-performing organizations introduce tension to increase performance and improve resilience (antifragility). Instead of fearing idle work centers they take the opportunity to experiment to improve performance and safety.

Question 39

Why is top-down leadership not effective?

Answer 39

Traditional leaders were expected to make the correct decisions. There is significant evidence that greatness is not achieved by leaders making the correct decisions - instead leaders should create conditions so their team can discover greatness in their daily work.

Top-down transformations are possible, but they require highest level of management and susitained focus on the outcome.

Question 40

What does the ‘greenfield vs brownfield’ software categorization mean?

Answer 40

Term originally used for urban planning and building projects. Greenfield is underdeveloped land and brownfield is developing on contaminated land.

Greenfield projects are often pilots to demonstrate feasibility.

Brownfield are existing software serving actual customers and come with technical dept.

Question 41

What are the ideal phases used by change agents to build and expand their coalition and base of support?

Answer 41

1. Find innovators and early adopters: focus on teams who actually want to help. Ideally, these are also people with influence.
2. Build critical mass and silent majority
3. Identitfy the holdouts: high profile influential detractors who are most likely to resist.

Question 42

Why is it important to identify all members of a value stream when selecting a service for a devops initiative?

Answer 42

In value streams of any complexity, no one person knows all the work that must be performed in order to create value for the customer. We need to understand HOW value is being created in order to know if any areas in our value stream jeopardizing our goals of fast flow, short lead times and reliable customer outcomes.

Question 43

What is a process block?

Answer 43

When documenting value stream activities at the high level, we can use process blocks in a diagram. They should include: lead time (received task -> customer received outcome), process time (how long it took to complete the task) and %C/A (% Complete and/or Accurate) as measured by the downstream customers of the output.

Question 44

Why should planning horizons and iteration intervals be short for devops teams?

Answer 44

• Flexibility to reprioritize and replan quickly
• Decrease the delay between work expended and improvement realized.
• Faster learning generated from the first iteration
• Less risk that the project is killed before we can generate outcomes.

Question 45

What are non-functional requirements (NFRs)?

Answer 45

Sometimes referred to as the “ilities”: maintainability, scalability, reliability, testability, deployability and security.

Question 46

What is Conway’s Law?

Answer 46

Organizations whcih design systems… are constrained to produce designs which are copies of the communication structures of these organizations.

If you have four groups working on a compiler, you will get a 4-pass compiler.

Question 47

How many primary organizational structures is described by Dr. Roberto Fernandez?

Answer 47

Three:
- Functional-oriented organisations
- Matrix-oriented organisations
- Market-oriented organisations

Question 48

What is a functional-oriented organisation?

Answer 48

Optimise for expertise, division of labor and reducing cost. Often have a tall hierarchical organisational structure. These organisations centralise expertice: fx server admins, db admins, network admins etc.

Question 49

What is a matrix-oriented organisation?

Answer 49

Attempt to combine functional and market orientation. Often result in complicated organisational structures, and sometimes achieve neither benefits of the functional or the market oriented organsation structure.

Question 50

What is a market-oriented organisation?

Answer 50

Optimise for responding quickly to customer needs. Tend to be flat, composed of multiple cross-functional disciplines.

Question 51

Can we reorganize ourselves to continous improvement and adaptiveness?

Answer 51

NO! What is decisive is not the form of the organisation, but how people act and react. It is the habits and capabilities in people and the workforce that lead to succes.

But structure and culture helps (personally)

Question 52

What is one of the largest challenges as organisations grow?

Answer 52

Maintaining effective communication and coordination between people and teams. Often when people and teams reside on different floors, buildings etc. creating and maintaining a shared understanding and mutual trust becomes more difficult, impeding effective collaboration.

Question 53

What is a bounded context and why are they important?

Answer 53

A bounded context is simply the boundary within a domain where a particular domain model applies.

Bounded contexts ensure that services are compartmentalized and have well-defined interfaces, which also enables easier testing.

Observation former engineering director for google app engine: organisations with service-oriented architectures have incredible flexibility and scalability. Many developers in small teams can still be incredibly productive.

Question 54

What are the factors in the twelve-factor methodology?

Answer 54

1. Codebase: One codebase tracked in revision control, many deploys.
2. Dependencies: Explicitly declare and isolate dependencies.
3. Config: Store config in the environment.
4. Backing services: Treat backing services as attached resources.
5. Build, release, run: Strictly separate build and run stages.
6. Porcesses: Execute the app as one or more stateless processes.
7. Port binding: Export services via port binding.
8. Concurrency: Scale out via the process model.
9. Disposability: Maximize robustness with fast startup and graceful shutdown.
10. Dev/prod parity: Deep development, staging and production as similar as possible.
11. Logs: Treat logs as event streams.
12. Admin processes: Run admin/management tasks as one-off processes.

Question 55

Why is the one codebase, one application principle in the 12-factor app important?

Answer 55

Important to maintain a good lead-time.

The single codebase for an application is used to produce any number of immutable releases that are destined for different environments.

Following this particular discipline forces teams to analyze the seams of their application and potentially identify monoliths that should be split off into microservices.

Multiple applications within a single codebase can get ugly for a number of reasons. Conway’s law states that the organization of a team will eventually be reflected in the architecture of the product that team builds. Dysfunction, poor organization, and lack of discipline among teams usually results in the same dysfunction or lack of discipline in the code.

One codebase, one application does not mean you’re not allowed to share code across multiple applications; it just means that the shared code is yet another codebase.

Question 56

Why are dependency management important in the 12-factor app?

Answer 56

One benefit of explicit dependency declaration is that it simplifies setup for developers new to the app. The new developer can check out the app’s codebase onto their development machine, requiring only the language runtime and dependency manager installed as prerequisites. They will be able to set up everything needed to run the app’s code with a deterministic build command.

Twelve-factor apps also do not rely on the implicit existence of any system tools. If the app needs to shell out to a system tool, that tool should be vendored into the app.

Question 57

What is a config and why should it be stored in the environment?

Answer 57

An app’s config is everything that is likely to change between deploys (staging, production, developer environments, etc.). Including: resource handlers, credentials and per-deploy values such as canonical hostname for the deploy.

The twelve-factor app requires strict separation of config from code. Configs varies substantially across deploys, code does not.

The twelve-factor app stores config in environment variables. Unlike config files, there is little chance of them being checked into the code repo accidently. And they are language- and OS-agnostic.

In a twelve-factor app, env vars are granular controls, each fully orthogonal to other env vars. This is a model that scales up smoothly as the app naturally expands into more deploys over its lifetime (dev, test, prod, staging, qa, demo, etc.).

Question 58

What is a cononical name (CNAME)?

Answer 58

Its a properly denoted host name of computers or a network server. CNAME is a record in the DNS database that indicates the true host name of a computer associated with its aliases.

A machine’s host name could be “barkley.example.com”, but because it runs the web and FTP services for the domain “example.com”, it could have canonical names of “www.example.com” and “ftp.example.com”.

Question 59

What is a backing service?

Answer 59

Any service the app consumes over the network as part of its normal operation: datastores such as MySql, messaging/queueing systems, SMTP services for outbound email and caching systems such as memcached.

Question 60

Why should backing services be treated as attached resources?

Answer 60

By accessing backing services as attached resources via a URL or other locator/credentials in the config. We can swap backing services in the config without changing the app’s gode.

Question 61

How is a codebase transformed into a (non-development) deploy?

Answer 61

Through three stages:
1. Build: fetches dependencies and compiles binaries and assets.
2. Release: takes the build and combines it with the deploy’s current config.
3. Run: runs the app in the execution environment by launching a set of the app’s processes against a selected release.

Question 62

Why should an app be executed as one or more stateless processes?

Answer 62

Tweleve-factor processes are stateless and share nothing. Any data that needs to persist must be stored in a stateful backing service, typically a database, or fx memcache or redis (fx for user session with time-expiration).

When a process is stateless, instances can be added and removed to address a particular load burden at a given point in time. Since each process operates independently, statelessness prevents unintended side effects.

Question 63

Why should services be exported using port binding?

Answer 63

The reasoning is that domain names and associated IP addresses can be assigned on-the-fly by manual manipulation and automated service discovery mechanisms. Thus, using them as a point of reference is unreliable. However, exposing a service or application to the network according to port number is more reliable and easier to manage.

Question 64

Why should we scale out via the process model?

Answer 64

In the twelve-factor app, processes are a first class citizen. The process model shines when it comes time to scale out. The share-nothing, horizontally partitionable nature of twelve-factor app processes means that adding more concurrency is a simple and reliable operation.

Question 65

Why should apps be disposable (able to start and stop at a moments notice)?

Answer 65

Facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.

Question 66

Why should apps be disposable (able to start and stop at a moments notice)?

Answer 66

Facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.

In terms of shutdown, disposability advocates ensuring that all database connections and other network resources are terminated properly and that all shutdown activity is logged

Question 67

Why should we keep development, staging and production as similar as possible?

Answer 67

The twelve-factor app is designed for continuous deployment by keeping the gap between development and production small. Looking at the three gaps described above:
- Make the time gap small: a developer may write code and have it deployed hours or even just minutes later.
- Make the personnel gap small: developers who wrote code are closely involved in deploying it and watching its behavior in production.
- Make the tools gap small: keep development and production as similar as possible.

Question 68

Why should logs be treated as event streams?

Answer 68

The Logs principle advocates sending log data in a stream that a variety of interested consumers can access. The process for routing log data needs to be separate from processing log data.

Question 69

Why should admin/management tasks be ran as one-off processes?

Answer 69

The principle of Admin Processes states that admin processes are first-class citizens in the software development lifecycle and need to be treated as such.

Question 70

What are the overall aspects included in the concept continous delivery?

Answer 70

Automated deployment pipeline, automated tests that ensure we are in a deployable state, having developers trunk daily, and architecture environments and code to enable low-risk releases.

Question 71

Why should infrastructure be easier to rebuild than to repair?

Answer 71

• We can quickly rebuild if something goes wrong instead of repairing.
• Repeatable environment creation systems, allows easier increase in capacity (horizontal scaling).
• Easier to restore in case of catastrophic failure (instead of having to replicate years of undocumented work).

Question 72

What is immutable infrastructure?

Answer 72

We create new virtual machines or containers from the automated build mechanism and deploy them into production, destroying the old ones or taking them out of rotation.

Manual changes to the production environment are no longer allowed - the only way changes can be made is to put the changes into version control and re-create the code and environments from scratch.

Question 73

What did Gary Gruver observe about automated testing and price?

Answer 73

Without automated testing, the more code we write the more time and money is required to test our code - in most cases this is a totally unscalable business model.

Question 74

Why are automated build and test processes that run in dedicated environments necessary for a working deployment pipeline?

Answer 74

• The build and test processes can run all the time, independent of work habits.
• A segregated build and test process ensures we understand all the dependencies required to build, package, run and test our code (no it ran on my PC situations).
• We can package the application to enable repeatable installation of code and configurations into an environment.
• Environments can be made more production-like in a consistent and repeatable way (compiler versions, debugging flags etc. are removed).

Question 75

What are the three capabilities required for continuous delivery?

Answer 75

• Comprehensive and reliable set of automated tests that validate we are in a deployable state.
• A culture that stops the production line when validation tests fail.
• Developers working in small batches on trunk rather than long-lived feature branches.