Experimentation for Business-to-Business Mission Critical Systems: A Case Study

Question 1

What is continuous experimentation (CE)?

Answer 1

A way to rapidly validate software value and quality.

Question 2

What has CE been primarily been used for?

Answer 2

Software-as-a-Service and web-facing systems. But this paper explores its use in mission-critical systems in a business-to-business (B2B) context.

Question 3

What is the HURRIER process?

Answer 3

A set of generic activities organized in four main areas that can be implemented by both the R&D organization and customers to deliver high-quality solutions that are valued by customers.

Question 4

What three phases does Fitzgerald and Stol’s holistic view of software development activities be dividen into?

Answer 4

Business strategy and planning, development, and operations.

Question 5

For what purpose do CE link strategy and operations with development?

Answer 5

To guide product improvement and innovation through repeated cycles of build, measure, and learn.

Question 6

CE encompasses various techniques beyond randomized controlled experiments; what two groups of goals is it classified into?

Answer 6

business-driven experiments and regression-driven experiments.

Question 7

What does vendors do in the B2B-domain and what challenges do they meet?

Answer 7

Sell products and services to other companies that sell to users, making it challenging to collect user data, feedback, and deploy product improvements.

Question 8

When Rissanen and Münch investigated challenges, benefits, and organizational aspects when introducing CE in the B2B domain, what did they find?

Answer 8

That customers play a major role in experiment design and deployment.

Question 9

In what four main areas can the generic activities that composes the HURRIER process be divided into?

Answer 9

1. the R&D organization
2. the internal validation
3. single customer validation
4. multiple customer validation

Question 10

In HURRIER, what is the R&D organisation resposible for?

Answer 10

The development of the feature or change that is going to be deployed and starts after a development idea is generated.

Question 11

What does the pre-study activity consist of?

Answer 11

Scoping the project and planning its development.

Question 12

What does incremental development activity consist of?

Answer 12

Incremental steps that can be rapidly evaluated in the field in collaboration with the customer.

Question 13

What does the single customer validation activity consist of?

Answer 13

Conducting experiments with one customer.

Question 14

What does the multiple customer validation activity consist of?

Answer 14

Conducting experiments with multiple customers.

Question 15

What does the HURRIER process enable for the R&D organization?

Answer 15

To receive feedback from the customers and minimize the risk imposed on the services provided by the customers.

Question 16

Why is single customer validation done, and what does it involve?

Answer 16

• Because internal validation is not enough to cover all aspects of the system.
• Customer laboratory evaluation provides confidence to the customer to deploy the software.
• Passive launch, restricted launch, and gradual rollout are techniques used to validate the system.
• One customer is usually involved since the beginning in the pre-study phase.

Question 17

Why is multiple customer validation done?

Answer 17

While making a feature for general availability is done when enough coverage and confidence is achieved with a single customer, additional customers may be selected to increase the coverage for more field experiments. And identifying corner cases can increase the feature coverage and the delivered value by choosing different experimental designs.