Testing Flashcards
What is Testing?
a. Validating a new or improved process, particularly when there is some level of risk involved.
Why is testing necessary?
- To establish effective performance of a product or service
- To ensure the product or service doesn’t result in unintended consequences
- To verify and validate the product or service meets our customer requirements
What factors affect the level of testing needed the most?
Solution complexity and risk to the business
What types of testing do we use most frequently in OpEx?
Proof of Concept (A “prototype” to show a concept is viable but, will need more work to deploy and sustain. This is often referred to as a “Feasibility” test)
Beta Testing (Test of a solution known to have unresolved issues and often improved through Evolutionary Design.)
Pilot (A limited deployment to determine the overall effectiveness of a solution)
What are some basic deliverables of the Testing skillset? (ADD DISCUSSION QUESTION FOR EACH ONE)
a. Site selection
b. Metrics Development
c. Data Collection Plan
d. Exit Criteria
e. Training (testing team and end-user, if applicable)
f. Report Out
What 3 factors have the most influence on which test method to select?
Number of tests/experiments previously run
Maturity of the solution (is it ready for end user testing?)
Level of risk associated with deploying the solution everywhere
What do we check for in Proof of Concept testing?
*Performance
*Unintended Consequences
What do we check for in Beta testing?
*Performance
*Unintended Consequences
*Process Management Elements
What do we check for in Pilot testing?
*Performance
*Unintended Consequences
*Process Management Elements
*Behaviors (CAP Plan)
*Deployment Approach
What are common ways testing may fail?
*When CTQs are not achieved
*When extensive unintended consequences are discovered
*When cost of solution deployment/sustainment is greater than the benefit
What are some elements the Data Collection Plan should identify?
*Sampling plan (who is collecting the data, what data is being collected, when, where and how?)
*Check-in/callibration (early review of data collection output to confirm approach and make adjustments if necessary)
*Data analysis plan (what will be analyzed, what will be reported)
What is most critical when selecting sites to include in a pilot?
*Select locations that are representative of the population that is expected to implement the solution (e.g., different sizes, geographic areas, and various levels of staff)
*Include control sites
When Piloting it is most critical that you?
Pilot as close as possible to real life deployment
What are some common challenges when Piloting?
*Local Leadership Understanding & Buy-In
*Communication
