Choosing a Testing Approach in a Complex World Flashcards
What happened during the 2003 Northeast Blackout?
Previously unknown fault causes power network alarms to be silent.
Unprocessed events were queued up in alarm system
- system fails under load
- backup alarm server starts and fails for same reason (does not refresh queue.)
What are lessons learned of the 2003 Northeast Blackout?
Race condition caused infinite loop in alarm program.
Very hard to trigger and trace. Took 8 weeks to do so.
Lesson:
We need better testing, we already had “more” testing, but it would not have likely uncovered the fault had there been more.
What happened during the “Flash” crash?
DJI fell 600 points. Biggest one day decline thus far. Entirely reversed by the end of the trading day.
High frequency trading algorithms followed pre-set rules. They start buying and selling a particular contract in huge amounts after a mutual fund dumped them. This exhausts ordinary buyers and the price spirals as the HFTs keep buying and selling.
Stop logic occured for 5 seconds to pause trading, price rallies.
What are lessons learned of the “Flash” crash?
It is super hard to track and unveil faults that can be caused in situations where you have an:
- emergent and ever-changing event and environment.
- you have multiple independent software agents that influence this event and environment.
What happened during the Therac-25 incidents?
A fault occurred when a user would type too quickly. Which delivers a strong beam and does not filter out the radiation. Causes massive overdoses.
This was a fatal accident in some cases.
What are lessons learned from the Therac-25 incidents?
We NEED to get better at testing!
Also, maybe have more physical fail-safes.
Why do people often don’t test well?
Testing activities get pushed to the end. Where they may be dropped entirely.
Benchmarks look great, issues don’t
Bad news is squashed by management. Perhaps before a release.
What are some definite non-solutions.
Becoming a testing zealot. You will piss people off.
Becoming a one technique fanatic.
These will definitely:
- Lead you to bad decisions.
- Destroy your credibility and the reputation of valuable techniques.
What can you do to be a better tester?
Know a wider range of techniques.
Know many ways to think of those techniques.
Be flexible.
What is a better model for testing?
Testers Coverage Potential problems Activities Evaluation
Who does the testing?
In-house team
Realistic End users
Subject matter experts?
…
How do you do the tests (activities)
Regression Manually scripted Exploratory Installation Load Testing Long sequence
How can you learn more about testing documentation standards and other standards
Should work top down: find what is specifically relevant to you and then work in more details.
Why can your testing “advocacy” fall short?
Commercial incentives are too strong! Time-to-Market first.
Quality improvement might cost you market share. Especially if customers can’t see your higher quality.
Subject of this is called testing economics or quality economics.
What can make a difference in your advocacy?
Having a relevant standard would help
It can make your case for testing, but you still need to understand it. Sometimes standards really are just bloat.
Standards are good for good testers.
