Reviews and Evaluating Testing Methods Flashcards
What constitutes a review?
Qualitative evaluation of correctness based on informal techniques.
ex. Checklist
Define Analysis
provides repeatable and detailed evidence of correctness.
Do Reviews or Analysis involve running code?
No
Peer Review
Evaluation conducted by one or more people with similar competencies and expertise to the author.
Why is Peer review useful?
- People most fit to assess expert’s work is other experts.
- Large and diverse group finds more weaknesses.
- Provides external validation (stamp of approval)
Limitations of peer review?
- Slower than checking own work.
- Asking people to point out errors in your work is uncomfortable.
- Encourages Opportunism (let reviewers find errors and how to fix) and Perfectionism.
Software Review
Evaluation of software elements or project status to identify discrepancies from planned results and to recommend improvement.
Name types of Review
-inspection
-team review
-walkthrough
-pair programming
-peer check (desk check)
-ad hoc review
(from formal to least formal)
Ad hoc review
Focus is on problem solving (“help me figure out this bug for 5 minutes”)
Peer desk check/passaround
desk check is one person, passaround many. Author only sees output.
(“check out this pull request please?”)
Pair programming
Observer reviews each line of code as it is written.
improves quality, more a development strategy.
Walkthrough
Formal meeting where developers will attend and a demo of the code is shown.
Goal is to evaluate software or educate an audience about a piece of software. Author guides participants through code and explains how it works.
Inspection
Systematically identify defects with program. Reader presents software to team and outcome is (accept, accept with re-work, re-inspect).
- formal, systematic and rigorous review.
- Carried out by inspectors trained to find common defects using checklists and analysis.
- May be used for conformance to regulations or certifications.
- Can be used to collect metrics on defect rates.
Checklist for code
Reliability: fault tolerant? effective exception handling?
Efficiency: how much memory consumed? Optimized algorithms?
Reusability: Can components be used in other places?
Scalability: Can system grow?
Why independence in review?
Political view? commercial incentives
Psychological view? reviewing own work is difficult.
Technical view? independence builds redundancy
Systems view? review creates feedback loop.
Types of independence
Financial, organizational, task independence, knowledge independence (a, b make different assumptions.)
Trade off of independence
- Financial (expensive),
- system knowledge (you’re an outsider.)
- Relationship (seen as an outsider.)
Review in support of design?
not usually good.
big loss developer is more suited to understand failures.
design and verification out of sync.
verfication doesn’t influence design.
When to use walkthrough, review and inspection
W: for training.
R: for consensus.
I: improve quality of document.
Error seeding
Early example of fault injection.
plant bugs and see how many are caught.
Mutation testing.
Assesses ability of your test set to distinguish between original program and minor small changes (mutants). More caught mutants the better.
Common mutations
Operand replacement
Statement modification operators (while-> repeat until)
Expression modification operators.
Types of mutants
Do Fewer: Mutant sampling, constrained mutation, mutant clustering.
Do Faster: Schema based mutation, separate compilation approach. Runtime optimization
Do Smarter: Human error vs completeness.
Limitations of mutation testing
too expensive. still lack of safety critical systems support.
Benefits of mutation Testing
Increasing developments towards automation.
