Chapter 2.4 - Testability Flashcards
What is software testing, and how does it differ from debugging?
Software testing is the process of finding input values to check against software to reveal failures (mismatch between actual and intended output).
Debugging is the process of finding and fixing a fault in the code after a failure is identified.
What are some challenges in software testing?
There are multiple different kinds of system that need to be tested
What are some challenges in Testing GUIs ?
dynamic elements, multiple environments
What are some challenges in embedded systems ?
smartcards, hardware dependencies
What are some challenges in non-deterministic systems?
same input may yield different outputs
What is the difference between software testability and software testing?
Software testing assesses software quality by checking correctness against expected results.
Software testability describes how easy it is to design, execute, and analyze tests, which reduces testing effort and costs.
What are key properties of software that affect testability?
Observability – Can we see the internal state of the system?
Controllability – Can we easily set up test conditions?
Isolateability – Can we test components independently?
Complexity/Simplicity – How understandable and modular is the code?
What measures determine how easily a system reveals its faults?
Effort required to find faults.
Probability of a fault being revealed.
Time taken to perform tests.
Complexity of test dependencies.
Risk of remaining undetected errors.
What are the four conditions necessary for a failure to be revealed in testing?
Reachability – The fault must be executed.
Infection – Execution of the fault must corrupt program state.
Propagation – Corrupted state must lead to a visible failure.
Revealability – The test oracle must detect the failure.
How does Test-Driven Development (TDD) improve software testability?
Test-Driven Development (TDD): Write failing tests first, then implement code to pass them.
How does Design for Testability (DFT) improve software testability?
Design for Testability (DFT): Add test interfaces and controlled system states.
How does Record & Playback improve software testability?
Record & Playback: Capture system states for reproducible testing.
Limiting Complexity: Reduce dependencies and non-determinism.
How does Limiting Complexity improve software testability?
Limiting Complexity: Reduce dependencies and non-determinism.
How can software testability be assessed?
Domain/Range Ratio (DRR): The ratio of input values to output values—higher DRR means lower testability.
Mutation Testing: Injecting defects and observing how often they cause failures.
Implicit Losses: When different inputs yield the same output, reducing test effectiveness.
What are the tradeoffs involved in improving testability?
High controllability improves testing but may break encapsulation.
Increased observability helps debugging but may expose security risks.
High modularity simplifies unit testing but may require extensive integration testing.
What are some techniques to improve software testability?
Test-Driven Development (TDD): Write failing tests first, then implement code to pass them.
Design for Testability (DFT): Add test interfaces and controlled system states.
Record & Playback: Capture system states for reproducible testing.
Limiting Complexity: Reduce dependencies and non-determinism.
What are the steps of TDD?
Write a failing test – Test fails because no code exists.
Make the test pass – Implement the minimal code.
Refactor – Improve code while keeping the test passing.
What is the software testing pyramid?
Unit Tests (Fast, cheap, most numerous).
Integration Tests (Verify component interactions).
End-to-End Tests (Slowest, most expensive, least frequent).
Why do microservices favor integration tests over unit tests?
Individual microservices depend on interactions, making unit tests less effective.
Integration tests ensure correct communication between services.
Excessive system-level tests are impractical for large architectures.
What is Continuous Integration (CI), and how does it support testing?
Developers frequently integrate code into a shared repository.
Automated builds and tests ensure changes don’t break the system.
Helps detect integration errors early.
Why should software engineers prioritize testability in software design?
Reduces testing effort and cost (Easier to design, execute, and analyze tests).
Increases fault detection (Better observability and controllability).
Speeds up debugging (Faster identification and resolution of defects).
Improves maintainability (Easier refactoring and adaptation to new requirements).
How does information hiding impact testability?
Encapsulation improves modularity but can limit testability.
Too much information hiding makes it difficult to observe internal state for debugging.
Well-designed APIs provide controlled access for testing while maintaining security.
What are implicit and explicit losses in software testability?
Implicit Losses:
When many different inputs map to the same output. Example: Integer division (8/3 = 2, 9/3 = 3) loses information compared to real division.
Explicit Losses:
When internal states are not observable. Example: A function modifies internal variables but does not expose them externally.
How do software requirements affect testability?
Clear, detailed specifications improve testability by defining expected behavior.
Ambiguous or incomplete requirements make it harder to define test cases.
Formal specifications (e.g., contracts, pre/post-conditions) improve automated testing.
What are some unique challenges in testing embedded systems?
Limited observability (No direct access to internal state).
Hardware dependencies (Requires specialized tools to test).
Real-time constraints (Must ensure timing correctness).
State persistence (E.g., smartcards retain state after power loss, affecting reproducibility).
What is mutation testing, and how does it assess test quality?
Defects (mutations) are deliberately injected into the program.
A strong test suite should detect these changes.
If the test suite fails to catch mutations, it indicates poor fault detection.
Helps measure test coverage quality, not just quantity.
How does chaos engineering relate to software testability?
Introduces random failures to test system resilience.
Example: Netflix’s Chaos Monkey randomly shuts down services.
Helps identify weaknesses that are difficult to catch with traditional testing.
What are the key challenges in testing cloud-based systems?
State is managed externally (Difficult to control test conditions).
Distributed nature increases complexity.
Snapshotting & restoring state can help mitigate some challenges.
Network variability makes performance testing unpredictable.
What are some ways to measure software testability?
Code complexity (Cyclomatic complexity, coupling, cohesion).
Observability (How easily internal states can be inspected).
Controllability (How easily test cases can set up conditions).
Defect detection probability (How likely is a test to find a fault?).
Effort required for test case development (Time & cost).
What is the difference between unit and integration testing?
Unit Testing:
Tests individual components in isolation. Faster, more focused, detects low-level bugs.
Integration Testing:
Tests interactions between components. Catches defects in API communication and data flow.
Why is testing non-deterministic software difficult?
Same input may produce different outputs due to randomness or concurrency.
Example: A multi-threaded program may behave differently in each run.
Solutions:
Use mocking to control inputs.
Implement deterministic logging for debugging.
Use property-based testing instead of fixed inputs.
How does strong encapsulation impact software testability?
Pros:
Improves modularity and security.
Reduces unintended side effects.
Cons:
Harder to access internal states for testing.
May require reflection, test hooks, or dependency injection.
How does testability differ between OO and functional programming?
OO Languages (Java, C#):
Testability depends on class design, dependency injection, and encapsulation.
Dynamic binding can make debugging harder.
Functional Languages (Haskell, Scala, Elixir):
Pure functions improve testability.
Less reliance on shared state reduces side effects.
Higher-order functions allow flexible test scenarios.
How does logging improve testability?
Provides observability into program execution.
Helps track failures and unexpected behavior.
Useful in non-deterministic systems where reproduction is difficult.
Good logging follows structured formats (JSON, XML) for analysis.
What is the difference between black-box and white-box testability?
Black-Box Testability:
Tests based on external behavior.
Independent of internal implementation.
Harder to diagnose specific failures.
White-Box Testability:
Tests internal logic (e.g., branch, path, mutation testing).
Requires code access and knowledge.
More effective in identifying edge cases and security flaws.
Why is testability difficult in AI/ML systems?
Non-determinism (Model outputs vary based on training data).
Lack of clear expected results (Difficult to define a test oracle).
Performance-driven correctness (Evaluated with accuracy metrics, not exact outputs).
Continuous learning (Model changes over time, making regression testing hard).
