Chapter 2. Part 1. Testing And The Software Life Cycle Flashcards
What is acceptance testing?
A test level that focuses on determining whether to accept the system.
What is black-box testing?
Testing based on an analysis of the specification of the component or system.
Synonyms: specification-based testing.
What is component integration testing?
The integration testing of components.
Synonyms: module integration testing, unit integration testing.
What is component testing?
A test level that focuses on individual hardware or software components.
Synonyms: module testing, unit testing.
What is confirmation testing?
A type of change-related testing performed after fixing a defect to confirm that a failure caused by that defect does not reoccur.
Synonyms: re-testing.
What is functional testing?
Testing performed to evaluate if a component or system satisfies functional requirements.
References: ISO 24765.
What is integration testing?
A test level that focuses on interactions between components or systems.
What is maintenance testing?
Testing the changes to an operational system or the impact of a changed environment to an operational system.
What is nonfunctional testing?
Testing performed to evaluate that a component or system complies with nonfunctional requirements.
What is regression testing?
A type of change-related testing to detect whether defects have been introduced or uncovered in unchanged areas of the software.
What does shift-left mean in software testing?
An approach to performing testing and quality assurance activities as early as possible in the software development life cycle.
What is system integration testing?
The integration testing of systems.
What is system testing?
A test level that focuses on verifying that a system as a whole meets specified requirements.
What is a test level?
A specific instantiation of a test process.
Synonyms: test stage.
What is a test object?
The work product to be tested.
What is a test type?
A group of test activities based on specific test objectives aimed at specific characteristics of a component or system.
After TMap.
What is white-box testing?
Testing based on an analysis of the internal structure of the component or system.
Synonyms: clear-box testing, code-based testing, glass-box testing, logic-coverage testing, logic-driven testing, structural testing, structure-based testing.
SDLC - Software Development Life Cycle
Abstract, high-level representation of the software development process. SDLC model describes the activities included in the software development process and the relationships between them, both logical and temporal. Determines how software will be developed, helps understand the succession of different phases of the manufacturing process
What are examples of classic SDLC model types?
Examples of classic SDLC model types include sequential models, iterative models, and incremental models.
What are sequential models in SDLC?
Sequential models include the waterfall model and the V model.
What are iterative models in SDLC?
Iterative models include Boehm’s spiral model and prototyping.
What are incremental models in SDLC?
Incremental models include the Unified Process.
EXAMPLES OF DETAILED MODELS OF SOFTWARE DEVELOPMENT PROCESS
• Scrum
• Kanban
Lean IT (development based on so-called lean management)
• eXtreme Programming (XP)
• Test-Driven Development (TDD)
• Acceptance Test-Driven Development (ATDD)
• Behavior-Driven Development (BDD)
• Domain-Driven Design (DDD)
• Feature-Driven Development (FDD
HOW THE CHOICE OF THE SDLC MODEL AFFECTS THE TESTING ISSUES:
• Scope and timing of test activities
• Selection and timing of test levels and test types
• Level of detail of the test documentation
• Selection of test techniques and practices
• Scope of test automation
SEQUENTIAL SDLC MODEL CHARACTERISSTICS
- assuming the execution of individual development activities one after another - LINEAR MODEL
- one phase can’t start until the previous is completed
- verification after every phase
- in early phases - TESTER PARTICIPATES IN STATIS TESTING: REQUIREMENT REVIEWS AND TEST DESIGN
INCREMENTAL MODEL OF SDLC
• CYCLICAL MODEL
• PROCESS OF SPECIFYING REQUIREMENTS AND DESIGNING, BUILDING, AND TESTING THE SYSTEM IN PARTS - SOFTWARE FUNCTIONALITY GROWSN INCREMENTALLY
• CHANGES ARE LIMITED TO ONE PART - e.g. new query option, change in GUI
• successive increments
- the increment should get its final form in one stage
• A SINGLE INCREMENT CAN BE CONDUCTED IN A SEQUENTIAL OR ITERATIVE MODEL
ITERATIVE MODEL OF SDLC
• SOFTWARE DEVELOPMENT DONE IN CYCLES
• SPECIFYING, DESIGNING, BUILDING, AND TESTING TOGETHER GROUPS OF FUNCTIONALITY IN A SERIES OF CYCLES, OFTEN OF A STRICT DURATION
• ITERATIONS MAY INCLUDE CHANGES TO FUNCTIONALITIES PRODUCED EARLIER
• EACH ITERATION DELIVERS WORKING SOFTWARE THAT IS AN INCREASING SUBSET OF TOTAL SET OF FUNCTIONALITIES
• EACH ITERATION - BOTH STATIC AND DYNAMIC TESTING CAN BE PERFORMED AT ALL LEVELS
AGILE PRACTICES
• CHANGE CAN OCCUR AT ANY POINT IN THE PROJECT
• IN FAVOUR OF LIGHTWEIGHT DOCUMENTATION AND EXTENSIVE TEST AUTOMATION TO MAKE REGRESSION TESTING
• LOTS OF EXPERIENCE-BASED TESTING
WATERFALL MODEL
• SEQUENTIAL MODEL
• development activities one after the other -
REQUIREMENTS -> DESIGN -> IMPLEMENTATION -> TESTING -> MAINTENANCE
• TESTING LATE IN THE PLAN
• WHEN CAN BE BENEFICIAL:
- in typical „batch projects” ( e.g. when implementation projects require more complex configuration)
- in projects with well identified requirements with no risk of change occurring in them (e.g. requirements resulting directly form legislation)
V - MODEL
1.Req <—————————8.accept t.
2.Specif. <—————— 7.system t.
3.Arch.design<———6.integr. T.
4.Implement<—5.component t
———————————————->time
• modified waterfall - to address lack of testing, implementing early testing
• introduces levels of testing linked to each stage
• all tests levels occur sequentially but sometimes it can overlap
• drawing attention to testing activity that occurs at each phase - not only dynamic test but also during development phase
- at lest branch of this SDLC testers don’t test but they can review the test basis and design related test
* W-version: testing activities in the manufacturing phases should involve static testing, i.e. review of work products (e.g. requirements review, architecture review, code review etc.)
UNIFIED PROCESS (UP) MODEL
• ITERATIVE AND INCREMENTAL MODEL
• FLEXIBLE PROCESS FRAMEWORK
- each individual processes are defined (business modeling, requirements, analysis and design, implementation, testing, deployment
- theses processes can be selected and adapted to the needs of the project
-individual iterations take usually long time, and incremental parts of the system, are correspondingly large
BOEHM’S SPIRAL MODEL
• INCREMENTAL EXPERIMENTAL COMPONENTS ARE CREATED - THEY CAN BE EXTENSIVELY REBUILT OR ABANDONED AT LATER STAGES
• IN THIS MODEL COMPONENTS AND SYSTEMS - OVERLAPPING AND REPEATED TEST LEVELS
- each functionality testes at multiple levels approaching final release
- sometimes: teams use CONTINUOUS DELIVERY/CONTINUOUS DEPLOYMENT approach - with extensive use of automation across multiple levels of testing as a part of the software delivery pipeline
• concept of self-organizing teams
• Barry Boehm - creator - it’s model of models
- it can be configured to obtain any other model of development cycle
•RISK ANALYSIS PERFORMED BEFORE THE START OF EACH SUCCESSIVE DEVELOPMENT CYCLE
PROTOTYPING MODEL
• WHEN DESIGN GOALS ARE NOT STRICTLY DEFINED OR SPECIFIED IN ADVANCE
- e.g. client deformed set of goals but no detailed functional requirements, so developer may be uncertain about the effectiveness of the implemented algorithm
• helps stakeholders better understand what is to be built when requirements are fuzzy
• PLANNING PHASE FOLLOWED BY A SERIES OF RAPID PROTOTYPING ITERATIONS
- EACH ITERATION: representation forgoes aspects of the software that will be visible to end users; ends with the building of a working prototype
• each stage - stakeholders feedback
• ideally PROTOTYPE SERVES AS MECHANISM FOR IDENTIFYING SOFTWARE REQUIREMENTS
- LATER - USING EXISTING PROGRAM OR TOOLS TO ENABLE RAPID GENERATION OF WORKING PROGRAMS (e.g. „dynamic” GUI mock-ups)
METHOD VS FRAMEWORK VS METHODOLOGY
METHOD - set of rules for doing some work
METHODOLOGY - science that studies methods
FRAMEWORK - arrangement and interrelationship of elements that make up a whole
SCRUM
• most common software development methods
• Scrum divides software development into SHORT ITERATIONS OF THE SAME LENGHT (usually 1-4 weeks) and THE ITERATIVE PARTS OF THE SYSTEM ARE CORRESPONDINGLY SMALL - e.g. they include few enhancements or new functionality
• PROBLEM WITH TESTING: short iterations and frequent changes, instead of thorough nad lengthy test case design process -> MORE COMMON EXPLORATORY TESTING
-> EMPHASIS ON EXTENSIVE TEST AUTOMATION (regression testing, number of those grows rapidly with each iteration)
• MIX OF PUSH AND PULL SYSTEM - with each iteration, team selects („push”) from product backlog a predetermined set of tasks to be completed in that iteration , and within a sprint, each team member starts working on the next task when the previous one is finished („pull”
KANBAN
• originally from manufacturing companies like Toyot, where products are developed in series
• DELIVERING ONE ENHANCEMENT OR FUNCTIONALITY AT A TIME (immediately after preparation) OR TO GROUP MORE FUNCTIONALITY FOR SIMULTANEOUS TRANSFER TO THE PRODUCTION ENVIRONMENT
• usage of KANBAN BOARD - to visualise, plan and supervise the development process
• based on PRODUCT CARDS
• organizes the devOp process in away that each production station produces exactly as much as is needed at any given moment
• each workstation has a TOP-DOWN LIMIT OF WORK it can do WITHIN A CERTAIN UNIT OF TIME (WORK IN PROCESS LIMIT = WIP)
- not to produce unnecessary elements
- the „mass” of tasks to be performed „flows” smoothly through the production system to optimize resource consumption and production time
• KANBAN is a „PULL” SYSTEM ->. Worker at a given job „pulls” a task from the previous workstation where a product has already been completed and is ready to be passed on
PRINCIPLES FOR SELECTING SDLC MODEL
• TAILORED TO THE CONTEXT RESULTING FROM THE CHARACTERISTICS OF TEH PROJECT AND PRODUCT
• STUFF TAKEN INTO ACCOUNT:
- PURPOSE OF THE PROJECT
- TYPE OF PRODUCT BEING DEVELOPED
- BUSINESS PRIORITIES (such as time-to-market)
- identified PRODUCT RISKS AND PROJECT RISKS
SELECTING SDLC MODEL AND TESTING - ISSUES
• DEPENDING ON THE PROJECT CONTEXT - COMBINE AND REORGANIZE SOME TEST LEVELS AND/OR TEST ACTIVITIES
- integration of COMMERCIAL OFF-THE-SHELF (COTS) software with a larger system, the buyer may perform interoperability testem at the system intergration level (for integration with infrastructure and other systems) and at the acceptance test level (functional testing and nonfunctional testing along with user acceptance testing and operational accepting testing
• different models can be COMBINED
- V-MODEL for development and testing back-end part AND AGILE MODEL for development and testing of the front-end
TESTING FOR SYSTEMS RELATED TO INTERNET OF THING (IoT):
many different objects (devices, products, and services)
• separate SDLC models to individual parts of system
• more emphasis is placed on the later stages of the SDLC, after objects are already I Beth operation (e.g. the production, upgrade, and decommissioning phases)
PROCESS OF SELECTING SDLC MODEL
- DEFINING THE CRITERIA BY WHICH WE WILL EVALUATE THE SUITABILITY OF EACH MODEL IN OUR PROJECT
- examples of criteria
- size and experience of the team
- size, type and level of complexity of the project
- customer relations, stability of requirements, frequency of their changes
- geographic dispersion of the team
- model’s compatibility with the company’s business/organizational strategy
- examples of criteria
- COMPARING potential SDLC models - adv and disadv
- for this projects, with this team, experience with certain models etc
- ASSESSMENTS OF NEEDS OF OUR ORGANIZATION
- nature and operations of the company team - risks, law issues etc
- APPLICATION OF TEH PREVIOUSLY SELECTED CRITERIA IN THE CONTEXT OF OUR ORGANIZATION’S NEEDS
SDLC AND GOOD TESTING PRACTICES IN EACH AND EVERY OF THEM
- FOR EVERY SOFTWARE DEVELOPMENT ACTIVITY, TEHRE IS A CORRESPONDING TEST ACTIVITY
attention of TEH „totality” of testing activities; every product subjected to quality control - EACH TESTING LEVEL CORRESPONDS TO TESTING OBJECTIVES APPROPRIATE TO THE PHASE OR GROUP OF ACTIVITIES WITHIN THE SOFTWARE DEVELOPMENT CYCLE
- testing objectives may differ significantly
• at one level - mainly interested in detecting as many defects as possible
• different level - more interested in validation that system meets the customer’s requirements and needs
- testing objectives may differ significantly
- TEST ANALYSIS AND TEST DESIGN FOR A GIVEN TEST LEVEL SHOULD BE STARTED ALREADY DURING EXECUTION OF THE CORRESPONDING SOFTWARE DEVELOPMENT ACTIVITY
- principle of early testing ; defect cheaper to fix in early stages
- some defects are found through test design activities not through physical testing of apps
- TESTERS SHOULD PARTICIPATE IN REVIEWS OF WORK PRODUCTS (e.g. requirements, design, or user stories) AS SOON AS DRAFT VERSIONS OF THE RELEVANT DOCUMENTS ARE AVAILABLE
- role of tester in quality control - SHIFT LEFT PRINCIPLE
- during such reviews tester can find a lot of ambiguities, errors or contradictions
TEST-FIRTS APPROACH IN SOFTWARE DEVELOPMENT - 3 METHODS
• TEST-DRIVEN DEVELOPMENT (TDD)
• ACCEPTANCE TEST-DRIVEN DEVELOPMENT (ATDD)
• BEHAVIOUR-DRIVEN DEVELOPMENT
—> BEFORE source code that implements a specific function is written a test for that function is created
• from XP - EXTREME PROGRAMMING, core of agile software development
BENEFITS OF TEST FIRST APPROACH
- TESTING REPLACES THE TRIAL-AND-ERROR METHOD
instead of trial-and-error checking if an implementation function works -> set of meaningful test cases is created with strictly defined input and output values, and the noose is written to pass these test - TEST CASES PROVIDE OBJECTIVE FEEDBACK ON PROGRESS
each test passed - evidence that the project is moving forward - TEST CASES REPLACE SPECIFICATIONS
tests become a collection of control procedures but also DEFINE SAMPLE BEHAVIOUR OF THE TEST OBJECT - „LIVING DOCUMENTATION”
change in requirements requires a change in tests, so it forces a change in documentation, making the documentation relevant at any point in the project - TEST-FIRST APPROACH IMPROVES THE QUALITY OF PUBLIC INTERFACES (APIs)
component testing and component integration use the public methods of the classes under test -since the tests are written before the code is created, the tests define names of the public methods of the class under test, their parameters, and examples of method usage
DESIGN OF TEH TEST BECOMES THE DEFINITION OF THE INTERFACE - TEST-FIRST APPROACH IMPROVES THE TESTABILITY OF THE CODE BEING DEVELOPED
once the tests are implemented - developer must write code that passes them -> code must invoke the interfaces required by the tests
written code must be „compatible” with designed tests
developer - better control over higher level of code coverage with tests
TEST-DRIVEN DEVELOPMENT (TDD) PROCESS
FOCUS ON COMPONENT TESTING AND COMPONENT INTEGRATION TEST LEVELS
1. Dev creates a new test case, corresponding to a specific requirements for the code
2. Existing tests are run - should fail (RED), because no corresponding code yet
if the code passes - the test itself is an issue
• running the remaining tests acts as regression tests
3. Dev writes a minimum amount of code sufficient to pass the new tests acts. All previously written test must also pass (GREEN)
4. Dev refactors the code (if necessary) to keep the quality of the code - many short test-code cycles can degrade the code
rerunning all test tests to make sure the refactoring didn’t break anything
5. All test repeated till there is no code to write
• TDD approach typically with test framework like unit to support automated testing
• through test design alone - DEVS CAN TEST OR ESTABLISH IMPLEMENTATION ASSUMPTIONS FOR A COMPONENT
CODE REFACTORING
In computer programming and software design, code refactoring is the process of restructuring existing source code-changing the factoring-without changing its external behavior. Refactoring is intended to improve the design, structure, and/or implementation of the software (it’s non-functional attributes), while preserving its functionality.
GOALS:
- improving code readability
-reducing complexity
- improving source code’s maintainability
- creating a simpler, cleaner, more expressive internal architecture or object model to improve extensibility
-improving performance
BEHAVIOUR-DRIVEN DEVELOPMENT
• TEST FOCUSED ON EXPECTED BEHAVIOUR OF TEH SYSTEM
SLIGHTLY HIGHER LEVEEL THAN COMPONENT TESTING IN TDD
•collaborative approach to generate acceptance criteria in plain language - part of a user story that can be understood by all stakeholders
• ACCEPTANCE CRITERIA - WRITTEN USING FRAMEWORK
-> typical format
GIVEN/WHEN/THEN
• CAN BE AUTOMATED
• framework based on the user story and associated acceptance criteria, automatically generates test case code and executes it
• test recorded in the form of a readable and understandable text document rotten in a language like GHERKIN
• CONCEPT OF „LIVING DOCUMENTATION”
specification usually follows the principle of SPECIFICATION BY EXAMPLE
ACCEPTANCE TEST-DRIVEN DEVELOPMENT (ATDD)
• same procedures as TDD and BDD but with automated test cases that are at appropriate level for acceptance testing
ACCEPTANCE TEST CASES - derived from acceptance criteria
generated jointly by devs, testers and business reps
ACCEPTANCE CRITERIA written form USER’S PERSPECTIVE in easy-to-understand format
* IMPLEMENTATION SCHEME OF ATDD:
1. Select a user story
2. Write an acceptance test
3. Implement a user story (coding)
4. Run an acceptance test
5. Refactor code, if necessary
6. Get user story approval (sign-off)
• ATDD - automated testing framework
• concept of „living documentation”
DevOps approach
Creating synergy by having development, testing, and operations work together
• based on the ITERATIVE/INCREMENTAL MODEL OF SDLC + AGILE principles (team autonomy, rapid feedback, integrated tool chains) + technical practices like:
Continuous integration (CI)
continuous Delivery (CD) of software
Continuous software deployment
CONTINUOUS INTEGRATION PROCESS
• NEW CODE UPLOADED BY DEVS to code repository is automatically merged, compiled and built; ideally also with component (unit) tests
-> TDD approach for best results
-> automatic component tests - to make sure that the uploaded coded passes tests - part of regression test suites
->also static analysis of the code
Feedback to dev - instantaneous
>maybe component integration tests at that stage
mix of new tests for new functionality + regression testing
MAIN ADVANTAGE OF CI:
provides devs with quick feedback if their code is flawed
CONTINUOUS DELIVERY
•extension of CI
•another level of testing performed on test environment that is representative of the production environment
PARTS OF CONTINUOUS DELIVERY PROCESS MAY BE:
- COMPONENT INTEGRATION TESTS
- SYSTEM TESTS
- NONFUNCTIONAL TESTS (regression)
If all those are passed -> deployment
Team can deploy the code manually or automatically
CONTINUOUS DEPLOYMENT
• step further than continuous delivery
• difference between continuous delivery and deployment - in CONTINUOUS DEPLOYMENT both deployment decision and deployment itself - AUTOMATED
•DEVOps team achieve as much confidence as possible that automated testing will detect any defects that should not escape to the production environment
• orgs go from c. Delivery to continuous deployment when they gain sufficient confidence in the automated deployment pipeline
DevOps PIPELINE
DevOps Pipelines - automates manual activities whenever possible
ideally: CONFIGURATION MANAGEMENT, COMPILATION, SOFTWARE BUILD, DEPLOYMENT, AND TESTING included in a single, automated, repeatable deployment pipeline
SO as much testing as possible MUST BE AUTOMATED. Otherwise slowing down code delivery and potentially allowing defects to seep into production
INFRASTRUCTURE AS CODE (IaC)
Infrastructure as code (laC) is the process of managing and provisioning computer data center resources through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. The IT infrastructure managed by this process comprises both physical equipment, such as bare-metal servers, as well as virtual machines, and associated configuration resources. The definitions may be in a version control system, rather than maintaining the code through manual processes The code in the definition files may use either scripts or declarative definitions, but la more often employs declarative approaches.
DEPLOYMENT PIPELINE
EXAMPLE:
COMMIT SENT TO REPOSITORY -> STATIC ANALYSIS (checking if code is formatted correctly, naming of variables follows the rules, code has low cyclomatic complexity etc.) -> COMPILATION-> COMPONENT TESTING -> CODE MERGED TO TEST BRANCH -> INTEGRATION TESTING -> STAGING -> REGRESSION TESTING -> DEPLOYMENT
• if at any point anything goes wrong process is stopped and devs get their feedback
•ADVANTAGES OF AUTOMATIC DEPLOYMENT PIPELINE supported by the code management (versioning) process:
at any point in the development cycle -> code is compiled, executable, and ready for release
Any change to the code automatically triggers the deployment pipeline and devs receive immediate feedback. If there is a problem coming up - code reverts to the (working ) version from before the changes
DevOps APPROACH BENEFITS FORM TESTING PERSPECTIVE:
- FAST FEEDBACK on the code quality and info about changes adversely affect existing code
- CI promotes shift left approach - devs are encouraged to to submit high-quality code accompanied by component test and static analysis
- DevOps facilitates stable test environments by promoting automated continuous integration and continuous delivery processes
- Increase the view on nonfunctional quality characteristics (e.g. performance, reliability)
- Automation through a delivery pipeline reduces the need for repetitive manual testing
- The risk in regression is minimized due to the scale and range of automated regression tests
RISK AND CHALLENGES OF DevOps
- The DevOps deployment pipeline must be defined, established, and maintained
- Continuous integration tools must be in place and maintained
- Test automation requires additional resources and be difficult to establish and maintain
- Sometimes teams become overly dependent on component testing and perform too little system and acceptance testing
SHIF LEST APPROACH
AD principle „Early testing saves time and money”
Testing early and on very level/part of SDLC
BEST PRACTICES WITH SHIFT-LEFT APPROACH
1. USE OF REVIEWS
can be performed early in developmental cycle, even before code is written to find potential defects in the specification, like ambiguities, incompleteness, inconsistency, superfluous elements, and contradictions
2. USE OF CONTINUOUS INTEGRATION AND CONTINUOUS DELIVERY
rapid feedback on code quality
forces the creation of automated component tests for source code as it’s’s submitted to code repository
3. PERMORMING STATIC ANALYSIS
before dynamic test or as a part of an automated process such as DevOps pipeline
4. PERFORMING NONFUNTIONAL TESTING AS SOON AS POSSIBLE
usually those are done at the end
5. USING MODEL-BASED TESTING
in this approach tester uses or creates a model of the software, and a special tool on the casus of this model automatically creates test cases that achieve a certain coverage criterion. Model can usually be created before the implementation work begins
IN WHAT SDLC MODELS WE CAN SEE SHIFT-LEFT APPROACH
- V-MODEL testing activities taken lace when the corresponding development phase begins
- ITERATIVE MODLES testing in every iteration
- TEST-DRIVEN DEVELOPMENT writing tests before writing code in TDD or ATDD
RETROSPECTIVE MEETINGS - LESSON LEARNED MEETINGS - TOPICS COVERED
- PROCESSES AND ORGANIZATIONS (e.g. did the procedures used delay, hinder or speed up and facilitate the performance of certain tasks)
- PEOPLE (right people for the task? Deficiency in training?)
- RELATIONSHIPS (smooth teamwork? Communication effective)
- TOOLS (are there better options etc.)
- THE PRODUCT THAT WAS DEVELOPED AND TESTES (what can be improved, technical features)
RESULTS SHOULD BE RECORDED AND ADDED TO COMPLETION REPORT
BENEFITS OF RETROSPECTIVES
- INCREASING TEST EFFICIENCY
- INCREASING TEST EFFECTIVENESS
- IMPROVING QUALITY OF TEST CASES
- INCREASING TEST TEAM SATISFACTION
- IMPROVING COLLABORATION BETWEEN TESTERS AND DEVELOPERS
- IMPLEMETNING IMPROVEMENT ACTION ADDED TO NEXT ITERATION BACKLOG - PRODUCT REALTED OT TEAM RELATED
