Module 8 - Test and Evaluation Flashcards
What technical processes in SE V model is T&E involved in?
Verification - (Development Testing; Did we build it right?)
Validation - (Operational Testing; Did we build the right thing?)
Three reasons T&E activities are part og the developemnt process
Provide engineers and decisionmakers with knowledge to assit in managing risks
Measure technical progress
Characterize operational effectiveness, operational suitability, interoperability, survivability, and lethality
Two key players in DoD T&E
Director of Operational Test and Evaluation (DOT&E)
Director, Development Test, Evaluation and Assessment (D(DTE&A))
Director, Development Test, Evaluation and Assessment (D(DTE&A)) Functions
Provide indepenet program risk assessment to MDA
Ensure acquisition decisions supported with the right T&E info
Ensure each acquisition program is supported by an appropriate DT&E startegy
Participate in T&E activities
Director of Operational Test and Evaluation (DOT&E) Functions
Review and update T&E planning documents
Ensure adequacy of T&E on major weapon systems
Report T&E results to OSD and Congress
Independently assess systems under DOT&E oversight
Participate in T&E IPT activities
Development Test and Evaluation (DT&E)
Measure technical performance against design specs in controlled environment
Operational Test and Evaluation (OT&E)
Determines operational effectiveness and suitability defined in CDD
Development Test and Evaluation (DT&E) Key Assessment Activities
Verify achievement of Critical Technical Parameters (CTPs) and the ability to achieve Key Performance Parameters (KPPs)
Assess the system’s ability to achieve the thresholds prescribed in the capabilities documents
Validate system functionality
Assess system specification compliance
Report on program progress to plan for reliability growth and to assess reliability and maintainability performance for use during key reviews
Assess system safety
Assess compatibility with legacy systems
Support cybersecurity assessments and authorization
Assess entry criteria for Initial Operational Test and Evaluation (IOT&E) and Follow-On Operational Test and Evaluation (FOT&E)
Assess the maturity of the chosen integrated technologies
Development Test and Evaluation (DT&E) Key Data Activities
Provide data to the Program Manager to enable root cause determination and to identify corrective actions
Provide information for cost, performance, and schedule tradeoffs
Identify system capabilities, limitations, and deficiencies
Stress the system within the intended operationally relevant mission environment
Support the interoperability certification process
Document achievement of contractual technical performance, and verify incremental improvements and system corrective actions
Provide DT&E data to validate parameters in models and simulations
Development Test and Evaluation (DT&E) Performance Factors
Accuracy
Maneuverabillity
Interoperability
Material Strength
Reliability and Maintainability
Software Integration
Software Functionality
Development Test and Evaluation (DT&E) Enviromental Factors
Temperature
Pressure
Vibration
Shock
Humidity
Sea salt spray
Rain, snow, ice, lightning
Production Acceptance Test and Evaluation (PAT&E)
Used to verify each production unit meets contract reuqirments
Conducted at the contractor’s facility (can be supported by DCMA)
Live Fire Test and Evaluation (LFT&E)
Evaluated in both DT and OT
Required for certain ACAT I / II programs or modifications impacting vulnerability and lethality
Mandated by Congress in Title 10 USC 2366; reported prior to a FRPD Review
Early Operational Assessments (EOA)
Conducted prior to Milestone (MS) B during TMRR or after Milestone B early in EMD
Conducted to support a formal MS B decision, Preliminary Design Review
Operational Assessment (OA)
Typically conducted during EMD with production representative articles
Assess the system’s potential to meet capability needs
Support Milestone C
Support a Low-Rate Initial Production (LRIP) decision
Initial Operational Test and Evaluation (IOT&E)
Conducted on production or production representative articles to support a Full-Rate Production (FRP) decision review
Follow-On Operational Test and Evaluation (FOT&E)
Conducted after the Full-Rate Production Decision Review, and may continue throughout the duration of the life cycle.
Confirm correction of deficiencies noted in IOT&E
Refine tactics and doctrine for the new system
Evaluate major modifications and/or changes to the system
Test and Evaluation Master Plan (TEMP) must
DoDI 5000.89 Para 3.4.a.(1)
Integrate T&E with the overall Acquisition Strategy (AS)
Reflect the user’s capability needs and describe how these capability needs will be tested in DT&E, LFT&E and OT&E
Document the T&E program for the entire life cycle
Specify personnel, funding, and test range support requirements
Be developed prior to Milestone A Review and updated before each subsequent Program Decision Review
Unique TEMP Terms: Critical Operational Issues (COIs)
Top-level issues that must be examined in OT&E to determine the system’s capability to perform its mission.
Two Categories of Critical Operational Issues (COIs)
- Effectiveness: Will the system detect the threat in a combat environment at adequate range to allow successful engagement?
- Suitability: Will the system be safe to operate in a combat environment?
Unique TEMP Terms: Critical Technical Parameters (CTPs)
Engineering design factors that a system must meet or exceed to ensure that established performance thresholds are achieved
CTP is stated as a specific numerical value and may change as the system matures during development
Example is system weight
Unique TEMP Terms: Measures of Effectiveness (MOEs) and Measures of Suitability (MOSs)
Measures are derived from the Critical Operational Issues (COI) developed in Operational Test and Evaluation plans and planning processes.
These measures (MOEs and MOSs) are used to determine the attainment of the top-level performance parameters.
MOE Example: maximum effective range
MOS Example: Transportability
T&E during MSA in support of Milestone A
Formation of Working Level Integrated Product Teams (WIPT)
Creation of the TEMP summproting T&E through TMRR< EMD, and PD phases
Provide EOT results to inform. CDD Validation, Development Request for Proposal Release Decision (DRFPRD) and Milestone B
T&E During EMD in support of Milestone B
Integrated testing
DT on components, subsystems, and fully integarted articles
Operational Assesments
Systems modeling, simulations
Updating TEMPO
T&E During after Milestone C
IOT&E
LFT&E if applicable
Provide eval of operational effectiveness and suitability to support FRPDR
Plan follow on FOT&E
