Module 8 - Test and Evaluation

Question 1

What technical processes in SE V model is T&E involved in?

Answer 1

Verification - (Development Testing; Did we build it right?)

Validation - (Operational Testing; Did we build the right thing?)

Question 2

Three reasons T&E activities are part og the developemnt process

Answer 2

Provide engineers and decisionmakers with knowledge to assit in managing risks

Measure technical progress

Characterize operational effectiveness, operational suitability, interoperability, survivability, and lethality

Question 3

Two key players in DoD T&E

Answer 3

Director of Operational Test and Evaluation (DOT&E)

Director, Development Test, Evaluation and Assessment (D(DTE&A))

Question 4

Director, Development Test, Evaluation and Assessment (D(DTE&A)) Functions

Answer 4

Provide indepenet program risk assessment to MDA

Ensure acquisition decisions supported with the right T&E info

Ensure each acquisition program is supported by an appropriate DT&E startegy

Participate in T&E activities

Question 5

Director of Operational Test and Evaluation (DOT&E) Functions

Answer 5

Review and update T&E planning documents

Ensure adequacy of T&E on major weapon systems

Report T&E results to OSD and Congress

Independently assess systems under DOT&E oversight

Participate in T&E IPT activities

Question 6

Development Test and Evaluation (DT&E)

Answer 6

Measure technical performance against design specs in controlled environment

Question 7

Operational Test and Evaluation (OT&E)

Answer 7

Determines operational effectiveness and suitability defined in CDD

Question 8

Development Test and Evaluation (DT&E) Key Assessment Activities

Answer 8

Verify achievement of Critical Technical Parameters (CTPs) and the ability to achieve Key Performance Parameters (KPPs)

Assess the system’s ability to achieve the thresholds prescribed in the capabilities documents

Validate system functionality

Assess system specification compliance

Report on program progress to plan for reliability growth and to assess reliability and maintainability performance for use during key reviews

Assess system safety

Assess compatibility with legacy systems

Support cybersecurity assessments and authorization

Assess entry criteria for Initial Operational Test and Evaluation (IOT&E) and Follow-On Operational Test and Evaluation (FOT&E)

Assess the maturity of the chosen integrated technologies

Question 9

Development Test and Evaluation (DT&E) Key Data Activities

Answer 9

Provide data to the Program Manager to enable root cause determination and to identify corrective actions

Provide information for cost, performance, and schedule tradeoffs

Identify system capabilities, limitations, and deficiencies

Stress the system within the intended operationally relevant mission environment

Support the interoperability certification process

Document achievement of contractual technical performance, and verify incremental improvements and system corrective actions

Provide DT&E data to validate parameters in models and simulations

Question 10

Development Test and Evaluation (DT&E) Performance Factors

Answer 10

Accuracy

Maneuverabillity

Interoperability

Material Strength

Reliability and Maintainability

Software Integration

Software Functionality

Question 11

Development Test and Evaluation (DT&E) Enviromental Factors

Answer 11

Temperature

Pressure

Vibration

Shock

Humidity

Sea salt spray

Rain, snow, ice, lightning

Question 12

Production Acceptance Test and Evaluation (PAT&E)

Answer 12

Used to verify each production unit meets contract reuqirments

Conducted at the contractor’s facility (can be supported by DCMA)

Question 13

Live Fire Test and Evaluation (LFT&E)

Answer 13

Evaluated in both DT and OT

Required for certain ACAT I / II programs or modifications impacting vulnerability and lethality

Mandated by Congress in Title 10 USC 2366; reported prior to a FRPD Review

Question 14

Early Operational Assessments (EOA)

Answer 14

Conducted prior to Milestone (MS) B during TMRR or after Milestone B early in EMD

Conducted to support a formal MS B decision, Preliminary Design Review

Question 15

Operational Assessment (OA)

Answer 15

Typically conducted during EMD with production representative articles

Assess the system’s potential to meet capability needs

Support Milestone C

Support a Low-Rate Initial Production (LRIP) decision

Question 16

Initial Operational Test and Evaluation (IOT&E)

Answer 16

Conducted on production or production representative articles to support a Full-Rate Production (FRP) decision review

Question 17

Follow-On Operational Test and Evaluation (FOT&E)

Answer 17

Conducted after the Full-Rate Production Decision Review, and may continue throughout the duration of the life cycle.

Confirm correction of deficiencies noted in IOT&E

Refine tactics and doctrine for the new system

Evaluate major modifications and/or changes to the system

Question 18

Test and Evaluation Master Plan (TEMP) must

Answer 18

DoDI 5000.89 Para 3.4.a.(1)

Integrate T&E with the overall Acquisition Strategy (AS)

Reflect the user’s capability needs and describe how these capability needs will be tested in DT&E, LFT&E and OT&E

Document the T&E program for the entire life cycle

Specify personnel, funding, and test range support requirements

Be developed prior to Milestone A Review and updated before each subsequent Program Decision Review

Question 19

Unique TEMP Terms: Critical Operational Issues (COIs)

Answer 19

Top-level issues that must be examined in OT&E to determine the system’s capability to perform its mission.

Question 20

Two Categories of Critical Operational Issues (COIs)

Answer 20

• Effectiveness: Will the system detect the threat in a combat environment at adequate range to allow successful engagement?
• Suitability: Will the system be safe to operate in a combat environment?

Question 21

Unique TEMP Terms: Critical Technical Parameters (CTPs)

Answer 21

Engineering design factors that a system must meet or exceed to ensure that established performance thresholds are achieved

CTP is stated as a specific numerical value and may change as the system matures during development

Example is system weight

Question 22

Unique TEMP Terms: Measures of Effectiveness (MOEs) and Measures of Suitability (MOSs)

Answer 22

Measures are derived from the Critical Operational Issues (COI) developed in Operational Test and Evaluation plans and planning processes.

These measures (MOEs and MOSs) are used to determine the attainment of the top-level performance parameters.

MOE Example: maximum effective range

MOS Example: Transportability

Question 23

T&E during MSA in support of Milestone A

Answer 23

Formation of Working Level Integrated Product Teams (WIPT)

Creation of the TEMP summproting T&E through TMRR< EMD, and PD phases

Provide EOT results to inform. CDD Validation, Development Request for Proposal Release Decision (DRFPRD) and Milestone B

Question 24

T&E During EMD in support of Milestone B

Answer 24

Integrated testing

DT on components, subsystems, and fully integarted articles

Operational Assesments

Systems modeling, simulations

Updating TEMPO

Question 25

T&E During after Milestone C

Answer 25

IOT&E

LFT&E if applicable

Provide eval of operational effectiveness and suitability to support FRPDR

Plan follow on FOT&E