EXAM2 (Develop the TEMP and Test Planning, Best Manufacturing Practices, Scheduling, Request for Proposal, Work Breakdown Structure, Leadership and Ethics, Systems Engineering in Acquisition, Systems Engineering Process Model, Technical Management Processes, Risk Management) Flashcards
Understand the purpose of T&E and why we test
The purpose of test and evaluation (T&E) in a development and acquisition program is to help reduce or manage risks in:
Development
Acquisition
Utilization
We test to measure specific individual performance factors.
Recognize the customers of T&E
Developers / Acquisition Managers
Designers and Engineers
End-Users
Recognize the various test agencies
Director, Operational Test and Evaluation (DOT&E)
Army Test and Evaluation Command (ATEC)
Army Evaluation Center (AEC)
Operational Test Command (OTC)
Recognize the types of DT&E and OT&E
Types of OT&E (Operational Testing)
Early Operational Assessment (EOA)
Operational Assessment (OA)
Initial Operational Test and Evaluation (IOT&E)
Follow-on Operational Test and Evaluation (FOT&E)
Types of DT&E (Developmental Test and Evaluation)
Research Effort/Test
Technical Feasibility Test (TFT)
Engineering Development Test (EDT)
Production Qualification Test (PQT)
Production Verification Test (PVT)
First Article Test (FAT)
Production Acceptance Test and Evaluation (PAT&E)
Define test and evaluation
Test: Program, procedure or process to obtain, verify, or provide data to determine the degree to which a system or systems meet, exceed or fail to meet stated objectives
Obtains raw data (1’s and 0’s)
Measures specific, individual performance factors
Is resource intensive
Evaluation: Independent process to review, analyze, and assess data obtained from testing or other sources
Produces analyzed information from test data, modeling and simulation, or other sources
Draws conclusions by looking at how the factors interact
Is intellectually intensive
Define operational effectiveness and operational suitability
Operational Effectiveness: How well it does the job?
Operational Suitability: How well it can be supported?
Understand the purpose of Live Fire Testing and when it applies
Purpose of Live Fire Testing:
Determines crew survivability and/or system vulnerability
Confirms lethality of munitions/missiles against intended target set
Mandated by FY87 Defense Authorization Bill
Applies to:
ACAT I and II major vehicles and weapons platforms
Conventional weapons that provide some degree of protection
Major munitions greater than 1M rounds
Note - Can be waived - LFT can be waived with justification
Prior to program initiation
Impractical or too expense
Must still accomplish an alternative
To determine vulnerability
Survivability, and lethality
Define Interoperability and JITC’s (Joint Interoperability Test Command) role
Interoperability - The ability of systems, units, or forces to provide data, information, materiel, and services to and accept the same from other systems, units, or forces and to use the data, information, materiel, and services so exchanged to enable them to operate effectively together.
JITC issues a Interoperability Test Certification indicating: System has successfully passed interoperability testing and has met the NR-KPP
Testing is performed on Low Rate Initial Production (LRIP) production representative systems
Recognize the importance of T&E to milestone decisions
Before Milestone A: Approved T&E Strategy
Before Milestone B: Approved TEMP; Identification of LRIP quantities – Only 10% max without waiver; DT&E Report
Before Milestone C: Updated TEMP; Final DT&E Report; Operational Assessment
Define Verification and Validation
Verification that the system meets the performance specifications accomplished through Developmental Testing. (Did you build it right?)
Validation that the system fulfills the user’s required capabilities accomplished through Operational Testing. (Did you build the right thing?)
Define concurrent testing, combined testing, and integrated testing
Concurrent Testing- Circumstances in which developmental testing and operational testing take place at the same time as two parallel, but separate and distinct, activities.
Combined Testing- Single test program conducted to support developmental test and operational test objectives.
Integrated Testing- The collaborative planning and collaborative execution of test phases and events to provide shared data in support of independent analysis, evaluation, and reporting by all stakeholders, particularly the development (both contractor and government) and operational test and evaluation communities.
Identify the forum in which the operational tester brings together all the representatives of agencies involved in the Operational or Force Development Test to determine overall readiness for test
T&E WIPT
Understand the purpose of the T&E WIPT and when it is formed
The T&E WIPT is a defined forum serving the PM, and PMO, as the test and evaluation subject matter experts responsible for supporting the PM, and other program WIPTs, on all aspects of a program’s test and evaluation effort. This effort includes T&E program strategy, design, development, oversight, and the analysis, assessment and reporting of test results.
The T&E WIPT should be established and chartered as early as possible during the Material Solutions Analysis phase so that it is involved in program strategy discussions and plans.
Recognize why independent agencies conduct and oversee OT&E
Recognize why independent agencies conduct and oversee OT&E
Recognize T&E planning and its function
As the essential feedback mechanism for the Systems Engineering Process
Understand that the Test and Evaluation Master Plan (TEMP)
Must be updated periodically to ensure that it stays current and integrates the various disciplines as the program evolves through its life cycle
Understand that Developmental Test and Evaluation (DT&E) is
Essential in determining a system’s readiness for Initial Operational Test and Evaluation (IOT&E)
Understand that Operational Test and Evaluation (OT&E)
Helps determine system operational suitability and effectiveness, and it addresses Critical Operational Issues (COIs) that are defined by the User
The Logistics Demonstration is normally conducted
On those tasks performed at field level and some sustainment level by User representatives
Understand that the Test and Evaluation Master Plan (TEMP)
Is prepared by the Program Manager (PM) in conjunction with principal Test Integration Working Group (TIWG) members
NDIs (Non-developmental item) are
Any previously developed items regardless of the source of development-used exclusively for governmental purposes by federal, state, local, or allied governments
The DoD Acquisition Guidebook states that
Producibility is the design effort priority.
Manufacturing integrated into the design process results in
Reduced program risk
Factors leading to increased cost and production delays
Changing design requirements
Late design releases
Lack of production planning
Unstable funding
!The three steps in Production and Manufacturing Integration include!
!Influence the Design Process
Prepare for Production
Execute the Manufacturing Plan!
A priority for manufacturing is
To develop a stable production process
Process Proofing involves
Proving that the design can actually produce a product that meets customer functional requirements in a realistic, “factory floor” type situation
!The 5 Ms of Manufacturing!
!Material
Method
Machinery
Manpower
Measurement!
!Five top-level design goals for producible products!
!Design for Ease of Fabrication
Design for Ease of Assemble
Design for Multiuse
Minimize Number of Parts
Maximize Number of Common Parts!
!The cost of quality can be broken down into three separate areas!
!Prevention
Appraisal
Correcting Failures!
Contractor must have
A quality management process, but the contractor defines the process (must meet Government objectives)
Learning curve theory states
That as the production of an item doubles, the man-hours needed to produce that item decrease at a fixed rate
!Two main principles of Lean Manufacturing!
!Minimization of Waste
Responsive to Change!
The five major causes of production problems
Unstable Rates and Quantities
Design Instability
Undue emphasis on Schedule
Inadequate Configuration Management System
Inattention to Environmental Impacts
The Integrated Master Plan (IMP) is
An event-based plan that identifies the work that must be accomplished to complete the key program activities and events.
An effective schedule correctly balances the tradeoffs between cost, schedule and performance while taking into account risk and program impact
The 5 step scheduling process includes:
Identify the tasks
Sequence the tasks
Estimate the task duration
Construct the schedule
Implement the schedule
Scheduling is
A listing of activities and events organized by time.
Milestones
Points in time where decision makers evaluate the status of the program and determine if the program should proceed to the next phase
What phases are?
Phases: logical means of translating broad capability needs into well defined system specific requirement ultimately into operationally effective, suitable, and survivable systems.
Types of Schedules
Gantt
Milestone
Network
Production
Milestone Charts
Depicts information about significant project events, focusing on specific points in time
The critical path is
The longest path through the network (the path with the greatest sum of task time duration). Any delay in tasks along the critical path will result in a delay in the overall project time
Time constraints are
Limitations imposed by the length of time, or dates, associated with the schedule
Lead is
Allows an activity to start before the predecessor activity is complete.
A lead allows an acceleration of a successor activity
Lag is
A lag directs a delay in a successor activity
Intermediate schedules tie (or roll up)
Is the detailed schedules to the key events and milestones in the master schedule(s), and provide traceability between the two levels
Given task durations, minimum durations, and cost to reduce the duration, be able to evaluate which tasks should be reduced in order to crash the schedule
Given task durations, minimum durations, and cost to reduce the duration, be able to evaluate which tasks should be reduced in order to crash the schedule
Recognize the roles of the Program Manager and the Contracting Officer
Program Manager: Responsible for bringing the program in on time and within budget
Contracting Officer: Responsible for advising the PM and ensuring that all acquisition laws and regulations are followed
Understand the difference between fixed-price and cost-reimbursement contracts
Fixed-Price: Contractor promises to deliver on time and to meet contract specifications for a negotiated price
Cost-Reimbursement: Contractor promises to put forth their best effort
!Understand that contracts greater than…!
!$3,000 but less than or equal to $150,000 are set-aside exclusively for small businesses, if at least two capable small businesses can be expected to submit offers!
Understand that DoD guidance allows contractors to..
Use the quality assurance process of their choice, as long as it meets program objectives
Identify the three generic classifications of DoD software-intensive systems
Weapons Systems
Command, Control, Communications, Computers and Intelligence, Surveillance and Reconnaissance (C4ISR)
Defense Business Systems (DBS)
Recognize software intensive system acquisition best practices
Identify and manage risk continuously throughout the life of the system
Estimate cost and schedule empirically
Use metrics to monitor risk, identify problems, and base decisions
Track earned value
Establish quality targets and track defects against those targets
Treat people as your most important resource
Implement a sound configuration management process
Manage and trace requirements to lowest level
Use system-based software design to document/evaluate process
Ensure data and database interoperability
Identify the four categories of data rights
Unlimited Rights
Limited Rights
Government Purposed Rights
Restricted Rights
Define Work Breakdown Structure
Product-oriented family tree
Composed of hardware, software, services, data, and facilities
Created from systems engineering process throughout the R&D efforts
Allows acquisition personnel to manage risk at levels lower than system level
Displays and defines the product to be developed and produced
The WBS is an output of the Systems Engineering Process
Understand that there are 11 WBS categories and 10 common elements (Level 2)
WBS Categories:
Aircraft
Electronic/Automated Software
Missile
Ordnance
Ship
Space
Surface Vehicle
Unmanned Air Vehicle
Unmanned Maritime Systems
Launch Vehicle Systems
Automated Information Systems
Common Elements:
Integration, assembly, test, and checkout efforts
Systems engineering/program management
System test and evaluation
Training
Data
Peculiar support equipment
Common support equipment
Operational/site activation
Industrial facilities (construction, conversion or expansion)
Initial spares and repair parts
Understand that the WBS is used for
Planning logistics
Establishing schedules
Tracking cost
Predicting variations
A Functional WBS answers the question
What it must do?
A Product WBS answers the question
What must it look like?
Identify the two types of WBS
Functional WBS
Product WBS
Identify the two types of Product WBS
Program WBS:
Developed by the Government (Program Management Office)
Entire WBS for the program - includes everything
Consists of first three levels
Contract WBS:
Developed by the Contractor (4th level – “n”)
Complete WBS for the contract
Contains lowest reporting level agreed upon in the contract
Understand that the Requirements Analysis application of Systems Engineering includes:
Analyze functions
Decompose higher level functions to lower level functions
Allocate performance requirements to the functions
Understand that the Overall System WBS Level is
Level 1
Identify who is responsible for producing the Program and Contract WBS’s and the levels associated with each.
Program WBS:
Developed by the Government (Program Management Office)
Entire WBS for the program - includes everything
Consists of first three levels
Contract WBS:
Developed by the Contractor (4th level – “n”)
Complete WBS for the contract
Contains lowest reporting level agreed upon in the contract
Understand that Level 2 of the WBS includes
Major Elements of the defense materiel item, such as hardware, software elements, program management, training, data, etc
!Three types of leadership!
!Supervisory, participative, and team (Team is preferred)!
!Ethics may be thought of as a set of behavioral standards for a group of people or society!
!Ethics may be thought of as a set of behavioral standards for a group of people or society!
Ethics can also be defined as
Standards of conduct that shape one’s behavior with respect to moral duties and obligations
Identify the two major aspects of our personal ethics, according to the Josephson Institute of Ethics
Our ability to distinguish right from wrong, good from evil and propriety from impropriety
Out level of commitment to doing what is good, right and proper
Understand that although ethical norms vary from organization to organization, and culture to culture
There are some core values that have been identified by leaders in education, business, religion and government.
Identify the six core ethical values, according to the Josephson Institute of Ethics
Trustworthiness
Respect
Responsibility
Fairness
Caring
Citizenship
Identify simplest approach to ethical decision-making
The “hero test”
Ask yourself how someone you greatly admire would handle the situation
Ask yourself what you would do if your actions were going to appear on the front page of the newspaper
Understand the three classical models for resolving dilemmas
Consequentialism / Utilitarianism – end justifies the means as long as the greater good is served
The Golden Rule – “Do unto others as you would have them do unto you”
Kant’s Absolute Moral Duties – “Higher truths” are not flexible and there are no exceptions
Understand that the Principled Decision-Making model combines aspects of all three classical models, and
It calls for decisions to take into consideration the welfare of all stakeholders
It also expects ethical values, such as trustworthiness and fairness, to take precedence over other values, such as efficiency or self-interest
Finally, it offers help in prioritizing conflicting ethical values based on what will bring the most good and the least harm to others
Understand that Integrated Product Teams (IPTs) are
An important part of the acquisition process, and effective leadership of those teams is essential to their success
Recognize that your goal is to…
Create an atmosphere within your IPTs in which there is open, honest communication so that fear can be replaced with trust, and where the core ethical values are demonstrated consistently
Define the three primary types of leadership styles found in today’s workplace.
Supervisory – tends to be directive. They do a lot of one-on-one training. Get personally involved in resolving conflicts. React to changes as they occur. This is natural for first line supervisors, but we want IPT leader to take a different approach.
Participative –
1. Interested in obtaining inputs from the team
2. Consider different points of view prior to decisions
3. Seeks to Develop team member performance
4. Ensures changes lead to improvement
5. Focus is more on participation and involvement of the team members
Team Leadership – ideal style for most IPT leaders
1. Inspires individuals as well as the entire team
2. Creates a team identity and finds ways to harness team member diversity
3. Foresees and influences changes
4. Output of this team is greater than the sum of its members
Team Leadership is the ideal style for most IPT leaders
Team Leadership is the ideal style for most IPT leaders
In the DoD, Systems Engineering is:
Typically implemented via Integrated Product and Process Development (IPPD). IPPD is performed by multidisciplinary teams, typically formally charted as Integrated Product Teams (IPTs).
Both industry and the DoD have…
Similar goals in product development and share key Systems Engineering processes
The three Technical Processes for Designing Systems are:
Stakeholder Requirements Definition
Requirements Analysis
Architecture Design
The 8 Technical Management Processes are:
Technical Planning
Requirements Management
Interface Management
Risk Management
Configuration Management
Technical Data Management
Technical Assessment
Decision Analysis
Technical Management Processes are….
Used to manage the development of system products, including supporting or enabling products.
!MIL-STD 499A used to be…!
!The mandatory standard used for DoD Systems Engineering. It has be cancelled and replaced with EIA 632 and IEEE 1220!
Standards provide….
A set of processes that, if performed by qualified persons using appropriate tools and methods, will provide a capability for effective and efficient engineering of systems, including their software components
A System is…
An aggregation of end products and enabling products to achieve a given purpose.
Every system consists of :
One or more End Products
A set of Enabling Products
System Specifications:
This is a type of program-unique specification that describes the requirements and verification of the requirements for a combination of elements that must function together to produce the capabilities required to fulfill a mission need, including hardware, equipment and software.
Item Specification:
This is a type of program-unique specification that describes the form, fit and function and method for acceptance of end products (e.g., parts, components and other items that are elements of a system model).
A ‘program’ WBS displays and defines…
The product(s) to be developed and/ or produced by the program. It relates the elements of work to be accomplished to each other and to the End Product.
In the DoD, the Program WBS is…
Extended to a Contract WBS that defines the relationships between the elements of the program and corresponding elements of the contract work statement.
!An Engineering ‘V” Model!
!Is a system structure based on a hierarchy of system models and provides the basis for both Top-down System Design and Bottom-up product Realization!
The typical sequence of the ‘Bottom-up/Realization’ Systems Engineering processes is:
Implementation or Integration
Verification
Validation
Transition
Iteration
Is the re-application of processes already applied to a system model based on feedback indicating a problem that needs resolution before progress can continue.
The System Requirements Review (SRR) is …
A multifunctional technical review to ensure that all system and performance requirements derived from the Capability Development Document are defined and consistent with cost (budget), schedule (program schedule), risk and system constraints.
Detail Specifications…
These specify design requirements, such as materials to be used, how a requirement is to be achieved or how an item is to be fabricated or constructed.
Program SEPs are…
Formally assessed during milestone reviews
There are 8 System Engineering Technical Processes
Stakeholder Requirements Definition
Requirements Analysis
Architecture Design
Implementation
Integration
Verification
Validation
Transition
An Engineering Work model consists of…
Top Down Design and Bottom Up Realization.
In early stages of the acquisition life cycle many Systems Engineering technical processes are..
Performed by the government “Front-end” systems engineering-like activities include defining architectures and capabilities and conducting functional analyses, and planners usually complete these activities before a program is initiated.
A stakeholder…
Is any party that has an interest in the outcome of the project
The acquirer requirements come from…
The end users of the system
The JCIDS process formally documents stakeholder needs in terms of capabilities via a series of progressively more detailed documents. JCIDS key documents include:
Initial Capability Document (ICD) at Milestone A
Capability Development Document (CDD) at Milestone B.
Capability Production Document (CPD) at Milestone C.
The functional baseline is described by
The System Specification
Logical Analysis enables…
A more detailed definition and understanding of the problem to be solved.
A derived requirement is…
One that is not explicitly stated in a higher-level set of requirements yet required to be satisfied in order to provide a complete system solution
Two Broad Types of Derived Requirements:
- Derived Stakeholder Requirements
- Derived Technical Requirements
Behavioral analysis involves…
The simulation or stimulation of functional architectures utilizing operational scenarios to expose the model to stressful situations that reflect anticipated usage and environments
Interfaces, both for hardware and software, should be defined using…
Open System standards to the maximum extent possible wherever feasible and cost-effective
Implementation:
Is where the lowest system element items are made, bought, or reused. (Involves hardware fabrication and software coding)
Integration:
Is the Technical Process involved in realization of an End product when the assembled product consists of two or more end products from lower-layer system models. (Involves simulations, EDMs, LRIP models)
Verification Process (developmental testing):
Confirms that the system element meets the design to or build-to specifications as defined in the functional, allocated, and product baselines.
The four types of Verification are:
Analysis
Inspection
Demonstration
Test
The Validation Process…
Tests the performance of systems within their intended operational environment, with anticipated operators and users.
-Most tests conducted by Operational Test and Evaluation (OT&E) are used for Systems Engineering validation events.
VV&A stands for:
Verification, Validation, and Accreditation
PHS&T stands for:
Packaging, Handling, Storage, and Transportation.
Quality Function Deployment (QFD) starts with…
‘The voice of the customer.’ It uses a series of matrices to “flow down’ requirements to various levels.
There are 8 System Engineering Technical Management Processes:
Technical Planning
Requirements Management
Configuration Management
Technical Assessment
Decision Analysis
Risk Management
Interface Management
Technical Data Management
Technical Management Processes:
Establish and evolve project plans
Assess progress against the plan
Evaluate and select alternatives.
Control technical project execution
Technical Planning, as opposed to program planning addresses…
The scope of the overall technical effort needed to develop system products
The Systems Engineering Plan (SEP) is…
The Government’s foundation plan for Systems Engineering activities planned for use on a project.
Event-based schedules are based on…
Key events, related tasks and relevant completion criteria for the applicable engineering work model
An Interface refers to…
The functional and physical characteristics required to exist at a common boundary between two or more systems, end products, enabling products or subsystems.
The two types of interfaces are External and Internal:
–External Interfaces are the boundaries:
•between a system End Product and another external system End Product; or
•between a human and the environment in which the system End Products will be used or operated.
–Internal: those boundaries between products within a system model that are controlled by the developer or by the government acquirer’s technical effort.
The government Program Office and the project IPT will…
Play key roles in the development of external interfaces because these may involve interfaces with systems outside the control of the contractor.
!The three components of Risk are!:
!A future root cause
A probability or likelihood of that future root cause occurring
The consequences (or effect) of that future occurrence!
Risk Management includes these key elements:
Risk Identification
Risk Analysis
Risk Mitigation Planning
Risk Mitigation Plan Implementation
Risk Tracking
Configuration Management (CM) Establishes and maintains…
Consistency of a product’s attributes with its requirements and product configuration information
Three formal technical baselines established for a system are:
Functional
Allocated
Product
The product baseline is…
At the component level.
Technical Performance Measurement (TPM) measures…
The current state of a given technical parameter and then estimates how it will evolve over a specific time period based on the planned technical development effort
The Technical Authority should be…
Independent of the Program Manager in making assessments
A Technical Data Package (TDP) provides…
A comprehensive description of a given product.
Requirements change for a variety of reasons which can include:
Emerging technologies
Changing threats
New laws and policies
New missions
Evolving user needs
Government program office’s responsibilities:
The program manager is responsible for long term storage and retrieval of technical data
Risk is…
A measure of future uncertainties in achieving program technical performance goals within defined cost and schedule constraints.
Aspects of Risk include:
Threat
Technology maturity
Supplier capability
Design maturation
Performance against plan
Know the key elements of Risk management:
Risk Identification
Risk Analysis
Risk Mitigation Planning
Risk Mitigation Plan Implementation
Risk Tracking
The Waterfall Effect:
As risk mitigation efforts are successfully implemented for an identified root cause, the amount of risk present for that root cause is reduced.
Know what Risk is NOT
Risk is NOT an opportunity to exceed expectations
Risk is NOT a problem or an issue. These are previously known or unknown risks that have already occurred.
Risk Tracking answers the question “How are things going?” by:
Communicating risk to all affected stakeholders
Monitoring risk mitigation plans
Reviewing regular status updates
Risk identification is structured to help answer the question?
What can go wrong?
Know who reviews and approves the final risk mitigation plan
The Program Manager and the Lead Systems Engineer should review and approve the final risk mitigation plan – PM has final approval
Technical Performance Measures (TPM) measure?
Actual Results versus Expected Results
3 Components of Risk are…
A future root cause,
A probability or likelihood of that future root cause occurring,
The consequences (or effect) of that future occurrence.
Risk Management in a PM office may be centralized or decentralized
In a centralized risk management approach the program manager assigns a risk management coordinator or establishes a risk management board IPT to conduct most of the risk management efforts.
In the decentralized risk management approach, program personnel conduct risk management as part of their normally assigned duties within a program IPT. Since risks are present in every aspect/level of the program the decentralized approach is normally recommended.
The 4 Risk mitigation options are:
Avoiding
Controlling
Transferring
Assuming
