Intro Flashcards
1
Q
What is Devops
A
A cultural amd professional movement that stresses communication, collaboration and integration between software developers and IT operations professionals while automating the process of software delivery and infraestructure changes.
It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably
2
Q
Keys from the definition of devops
A
Core messages:
Cultural
Professionalism
Communication
Collaboration
Automation
Speed and frequency
3
Q
What DevOps is not
A
It’s not a title nor a team.
It’s not just a tool.
It’s not only culture
It can’t be reduced to just automation
It does not mean that developers can do whatever they want
4
Q
What are the goals of DevOps
A
- Smaller, mor frequent releases
- Reduced effor and risks
- Reduced cost of product iterations and delays
- A culture of communication and collaboration
- Consistency and speed through automation
5
Q
What are the improvements DevOps brings?
A
- time to market
- integration with the business
- Responsiveness
- Code and deployment quality
- Productivity
- Visibility
- Agility
6
Q
NoOps or NewOps
A
While automation is the goal of NoOps, there will always be a need for human competencies and roles, like managing the operational automation, designing, implementing, and managing internal and external infraestructures and enterprise architectures.
Migratino to the cloud does not signal the end of Ops. The role of Ops is evolving as much as the role of Dev.
7
Q
DevOps Values
A
- Culture. Relates to the people and processes aspects of DevOps and the need to improve communication and collaboration. Without the rigth cultureautomation attempts will be fruitless.
- Automation. Tools that enable automation are important aspects of DevOps.
- Lean: maximizing customer value while minimizing waste and improving flow
- Measurement: If yo can’t measure, you can’t improve.
- Sharing: Is the feedback loopback
8
Q
Autoamtion is essential
A
Automation enables agility, consistency, speed and realiability.
Automation and Toochains help increase the speed and consistency of IT delivery.
Inconsistency in tools, frameworks, and methodologies are the enemy of broad, enterprise-wide DevOps success
9
Q
DevOps stakehoders
A
Dev includes all the people involved in developing software products and services. Product and project managers are here.
Ops includes all the people involved in delivering and managing software products and services.all administrators are here.
10
Q
DevOps improves Cadency and Velocity
A
Every business has become a tech business.
Customers value outcomes, not products.
Time to value is replacing time to market
11
Q
Golden Circle
A
Simon Sinek, 2009.
Three circles: Why, How, What.
Why: Purpose, cause, belief, the reason the org exists.
How: What sets the organization apart.
What: Product and services.
12
Q
Business Value of DevOps
A
Companies with high IT performance are twice as likely to exceed their goals.
Value is defined by the customer. IT must contribute to the business value stream
13
Q
Challenges of IT in DevOps
A
Internal:
IT must go faster without risking quality
IT’s silo culture is constraining the value stream
Prior investments (ITSM, Agile, Automation) aren’t delivering end to end value.
14
Q
Wall of Confusion
A
Devs want change, Ops want stability
DevOps is a response to the growing awareness that there is a disconnect between what is traditionally considered development activity and what is traditionally considered operations activity.
What does the business want?
Equilibrium between extreme focus on change, quality and extreme focus in stability
15
Q
Top 5 predictors of IT performance
A
- Peer-reviewed change approval process.
- Version control for all production artifacts.
- Proactive monitoring.
- high-trust organizatinoal culture.
- Win-win relationship between Dev and Ops.
16
Q
The Three Ways
A
- Flow. Understand and increase the flow of work (left to right)
- Feedback. Create short feedback loops that enable continuous improvement.
- Continuous experimentation and learning. Create a culture that fosters them.
17
Q
The First Way: Flow
A
Increase flow by understanding and removing constraints.
Never passing a known defect downstream
Never allow local optimization to cause global degradation
Achieve profound understanding of the entire system
18
Q
Theory of Constraints
A
A methodology for identifying the most important limiting factor that stands in the way of achieving a goal and then systematically improving that constraint unitl it is no longer the limiting factor.
Every process has at least one constraint or bottleneck that affects its ability to consistently meet its goal.
The Process will only meet the capacity of its constraints and will be only as successful as its weakest link.
Improving constraints is the fastest and most efficient way to improve the entire process or system.
19
Q
Common Constraints
A
Development delays.
Environment creation.
Code deployment.
Test setup and run.
Security or QA assessments.
Overly tight architecture.
Product management .
Complex or bureaucratic processes.
20
Q
The Second Way: Feedback
A
Understand and respond tot he needs of all customers, both internal and external
Shorten and amplify all feedback loops.
It stresses the importance of right to left feedback
Without feedback teams lack the information needed to continuosly improve.
21
Q
Examples of feedback loops
A
Automated testing.
Peer review of production changes.
Monitoring/Event management data.
Dashboards.
Production logs.
Process measurements.
Post-mortems.
Shared on-call rotation.
Change, incident, problem and knowlwedge management data.
22
Q
The Third Way: Continual Experimentation and Learning
A
Allocate time for the improvement of daily work.
Create rituals that reward tje team for taking risks.
Introduce faults into the system to increase resilience.
Plan time for safe experimentation and innovation.
23
Q
Simian Army / Chaos Monkey
A
Simian Army was adopted by Netflix as a service that randomly terminates a production instance.
Response to attacks helps to build competencies to recover the productino environment from inevitable failures
Chaos Monkey is a service that identifies groups of systems and randomly terminates one of the systems in a group. It runs only in business hours with the intent that engineers will be alert and able to respond.
24
Q
Encourage a learning culture
A
Create a learning culture is critical to sustain growth and innovation
Encourage daily learning and knowledge sharing
Create training and skills-based education plans
Incorporte learning into processes
Use technology to accelerate learning
Make work educatinoal through experimentation, problem solving and demonstrations
Allow and use mistakes as sources of learning
Make the results of learning visible
25
DevOps practices
Continuous integration
Continuous delivery and deployment
Continuous testing
Rugged DevOps
DevSecOps
ChatOps
Kanban
26
Continuous Integration (CI)
This practice requires developers to commit ocde into a shared repository at least daily
It also mandatesrunning production-like environments, and passing acceptance and integration tests
Each check-in is validated by an automated build and automated unit, integratin and acceptance tests.
Is dependent of consistent codng standards
Requires version control repositories and CI servers to collect, build, and test committed code together
27
Continuous Delivery (CD)
This methodology focuses on making sure software always is in a releasable state throughout its lifecycle.
Continuous delivery is achieved by continuously integrating the software done by the development team, building executables, and running automated tests on those executables to detect problems.
You are doing continuous delivery when:
* Your software is deployable throughout its lifecycle
* Your team prioritizes keeping the software deployable over working on new features
* Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them
* You can perform push-button deployments of any version of the software to any environment on demand
28
Stages of deployment pipeline
Development
Commit
Build
Test
Stage
Deploy
Production
Feedback Loop
29
Continunous Delivery and Testing
Continuous delivery means that you are able to do frequent deployments but may choose not to do it, usually due to businesses preferring a slower rate of deployment.
Continuous delivery requires that whenever anyone makes a change that causes an automated test to fail, breaking the deployment pipeline, developers stop the line and bring the system back into a deployable state.
Automated tests in production-like environments assure the code and environment oeprate as designed and are always in a deployable state
30
Continuous Testing
Is the process of executing automated tests as part of the deployment pipeline to obtain immediate feedback on the business risks associated with a software release candidate.
Functional: Unit tests, API, integration, system testing
Non-Funtional: Performance, security, compliance, capacity.
31
Testing deployment strategies
Test-driven development
Performance testing
End-to-end use-case testing
Security testing
Manual testing
Testing in production
Shifting left is about building quality into software development process.
32
Continuous Deployment
Is a set of practices that enable every change that passes automated tests to be automatically deployed to production
Removes the manual step in the deployment pipeline
It may result in multiple deployments per day.
Code deployed into production may be invisible to customers but features can be run and tested by internal staff.
33
Continuous Delivery leads to Higher IT and Business Performance
The factors that form continuous delivery lead to the following:
Less rework
Lower levels of deployment pain
Higher throughput and stability for IT
Lower change failure rates
Indentifying strongly with the organization you work for
34
Rugged DevOps
This method includes security practices as early in the continuous delivery pipeline as possible to increase cybersecurity, speed, and quality of releases beyond what DevOps practices can yield alone.
Is a cultural approach to creating available, survivable, defensible, secure, and resilient software.
Is related but not the same as DevSecOps
35
DevSecOps
Everyone is responsible for security. The goal is safely distributing security decisions at speed and scale tot hos who hold the highest level of context whitout sacrificing the safety required.
It embeds security as code by "shifting-left" with continuous security testing throughout the deployment pipeline.
36
ChatOps
Communication approach that allow teams to collaborate and manage many aspects of their infraestructure, code, and data from a chat room.
It decreases MTTR.
Shorten feedback loops, improves information sharing, enhances team collaboration, and enables cross-training.
Slack, VictorOps, HipChat, Campfire, etc.
37
Kanban
Method that pulls the flow of work through a manageable pace.
It visualizes and manages workflow
Pulls work for teams when they are ready for it.
Enables people to work collaboratively to improve flow.
Measure teams velocity
Reduces idle time and waste in a process.
Makes work visible
Makes policies explicit
Limits Work In Progress to capacity
38
Devops and othe methodologies
DevOps isn't a methodology or a framework. It adopts multiple frameworks and methodologies like agile, lean and ITSM
39
Agile
The Agile Manifesto was written in 2001.
Agile software development are a group of software development methods in which requirements and solutions evolve through collaboration between self-organizing, cross-functional teams.
Agile software development methods deliver working software in smaller and more frequent increments.
It's more important to be agile than to do agile. Doing or managing project using Scrum does not make you agile.
40
Scrum
Simple framework for effective team collaboration on complex projetcs. It provides a small set of rules that create just enough structure for teams to be able to focus their innovation on solving what might otherwise be an insurmountable challenge.
Scrum increases the ability to release more frequently
41
Scrum basics
Roles: Product Owner, Scrum Master, team.
Artifacts: Product Backlog, Sprint Backlog, Burndown Charts, Potentially Shipable Increment
Meetings: Release Planning, Sprint Planning, Daily Scrum, Sprint Review, Sprint Retrospective
42
Ops involvement in SW development
Ops must Define, Plan, Build and Test, and Provide Feedback for the following:
Non-functional requirements of the project.
Define Ops acceptance criteria
Infraestructure provisioning
Design, build and test ITSM and IT Operations automation capabilities
Embed Ops knowledge in Dev
Ops involvement early on means that Done means fully tested and production ready, not just that the coding of a feature is done.
43
Scaled Agile Framework (SAFE)
Is a proven, publicly available framework formapplying Lean-Agile principles and practices at enterprise scale
It focuses on iterative and incremental development, agiile SW development, product development flow, lean thinking an field experience at enterprise scale.
44
DevOps increases agility
DevOps extends agile principles beyond the boundaries of the software to the entire delivered service.
Breaks down silos.
Improves constraints.
Takes a unified approach to systems engineering
Applies agile principles to both Dev and Ops.
Shares knowledge, skills, experiences and data.
Recognizes the criticality of automation.
Deploys faster with fewer errors.
45
IT Service Management (ITSM)
Is the implementation and management of quality IT services that meet the needs of the business.
It provides guidance and structure to processes such as Change, Configuration, Release, Incident and Problem Management.
ITSM processes underpin the entire service lifecycle from strategy, design, transition, operations, continual improvement and value creation.
46
ITIL
Is a set of best practices publications for IT Service Management.
Strategy, Design, Transition, Operation, Continual Service Improvement.
It can improve and streamline the processes of organization that have adopted DevOps practices.
Service: A means of delivering value to customers by facilitating outcomes customers want to achieve whitout the ownership of specific costs and risks.
Service Management: A set of specialized organizational capabilities for providing value to customers in the form of servics.
IT Service: Service provided by an IT service provider, a combination of people, technology and processes.
47
ITSM Process Model support DevOps and CD
ITSM favors predefined structures:
- Steps to be taken.
- Chronological order and dependencies.
- Responsibilities.
- Timescales and thresholds
- Escalation procedures.
Defines steps for handling specific types of transactions.
Ensures a defined path or timeline is followed.
It can be automated.
48
ITSM processes and DevOps
Change Management
Release Management
Service Asset and Configuration Management
Knowledge Management
Event Management
Incident Management
Problem Management
Capacity Management
49
Agile Service Management
It ensures that ITSM processes reflect Agile values and are designed with "just enough" control and structure in order to effectively eand efficiently deliver services that facilitate customer outcomes when and hoe they are needed.
It implements service management in small, integrated increments.
It ensures ITSM processes reflect Agile values from initial design through CSI.
50
Lean
Adjective: Spare, economical, lacking richness or abundance.
Value Stream: Lean tool that depicts the flow of information, materials and work across functional silos with an emphasis on quantifying waste, including timeand quality.
51
Lean tools
A3 Thinking: problem solving.
Continuous flow: Eliminating waste.
Kaizen: Continuous improvement.
Kanban: Pull system.
KPI: Key Performance Indicator.
Plan-do-check-act: Deming Continuous Improvement cycle.
Process Mapping: Visualize process to indentify areas of improvement.
Root cause analysis: Identify source of errors and waste.
SMART goals: Specific, measurable, achievable, relevant, time-bound.
Value Stream Mapping: Depict flow.
52
Five Principles of Lean Thinking
1. Define value preciselyfrom the perspective of the end customer.
2. Identify the entire value stream for each service, product or product family and eliminate waste.
3. Make the remaining value-creating steps flow.
4. As flow is introduced, let the customer pull what the customer wants when the customer wants it.
5. Pursue perfection.
53
The 3 M's of Lean
Muda, Mura, Muri
Waste, Inconsistency, Overburden
When ork is not standardized it can add waste in the form of wasted movement.
Sources of waste: DOWNTIME.
Defects
Overproduction
Waiting
Non-use
transportation
Inventory
Motion
Excessive Processing
54
Valule Stream Mapping
Is a lean tool that depicts the flow of information, materials and work across functional silos with an emphasis on quantifying waste, including time and quality.
Is a sequence of activities required to design, produce, and delivera specific product or service.
It makes work visible to everyone.
Is used to understand and streamline work processes using Lean tools and techniques.
55
Improvement Kata
A Kata is a any structured way of thinking and acting that you practice until the patternbecomes a habit.
Practicing the Improvement Kata should be a daily ocurrence
Steps:
1. Understand the long-term vision or direction.
2. Grasp the current condition.
3. Establish the next target condition.
4. PDCA and experiment toward the target condition
56
Improvement Kata - Five Questions
1. What is the target condition?
2. What is the actual condition now?
3. What obstacles do you think are preventing you from reachingthe target condition?
4. What is your next step?
5. When can we go and see what we have learnedfrom taking that step?
57
Devops Values - Culture
Organizational Culture: The values and behaviors that contribute to the unique social and psychological environment of an organization.
Cultural debt occurs when cultural considerations are disregarded or deferred in favor of growth and innovation.
IT's silo culture and other organizational challenges are a direct result of disregarding cultural considerations in favor of rapid increases in corporate technology.
58
DevOps - Culture. Characteristics.
Shared visions, goals and incentives
Open, honest, two-way communication
Collaboration
Pride of workmanship
Respect, Trust, Transparency
Continuous IMprovement
Data-driven
Safe
Reflection
Recognition
59
DevOps - Culture. Shifting thoughts and behaviors
Real culture change takes time. it must be incremental an performed at a realistic pace.
IT focus -> Customer focus.
Silos -> Cross-functional teams.
Command and control -> Collaborative.
Task-oriented -> Outcome-oriented.
Blame -> Responsibility.
Reactive -> Proactive.
Content -> Courageous.
Resistant -> Flexible.
Low trust -> Hight trust
60
DevOps - Culture. Flow of information
High trust organizations encourage good information flow, cross-functional collaboration, shared responsibilities, learning from failures and new ideas.
Characteristics:
- Information is actively sought.
- Messengers are trained.
- Responsibilities are shared.
- Bridging is rewarded.
- Failure causes enquiry.
- Novelty is implemented.
61
DevOps - Culture. Managing change.
People adapt at different paces to change:
Innovators, early adopters, early majority, late majority, laggards.
Stages of change acceptance:
Status Quo, Shock, Denial, Anger, Resistance, Resignation, Exploration, Acceptance, Building, Commitment.
62
DevOps - Culture. Communication is Critical
A DevOps requires timely and effective communication.
Shared tools facilitate timely and meaninful communication: chat platforms, task managers, social tools, alert management tools, knowledge sharing platforms.
Communication is passive.
Encourage collaborative relationships.
It enables people to work together to achieve goals that they could not reach individually.
Collaboration is active.
63
Devops - Culture. Conflict Management
Conflict Mode -> Approach -> Result
Competing -> Assertive and uncooperative -> win/lose
Collaborating -> Assertive and cooperative -> win/win
Compromising -> Partially assertive and cooperative -> Each wins and loses
Avoiding -> Unassertive and Uncooperative -> lose/lose
Accomodating -> Unassertive and cooperative -> lose/win
Change fatigue can sever peoples' ability to be committed to the organization
The amount of change fatigue that people experience is directly impacted by the way hange is managed.
64
DevOps - Culture. Empower new behaviors
Sharing between peers, organizations and industries is a crucial factor in the growth and acceptance of DevOps.
Improve communication and collaboration practices and shared tools
Create a common vocabulary
Job shadowing
Cross-killing
Immersion experiences
Team building
Communities of practice
Internal DevOps day
Game days (hackaton)
Simulations
65
DevOps - Automation
Automation alone cannot give you DevOps, but you cannot succeed without it.
Automation can be applied to:
CI and CD
Configuration Management
Build, testing and deployment processes
On-demand creation of development, test, staging, and production environments.
IaC
Monitoring and dashboards
Experimentation
Ongoing operations and support.
66
DevOps - Automation. Benefits
Faster lead times
More frequent releases
Less turbulent releases
Fewer errors
Higher quality
Improved security and risk mitigation
Faster recovery
Business and customer satisfaction
It allows people to weight evidence, solve problems, make decisions based on feedback and ser their skills, experience and judgement by removing rote and repetitive tasks.
67
DevOps - Automation. Important terms
Artifact. Any element in a software developmentproject including documentation, test plans, images, data files and executable modules.
API. A set of protocols used to create applicatinos for specific OS or as an interface between modules or applications.
Microservices. A software architecture that is composed of smaller modules that interact through APIs and can be updated without affecting the entire system.
OS Virtualization. A method of splitting a server into multiple partitions called containers or virtual environments in order to prevent applications from interfering with each other.
Container. An apprach to software development that packages pieces of code so that they can quickly be plugged in and run.
Open source. Software that is distributed with its source code s that end users organizations and vendors can modify it for their own purposes.
68
DevOps - Automation. Practices
Start with an understanding of the artifact and information flow an then begin looking at tools.
With teams collaborating and coordinating around the same tools, Dev and Ops teams can better understand each other's requirements and working methods.
Self-service can improve the timeliness of, for example, environment creation.
Test-driven development requires architecting software in a way that enables test automation.
69
DevOps - Automation. Communication and Collaboration
They can be automated too.
Each organization must determine the tools it will use and the rules of engagement associated with those tools.
Real-time collaboration helps improve flow.
A shared backlog allows both teamstos elect improvement projects from an organizational perspective. It can also be used to prioritize the work that delivers the greatest organizational value and helps to pay down technical debt.
70
DevOps - Automation. First steps.
Architect before automating.
Asses your existing tools and automatino capabilities.
Simplfy first: don't automate bad processes.
identify crititcal gaps.
Seek vendors who can meet your requirements.
Automate high value, repetitive and error-prone work.
Optimize workflow bottlenecks and communication.
Improve automated monitoringand notification practices
This process is iterative.
71
DevOps - Toolchains
A toolchain is composed of the tools needed to suport DevOps continuous integration, deployment, and release and operation initiative.
Toolchains automate tasks in the deployment pipeline.
Each element serves a specific purpose.
Applications within are connected via APIs.
Requires an architectural design to ensure interoperability and consistency.
The deployment pipeline breaks the software delivery lifecycle into logical stages. Each stage provides the opportunity to verify the quality of new features from a different angle, fast feedback for the team, visibility into the flow of changes.
DevOps toolchains provide the capabilities needed to automate and expedite each stage.
72
DevOps - Toolchains. Typical toolchain elements
Requirements Management
Orchestration and Visualization
Version Control Management
Artifact Management
Containers and OS Virtualization
Test and Environment Automation
Server configuration and deployment
System configuration Management
Alerts and alarms
Monitoring
73
DevOps - Toolchains - Deployment pipeline
Do not build a definitive toolchain that applies to all DevOps projects.
Multiple business applications require multiple deployment pipelines and toolchains.
The toolchain is a foundation requiring conitnuous innovation and customization to meet your specific and ongoing DevOps priorities.
Ensure that each member understands the capabilities and role of each tool in the DevOps toolchain toa void tooloverlap and toochain functionality gaps.
1. Model your value stream.
2. Automate build and deployment processes.
3. Automate unit tests and code analysis.
4. Automate acceptance tests.
5. Automate releases.
6. Add more automation as needed.
74
DevOps - Lean.
Lean principles are considered a core competency and value for DevOps.
In 2010 at a DevOps Days in Mountainview, California, John Willis and Damon Edwards coined the DevOps acronym CAMS, which stands for Culture, Automation, Measurement and Sharing. Jez Humble, author of the Continuous Delivery book later added an L, standing for Lean, to form CALMS. Since then Lean principles and tools have been considered to be a core competency and value for DevOps.
75
DevOps - Measurement.
You can't measure Dev and Ops separatedly.
What is measured: speed, quality, stability, culture.
Change lead time is one of the more important metrics as it represents what the customer sees.
Change failure rate is an important measure of reliability and stability.
MTTD: Mean time to detect incidents.
MTTR: Mean time to recover (component).
MTRS: Mean time to restore (service).
76
DevOps - Sharing
Actively sharint tools, knowledge, discoveries and lessons learned helps Dev and Ops to:
- identify new collaboration opportunities
- Avoid redundant work and overcome silo cultures
- Create common vocabularies and mindsets
- Create active exchanges of ideas and innovation
- Respect each other's skills, expertise and commitment
77
DevOps - Roles. Skills and characteristics
Skills:
Knowledge of business priorities and processes.
Specialist with broad generalist knowledge
Soft skills like communication, collaboration, team work.
Self-management. Initiative, time and stress management, self-motivation, focus.
Characteristics:
Adaptable, Customer-focused, Craftsmen, Curious, Data-driven, Engaged, Empathetic, Transparent.
Roles:
DevOps evangelist or leader
Software engineers, developers, and testers.
Release manager
Automation/continuous delivery architect
Build engineer
Security engineer
Quality assurance / Experience assurance
DevOps operations engineer
IT Support
Agile Service Manager
Agile Process Owner
78
DevOps - Organizations
DevOps teams.
Expand upon the concept of an Agile or Scrum team
Embed Dev and Ops skills into a single holistic group
May be temporary or dedicated to a specific product
May be cross-functional 'tiger-teams' for short projects
May evolve to provide shared services
Have shared accountabilities
Downsides of dedicated DevOps teams:
Less engagement across the IT value stream
Risk of beng another silo
Less accountability from Dev and Ops
DevOps activities become someone else's problem.
Reglardless of structure, a DevOps team should be flat, with continuous engagement and the right balance of people, practices and automation skills.
79
Getting started - Eight steps to transform your organization
Establish a sense of urgency
Create a powerful guiding coalition
Develop a vision and a strategy
Communicate that vision
Empower broad-based action
Create short-term wins
Consolidate gains and produce more change
Anchor new approaches in the culture
80
Getting started - Where to start?
Get clear on the business oportunities: the why?
Get the right people together
Get everyone on the same page
Invest in training and skills development
Build capabilities that lead to lasting change
Focus on critical behaviors
Experiment and learn
Consolidate gains and produce more change
Avoid inertia
81
Getting started - Learn by doing
Create a pilot where you can maximize the probability of success
It should be small enough where success is apparent and understood and consequences of failure aren't so large that a mistake could shut down the entire initiative
It should be large enough that you can show proof of improvement and you earn the right to make future improvements
82
Getting started - Consolidate gains and produce more change
Communicate successes, failures and lessons learned.
Document and make available reusable artifacts and measurements.
Expand your cycles of improvement
Continuously invest ind education
Introduce advanced tools and techniques as needed
83
Critical Success Factors
Management commitment to culture change
Creation of a collaborative, learning culture
Training and continuous skills improvement
Common values and vocabulary
System engineering that spans Dev and Ops
Meaninful metrics
Balance between automation and human interaction
Application of agile and lean methods
Open anf requent communication
84
Challenges and Risks
Lack of commitment and clarity
Transforming a them and us culture
Blending teams that are geographically dispersed, unfamiliar with each other and may include suppliers
Lack of education, training and skill
Immature service management processes
Inadecuate technologies
Poor communication
