Devops Flashcards
What is devops?
It’s a practice or culture adopted by your organization, that would increase your organization’s ability to deliver applications(Continuous delivery)
But is devops only about improving delivery? No
1. Improving delivery
2. Automation - Deliver more chips
3. Quality - Customer cares about this
4. Monitoring - How do you make sure automation and quality is being maintained? You have to build in monitoring. So whenever there’s an issue, someone has to report back to us. This is done by monitoring.
5. Testing - Without testing, we cannot ensure that the quality or automation is correct.
So the proper definition is - Devops is a process of improving your application delivery by ensuring that there is a proper automation, the application quality is maintained, ensure there is continuous monitoring and continuous testing in place. So this is devops.
What do we achieve with all these things? - As a devops engineer, your goal is to ensure there should not be any manual process or during your application delivery, you have to fasten your delivery process.
Why Devops?
To improve the process of delivery. when there were bare metal servers, it used to take many days to deploy and multiple people were involved in the process. It’s a manual process with server admin, system admin, build and release engineer and few other people. To speed up and automate this process, we adopted devops.
As a devops engineer, I’m focused on automating and improving the building, testing and deployment phase. There should not be any manual intervention.
How to introduce yourself as a devops engineer?
- Im working as a java + devops engineer from 5 yrs.
- Current Roles and responsibilities - I take care of automation, quality, monitoring. I have automated the testing process into the devops lifecycle.
What is SDLC?
It’s a culture/ methodology to design, develop and test the application. The end goal is to deliver a high quality product.
Stages:
Planning - Requirements. BA does this
Defining - Documenting
Designing - High level and low level design
Building
Testing
Deploying
What is hypervisor?
A software that can install VM’s on your computer/bare metal servers.
It does logical isolation/separation on the server.
Each VM has its own portion of CPU, hardware and memory
You can automate the creation of VM’s or EC2 instances through AWS CLI. What are other ways?
- AWS API. In Python, using Boto3 module you can directly make a request to the AWS API.
- AWS CFT - Cloud Formation Template. If you provide this template to AWS, it will return you with 1 to n number of VM’s you’ve requested.
- AWS CDK - Cloud Development Kit
- There’s a great competitor available in market to automate your resource creation - Terraform.
1 terraform - multiple cloud providers
How do you create 10 vm’s at once/ what is the automation that you’re using in your organization for infrastructure creation?
Terraform or AWS. Look at your organization and determine what you use. If your org is completely focused on AWS, the you don’t have to use terraform. You can either use AWS CLI, API, CFT or CDK
When is Terraform used?
Orgs these days are using hybrid cloud pattern. So they have their VM’s in one cloud platform, other resource infrastructure in other cloud platform. In this model, terraform is best for you, coz you have to automate the infra across different cloud platforms
What is Kernel?
Kernel is heart of the OS. Its responsibility is to establish a communication between your hardware and software. It has 4 responsibilities.
1. Device Management
2. Memory Management
3. Process management
4. Handling system calls
It cascades the requests from s/w to h/w and back to h/w.
Components in Linux
Down to Top
OS - Kernel - System Libraries - compilers| User processes | system software
Why Linux is preferred in Prod?
Lightweight, so Fast
Free (OSS)
Secure
Shell commands
! - this is called shebang. This is the first thing you write in a shell script.
Rename or move - mv file file
free -g : see memory of your server
nproc : count of CPU’s
df/df -h : disk size
Top : see all the above info with one command
man ls/ man touch : open manual for any command you want to reference .
chmod- grants permissions to a file
chmod. What are the permissions for admin user, What are the permissions for group, What are the permissions for all users?
chmod 777 - access for everybody
Find command - important command used in devops. It searches entire system.
sudo find / -name pam
Kill Java process - kill -9 processId
Linux uses 421 - 4 for read, 2 for write, 1 for execute.
What’s the purpose of writing a shebang? - to tell the kernel which interpreter should be used to run the commands present in the file.
What is shell scripting?
A process for automating your daily activities on your Linux computer
Command to execute a shell script
./file.sh
Or
sh file.sh
Where is shell scripting used?
As a devops engineer, you maintain all the Infrastructure, code repositories and do a lot of configuration management. For all these activities, on a day to day basis, you use shell scripting.
On a single automation, what a user expects is, you have to login to a specific machine where ansible automation is present and you have to execute the ansible automation.
Why are you using shell scripting?
To automate node health of my VM’s. We have close to 1000 VM’s and every time it is difficult to monitor the health or status of these VM’s, so I write shell scripts for that.
There are many automated tools, so why do you want to write shell scripting?
You can say ‘In our org, we’re not using any such tools
Or
These tools are restricted have restricted number of parameters, but the scripts can fetch more parameters that are not provided by these tools.
Other Shell scripting use cases
Infrastructure automation
Configuration management
Amazon example
Day to day activities monitoring - you want to monitor specific tools and send email notifications - we can use shell scripting
I/Q: How do you write script to monitor node health? / How do you monitor node health?
I can use the ‘top’ command or I can write custom shell scripts that monitor CPU and Ram usage.
Good practices and how to write shell script in a scructured manner.
- Start with shebang followed by the executable that we want to use- #!/bin/bash
- Metadata of the file like author name, date etc.
- Set -x #debug mode
- Set -e #exit script when there is error (set -e has a drawback. It doesn’t error out when there’s a pipe. So we have to use the below command when using pipe.
- Set -o #pipefail when there is a pipe
Shell command to find processes and process id’s?
ps
ps -ef
To filter for Amazon processes only - use grep
ps -ef | grep “Amazon”
What does the pipe parameter do?
To use it with grep command -
ps -ef | grep “Amazon”
./test.sh | grep 1
Pipe parameter sends the output of first command to second command.
I/Q: What will be the output of
date | echo “this” ?
It will print “this”, coz date is a system default command. It sends the output to stdin. And pipe won’t receive output from stdin. Pipe can only receive information if the command is not sending information to stdin and if the command is ready to pass the information to the next command .
Awk command
If you want to retrieve only a specific column from the result, then use awk.
It can filter out information from the output.
cat test.sh | grep Sandeep | awk -F” “ ‘{print $2}’
Difference b/w grep and awk:
Grep command gives you the entire sentence with all columns as a result.
Awk only give you a particular column from the output
When an application is failing, the first step a devops engineer does in any company is check the log files.
But when the log files are stored in Aws s3 or Azure blog storage or whatever storage outside your VM, how can you retrieve this information from the Linux terminal?
Using curl command, just provide the url of the log file location.
Curl command retrieves information from internet.
curl url | grep Error
To get response from a website -
curl -X GET website.com
I/Q: What’s the diff b/w wget and curl command?
wget will download the file, the we can perform the grep command on the downloaded file. So it’s two command.
curl command does not download. 1 command.
Command to login as root user
sudo su -
Meaning substitute user do switch user.
If you want to login to other person’s account - use su command meaning - switch user.
su sandeep
su rasajna
I/Q: What is trap command?
Used for trapping any signals that are available on your Linux machine.
When we press Ctrl C, it stops execution. When we trap a signal, even when you do Ctrl C, understand that I have a trap mechanism set on my machine, so don’t do anything. Or if they do Ctrl C, send me an email notification using SMTP server, or print an echo statement saying that you cannot do this operation coz the owner of this Linux machine said he doesn’t want to execute Ctrl C.
trap echo “don’t use Ctrl C” SIGINT
What are signals and what are the different signals available in Linux?
There are a lot of signals in Linux like SIGQUIT, SIGINT. When we use Kill command to kill a process, Linux receives a signal called SIGKILL.
Use case of trap command
You’re inserting data into database using a script. Someone presses Ctrl C. Use a trap signal echo statement saying it’s not allowed. Or you can delete the entire data that was inserted until that point using
trap “rm -rf *” SIGINT
What is set -u command
The set -u command in Linux is used to set the shell option that causes the shell to exit if a variable is accessed before it is set. This can help catch potential errors in shell scripts where an unset variable is being used.
After running this command, if a script tries to use an undefined variable, the script will exit and an error message will be displayed. This can be helpful for ensuring that all variables are properly initialized in a script, thereby reducing the risk of unexpected behavior due to unset variables.
Disadvantages of Shell scripting
- Slow execution speed, especially for heavy math operations.
- Not suited for large and complex tasks due to minimal data structure support.
- Prone to costly errors. A single mistake in a shell script can alter the command.
- If u dont use set -u to handle undeclared variables, the compiler will not complain.
Is bash dynamically typed or statically typed?
Modern day programming languages are statically typed, meaning the type checking is done at compile time - Java, Golang etc.
Shell, Python are dynamically typed, which means that the type checking of variables is done at runtime.
Network troubleshooting tool
traceroute.
traceroute Google.com - this can show you the many hops in between your traceroute and Google.com
You can also use tracepath.
tracepath google.com
How will you sort list of names in file?
Using sort command.
sort names.txt
sort -r names.txt #in reverse order
sort names.txt > sorted.txt #save sorted output to a new file
How will you manage logs of a system that generates huge log files everyday?
There is an efficient way to do this - logrotate
You can say logrotate and define how many days you want to rotate this logs.
Or
You can say, for every 24hrs, just zip this log. You can also define the format - you can say logrotate (zip, gzip), create a compressed version of this zip file and you can say after 30 days, delete this log file
Why would somebody move to cloud infrastructure?
- To reduce maintenance overhead
- To be Cost effective
Suppose we need only instance ID from the AWS resource output of the shell script. What command will you use to parse the json and retrieve only the
jq- json parser
yq- yaml parser
aws ec2 describe-instances | jq ‘.Reservations[].Instances[].InstanceId’
If you want to install it- sudo apt install jq -y
I’m maintaining a GitHub for my org. An employee is leaving. His access needs to be revoked. How do I write a given shell script and automate this process?
We will clone the shell script onto an EC2 instance, we will execute this against this particular repository and see if it is working fine or not.
Primary concept of version control system - it addresses two problems. What?
- sharing
- Versioning
I/Q: What is the difference bw Centralized and Distributed version control system? And How is git solving the problem?
Or Diff bw SVN and Git?
You have to push/pull the code to and from the central code repository in Centralized. In Distributed, you create multiple copies of the distributed system code base(fork) and send the changes to that copy. And ask someone else to take the changes from this copy.
If there’s a single point of failure that is offline, when their applications and servers go down, there was no way Developer A and Developer B can communicate - coz CVS or SVN was down. This was the problem with centralized version control systems.
Fork - creating an entire copy of your code
I/Q: Difference bw Git and GitHub?
Git is a distributed Version Control system. Open source.
Any org can download git and implement it in their org. Git also comes with a UI.
How do you do it?
Create an EC2 instance, install git in the instance and say every developer should commit their changes to this server.
GitHub is a platform for developers to store and manage their code. It uses git dvcs, can raise issues, bug tracking, code reviewing, commenting, talking to peers, and now GitHub also supports project management.
I/Q: What is the git workflow that you use in your org?
git add
Git commit -m “commit message”
Git push
I/Q: Difference bw git merge and rebase
If you want to track the commits- which commit came after which commit; and want your commit history will be in a linear way. In such cases always use rebase. whereas if you’re not bothered about it, you can use merge.
git merge is a way of combining changes from one branch (source branch) into another branch (target branch) where as git rebase is a way of moving the changes from one branch onto another branch
Git Rebase: On the other hand, git rebase offers a different approach. It rewrites the commit history by moving your changes on top of the updated remote branch. This creates a linear, cleaner history, which can be super helpful when working on feature branches. It’s like repackaging your changes as if you just started with the latest updates.
Rebase is one of two Git utilities that specializes in integrating changes from one branch onto another. The other change integration utility is git merge . Merge is always a forward moving change record. Alternatively, rebase has powerful history rewriting features.
When to use Git Rebase or Git Merge
Choose Merge
whenever we want to add changes of a feature branch back into the base branch.
if you want to keep the same history rather than rewrite it.
if you want to revert the changes quickly
Choose Rebase
whenever we want to add changes of a base branch back to a feature branch.
squash multiple commits
reiterate each commit and update the changes
reverting rebase would be very difficult
Difference bw git fetch and git pull
Git fetch only downloads the new data from a repository, but it doesn’t integrate the changes into your working files.
But Git pull is a combination of git fetch and git merge. It downloads the new code and also integrates it into our working directory.
In short, git fetch is great for reviewing changes before integrating them, while, git pull can fetch and integrate changes.
What is pre commit and post commit hooks?
Hook: If you want to perform an action before or after something, then it’s called a hook.
Pre commit hooks are actions that are taken before you do git commit.
Post commit hooks are actions that are taken after you do git commit.
You have the password files, secret information files, public or private keys etc that you don’t want to accidentally push to git. For that you can configure them in your pre commit hook, and tell git that before every commit, just execute this script, and what git does is that, if you’re accidentally even committing any such files, it executes this pre commit hook that you’ve given to git and says - I can’t commit this coz your pre commit hook is preventing it.
Exactly opposite, if you want to execute an action after a commit, it’s called post commit hook
Webhooks (w)
It’s a part of your GitHub or bit bucket or any of your git implementations.
If you want to trigger a pipeline or execute a python script after your git commit is done, or you want GitHub to perform an action, you can simple configure a w - means in GitHub you can configure something called hook, and GitHub does the action that you tell it to do.
If you go to GitHub web hooks, there are a lot of action you can do like perform w after issues, perform w after pull requests… everything. For every action in GitHub, you can configure a w.
CLI Command to add remote repository
git remote add “link”
