Kubernetes DevOps Flashcards
How do you automate Kubernetes deployment?
Development flow
- Code Repo
- Dockerize (jenkins, codebuild, gitlab)
- Container Image Repo
- Deploy [Helm, KubeCtl] ((jenkins, codebuild, gitlab)
How do you secure Kubernetes app?
When it comes to security in the Kubernetes application, there are two aspect of it
Application Security
DevSecOps = Security of the container DevOps Lifecycle
Kubernetes Security
- Application Security
- Pod, namespace, Node
- RBAC, IRSA
DevSecOps = Security of the container DevOps lifecycle
Authorization (IAM Role)
- Scan repository (SysDig Falco, ECR image scanner)
- Scan running container (Twist lock, Black duck)
Security Compliance
- FedRAMP, HIPAA, SOC, etc. (check with the service complaint with regulations)
How do you cost / performance optimize Kubernetes app?
Kubernetes Cost
Control Plane cost (Fixed, not much room for improvement)
Worker Node number and types
- Pod resource specification
- Unused CPU / Memory allocation
Detect CPU / Memory waste - Utilize Metrics Server
- CloudWatch Container Insights
- Kubecost (Dollar amount for resource)
- CloudHealth
- Kubernetes Resource Report
- Cloudability (dubizzle use this)
Tell me about a challenge you faced in k8s?
There are many challenges is running k8s
Run lots of ops components
- Metric server, Prometheus, Thanos, CoreDNS, Harbor for docker cache, Log aggregation, Ingress, etc.
Upgrade Kubernetes cluster very three months
- It grunt task to upgrade kubernetes cluster very three months to upkeep with the upsteam
More challenging to get the Total cost of the application
- In VM based solution like Elastic Beanstalk, EC2, LB, Storage can be determined per app, in Kubernetes, there are pods, which we can get cost but there is shared resource like ALB, Shared Ops component, which makes estimating TCO tricky
K8s Upgrade story
- Updating the Kubernetes version
- Self managed - node Challange Drain the node Respect pod disruption budget, Keep the application up and running while node are updated
Solution
Use managed node group AWS manage all the heavy lifting
One Click update
How do you scale Kubernetes?
- Horizontal Pod Autoscaler (HPA)
- Cluster Autoscaler
- Cluster Over-provisioning (Real World App)
How do you expose a Kubernetes microservice to consumers?
Go over services
- Nodeport
- Load Balancer
- ClusterIP
Ingress is what is used in actual application
Name the component of the control plane?
- Kube-api-server
- etcd
- kube-scheduler
- kube-controller-manager
- cloud-controller-manager
Does Kubelet manage containers which aren’t created by Kubernetes?
No, kubelet doesn’t manage container which aren’t created by kubernetes
What is a running the container in the privileges mode mean?
This means that if you are root in a container you have the privileges of root on the host system. Is only meant for special cases such as running Docker in Docker and should be avoided.
What is the difference between replication controllers and replica sets?
The only difference between replication controllers and replica sets is the selectors. Replication controllers don’t have selectors in their spec and also note that replication controllers are obsolete now in the latest version of Kubernetes.
Is there any other way to update configmap for deployment without pod restarts?
well you need to have some way of triggering the reload. ether do a check every minute or have a reload endpoint for an api or project the configmap as a volume, could use notify to be aware of the change.
Depends on how the configmap is consumed by the container.
What is the ingress, is it something that runs as a pod or on a pod?
An ingress is an object that holds a set of rules for an ingress controller, which is essentially a reverse proxy and is used to (in the case of nginx-ingress for example) render a configuration file. It allows access to your Kubernetes services from outside the Kubernetes cluster. It holds a set of rules. An Ingress Controller is a controller. Typically deployed as a Kubernetes Deployment. That deployment runs a reverse proxy, the ingress part, and a reconciler, the controller part. the reconciler configures the reverse proxy according to the rules in the ingress object. Ingress controllers watch the k8s api and update their config on changes. The rules help to pass to a controller that is listening for them. You can deploy a bunch of ingress rules, but nothing will happen unless you have a controller that can process them.
If a pod exceeds its memory “limit” what signal is sent to the process?
SIGKILL as immediately terminates the container and spawns a new one with OOM error. The OS, if using a cgroup based containerisation (docker, rkt, etc), will do the OOM killing. Kubernetes simply sets the cgroup limits but is not ultimately responsible for killing the processes.SIGTERM
is sent to PID 1 and k8s waits for (default of 30 seconds) terminationGracePeriodSeconds
before sending the SIGKILL
or you can change that time with terminationGracePeriodSeconds in the pod. As long as your container will eventually exit, it should be fine to have a long grace period. If you want a graceful restart it would have to do it inside the pod. If you don’t want it killed, then you shouldn’t set a memory limit
on the pod and there’s not a way to disable it for the whole node. Also, when the liveness probe fails, the container will SIGTERM and SIGKILL after some grace period.
What is psp (Pod security policy)
Pod Security Policies (or PSPs) are objects that control security-sensitive aspects of pod specification (like root privileges).
If a pod does not meet the conditions specified in the PSP, Kubernetes will not allow it to start, and Rancher will display an error message of Pod is forbidden: unable to validate….
https://k8s-examples.container-solutions.com/examples/PodSecurityPolicy/PodSecurityPolicy.html
What difference between Daemonset and Side Car? Apart for it runs on each node, how characteristic are different? Scheduling, IP?
Deamonset = certain pod running on every nodeSidecar = container in the same pod
Deamonset = Own separate IP
Sidecar = Same IP on different port
Deamonset example = ingress controller, log shippers like Datadog, etc
Sidecar example = log collector like fluentd or fluentbit which exports the logs from disk, envoy proxy for network