Kubernetes the Container Orchestration tool
Hello Team,
Working Devops Support Engineer having 8.2 years of experience in Unix, Shell Scripting, SQL. Since one year i started my journey as a DevOps support engineer where i am involved in Deployments, Infrastructure monitoring using AWS- cloud-watch, Automating simple task using shell scripts.
cate work
Kubernetes is a container orchestration platform. Containers are ephemeral in nature. Which means containers can die and revive anytime. Suppose on top of docker engine there were 100 containers and scenarios are there where containers which are there it is impacting each other means one container in impacting the other. Another scenario is there somebody kills the container immediately the application running inside the container will not be accessible unless the devops engineer manually starts the services. Auto healing is a behavior where without users manual intervention your container should start by itself. This can be achieved by Kubernetes.
Suppose There is a Docker Container which has 4 GB RAM and 4 CPU. Suppose on certain occasion the number of users increased from 10,00 to 10,000 in that scenario incase we have manually create container named C2, C3 so on so forth and create an load balancers which will be balancing the load in between container C1 and C2. But in real-time where we are going to manage 10000 of containers it's tough to manage the whole system so we need Scalability or high performance which means we can scale our application fast when we have more load on it or more users trying to access it and scale it down when the load goes down. It's makes my application flexible in increase or decrease in load.
Another important feature of Kubernatis is Disaster recovery. Suppose the data is lost or server is explode then the infrastructure must have to have a mechanism where it will automatically backup the data and restore it to the latest date. So that the containerized application will run from the latest date after the recovery.
So Diffrence between Docker and Kubernetes is mentioned below.
- Autoscaling 2. Autohealing 3. Disaster Recovery 4. Enterprise Support
Generally Kubernetes is installed as a Cluster. The Cluster is nothing but a group of nodes which generally consists of master Node and another will be the Slave node(VM). The smallest unit in Kubernetes is called as pod. The pod is nothing but the abstraction over container. Pod basically creating the running environment or layer on top of the container. Its the smallest deployable units which has shared storage and network resources and a specification for how to run containers. Kubernatis is basically abstracts away the container run time or container technology so that i can replace them if i want to because i don't have to directly work with docker or whatever container technology use in a kubernetes. We only interact with the Kubernetes layer. In kubernetes each pod gets its own I.P address and each pod will communicate with each other thru their I.P address. its basically an internal i.p address not the public one.
Here pods are ephemeral in nature. Means the nodes or the server which i am running them on ran out of resources then the pod will die. In that case a new pod will get created in that place which will be assigned a new I.P address but in this case there will be inconvenient while communicating with the database because the i.p has changed. To solve that problem another component of Kubernatis named
Service:-- Service is basically a static i.p address with a DNS name. It acts as a Load balancer which can be attached to each pod. It is the core component in kubernetes that are used to manage networking and traffic flow within a cluster. Usually one application is running on one pod.
The good thing about service is the Lifecycle of Pod and service is not connected. Which means even if the pod dies the service and it's I.P address will stay.
External Service/Internal Service:— Suppose i want my application to access thru browser for that i need an external service. External service access the application through external sources like:-- http://node-ip:portnumber like http://124.89.101.2:8080
But here i want to see my application as secure protocol domain name. Shown below https://my-app.com . For that we have another component of Kubernetes called as Ingress. So instead of service the request goes to ingress it does forwarding to service. Ingress is a Kubernetes resource that defines a set of rules for routing external HTTP(S) traffic to a service. Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Ingress resources requires Ingress Controller to be deployed in the cluster, which is responsible for implementing the routing rules.
But for database i dont want to open communication for external requests. So for that i need internal service.
Configmap and Secrets:-
Kubernetes has several ways to configure applications including ConfigMaps, Environmental variable and secrets. Configmaps are environmental variable which is used to store configuration data as Key-value pair. I can create a ConfigMap with the desired configuration data and mount it directly to the pod with the configMapRef field.
Below is the details which i can perform using Config map.
Environmental variable can be stored in a ConfigMap and it can be used to configure the application.
I can store DB Configuration files in a configMap and mount them as volume the container.
‘envForm’ field is used to define environmental variable for a container based on a ConfigMap or Secret. I can specify the name of configMap or Secret using ‘configMapRef' or ‘SecretRef’ fields respectively.
envFrom
# Sample YAML configuration apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: web image: myapp:latest ports: - containerPort: 80 envFrom: - configMapRef: name: db-config - secretRef: name: db-secrets
Let’s understand with an example mentioned below:
In the below example we could see there are two pods where in one pod my-app named application is hosted where as in another pod DB application is hosted and now there is a database endpoint named mongo-db-service (URL)is the medium by virtue of which my-app gets connected with DB service. Suppose the endpoint gets changed to mongo-db then i might need to adjust the URL in the application which means to rebuild the application and push it to the repository and then Pull it in your pod and restart the whole thing.
After end-point got changed:--
Here endpoint changed to mongo-db from mongo-db-service and below is the process which is very hectic.
Below is process with Configmap where we just use the configmap where we don’t need to create new image and don’t have to go thru the cycle of as shown in above screen shot.
Secret:- -
It’s similar to ConfigMaps, but are used to store sensitive information(secret data such as passwords, tokens or API Keys, certificates). I can create a Secret with the desired sensitive information, and then reference it in your Pod specification using the ‘secretKeyRef’ field.
a) Kubernetes Secrets are by default stored unencrypted in the API Server’s underlying data store (etcd). Anyone with API access can retrieve or modify a secret, so can anyone can access to etcd. In order to safely use Secrets, below steps can be taken:-
Configure RBAC rules with least-privilege access to secrets with this we can limit access to sensitive data to only authorized users and applications. Enable encryption at Rest for Secrets.
Consider Using External Secret Store providers to secure Sensitive data before storing it in Kubernetes Secrets which will enhance security.(HashiCorp Vault, Azure Key Vault, and AWS Key Management Service). Restrict Secret access to specific containers.
Command to create a secret by running:—
kubectl create secret command
kubectl create secret generic db-secret - -from-literal=username=myuser —from-literal=password=mypassword. (This command will create a secret named db-secret with two key-value pairs: username and password)
Commands to get the secrets:— kubectl get secrets
Commands to Edit a Secret:- kubectl edit secret db-secret.
Commands to describe Secret:- - kubectl describe secret db-secret
To use secret in a pod i can mount it as a volume or use it as a environment variable
yaml
# YAML file for secrets
spec:
containers:
- name: my-container
image: my-image
volumeMounts:
- name: secret-volume
mountPath: /etc/myapp/secret
readOnly: true
volumes:
- name: secret-volume
secret:
secretName: db-secret
Data Storage and Volume Concepts:-
Kubernetes doesn’t manage data persistence to solve the problem concept of Volume comes to picture. Suppose my pod which contains database data and i want that data to be persistent even after restarting the pod. This can be achieved with another component of Kubernetes called Persistence Volume(PV). PVC (PersistentVolumeClaim) requests specific storage characteristics. It basically attaches a physical storage or hard drive to the pod That storage might be on local machine on the same server node where the pod is running or could be on remote storage which means outside of the k8s cluster. k8s binds PVC to matching PV.
Deployment & StatefulSet:--
Senario:-
Suppose in a scenario where one of the pod dies(which contains my-app) then there might be a case there user might need to face downtime which is not good. In this case it’s advisable we will have a replica or clone of our application would run.
Here the interesting thing is the replica or clone of the application is connected to the same service, Where the functionality of the service is it will catch the request and forward it to the pod which is less busy. The service acts a Permanent I.P address and also as a Load balancer. But in order to create a second replica of the pod i don’t need to create a new pod. I just need to define blueprint for the Pods and specify how many replicas you want to have and that component or blue print is called as deployment. In practice i will be creating deployments and there i will specify how many replicates and i can scale up or scale down the number of replicates of pods i need. So Pod is a layer or abstraction on top of container and deployment is another abstraction on top of pods, which makes it more convenient to interact with the pods, and replicate them and some other configuration. So in practice we will working mostly with deployments
So if one of the application hosted in the pod will die then service will forward the request to another one. So the application will still be accessible to the user. But suppose the database pod will die in that case the DB can’t be replicated via Deployment. The reason is database has a state, which is its data meaning
Suppose if we have clones of replica of the database, they would all need to access, the same shared data storage. There i would need some kind of mechanism that to track which pods are currently writing to that storage or which pods are reading from the storage in order to avoid data inconsistencies and that mechanism in addition to replicating features is offered by another Kubernetes Component called statefulset. A StatefulSet runs a group of pods, and maintains a sticky identity for each of those Pods. This is useful for managing application that need persistent storage or a stable unique network identity.
So MySql, mongoDB, elastic search or any other stateful application or databases should be created using stateful sets and not deployments. The statefulset like deployment would take care of replicating the pods and scaling them up or scaling them down and making sure that database read and writes are synchronized so that no database inconsistencies are offered. But Deploying database applications using StatefulSet in Kubernetes cluster can be somewhat tedious that’s why DBs are often hosted outside k8s cluster.
DaemonSet:-
A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.
Some typical uses of a DaemonSet are:
running a cluster storage daemon on every node
running a logs collection daemon on every node
running a node monitoring daemon on every node
Suppose i want to collect logs in our cluster from each pods so we need a application that would run on each node and collect logs that pods are generating there
Or think about Kube Proxy need to run on each and every cluster node. Lets say for these two applications and log collector and kube proxy we create a deployment and we set the replicate count : 2(number of node)
Suppose after couple of days we add two new nodes to the cluster there i need to adjust replica number: 4 so that kube proxy or log collector pod will also run on that two new nodes.
Suppose the next day we deleted one of the Nodes we need to adjust replica number again. Which means we need to remove the one replicate again otherwise we are going to have two kube-proxies or log collector pods running on one of the nodes. Also we can’t make sure that each node will get just one pod of the application and pods will be equally distributed among the nodes. So how can we solve all these issues for such applications ?
To solve that problem DaemonSet component has been introduced. Daemonset is a kubernetes component which automatically calculates how many replicas of the application it should deploy, depending on how many nodes you have in the cluster and it also deploys just one pod or one replica on each node in the cluster. when i add new node to the cluster it will automatically start adding pod replicate there and when i remove a node it will just remove that pod replica and those Pods are garbage collected. Advantage is no need to define replica count with daemonset component because it will automatically scale down or scale up the replica count depending on the replica count and guaranty only one replica per pod.
Lets Summarize the concepts:—
Main Kubernatis components are as below:--
i) Pod ii) Service iii) Ingress iv) ConfigMap v) Deployment vi) Secret vii) Statefulset
Pod blue print can be achieved using Deployment and StatefulSet. StatefulSet is used specifically for the stateful applications.
Kubernetes Architecture:—
A Kubernetes cluster consists of a control plane plus a set of worker machines, called nodes, that run containerized applications. Every cluster needs at least one worker node in order to run Pods. In production environment the control plane usually runs multiple computers and cluster runs in multiple nodes, providing fault tolerance and high availability.
The control plane's Node components make global decisions about the cluster (for example, scheduling, monitoring, restart the pod, join a new node), as well as detecting and responding to cluster events (for example, starting up a new pod when a Deployment's replicas field is unsatisfied).
There are four process which runs on every control plane node .
i) Api Server:— **API serve**r is like a cluster gateway which gets the initial request of any updates into the cluster or even the queries from the cluster, acts as a gatekeeper for authentication to make sure only authenticated and authorized requests get through the cluster. Suppose a user wants to schedule new pods, deploy new applications, query the status of deployment, cluster health, create new services or any other components you need to talk to the API server on the control plane node and then API server validates request and everything is fine it will forward the request to the other process in order to schedule the pod or create the component which the user requested.
ii) Scheduler:-- Suppose the user wants to schedule a new pod. The API server validates the request and it hand it over to scheduler and it will start the application pod in one of the worker nodes . Instead of randomly assigning to any nodes the schedule has the whole intelligent of deciding on which specific worker node the next pod will be scheduled or next component will be scheduled based on the resource utilization.
iii) Controller Manager:-- Detects cluster changes e.g if any of the pods die controller manager detects that and tries to recover the cluster state ASAP. Which means the Controller manager requests to the scheduler to reschedule those dead part by checking the resource calculation which, which worker node should restarts the pod again and makes request to the corresponding kubelets on those worker node to actually restart the pods.
etcd:--
Its also called as a Cluster Brain. Its the key value store of a cluster state. Suppose any new pods gets scheduled, or a pod dies, all these changes get saved or updated into this key value store of etcd. Now question is how does scheduler know what resources are available on a worker node or how does controller manager know that a cluster changed in some way, for example pods died or that Kubelet restarted new pods upon the request of a scheduler. Or you make a query request to a API server about the cluster health or, for example, your application deployment state, where does API server get all this state information from ? So all of this information stored in etcd cluster. This is just cluster state information which is used for control plane processes to communicate with the worker processes and vice versa.
Example of a Cluster Set Up:-
In the very cluster set-up we have 2 Control plane Nodes and 3 Worker Nodes. The hardware requirement for Control plane and Node resources differ. because the control plane process has lower work load so they need less resources but the worker nodes need more resources due to higher work load and more resources.
Suppose i want to add a new control plane/Node server. Below are the steps:- i) Get a new bare server ii) Install all the control plane process/worker node process iii) Join it to the cluster. For the Worker node you get the worker node process like container runtime, kubelet and kubeproxy on it and join it to the Kubernetes Cluster.
Let’s understand the architecture of Worker Node:—
Here each node has multiple Pods on it. Three process must be installed on every node i.e i) Kubelet, ii)Kube Proxy, iii) Container runtime.
Nodes are the cluster services that actually do the work. First one is container runtime are Containerd, Docker. The container runtime must be installed on every node. But the process which actually schedules the pods and the containers underneath is Kubelet which is a process of Kubernetes itself. Kubelet interact with both container and the node. Kubelet is responsible for taking that configuration and actually running a pod or starting a pod with container inside and actually assigning resources from that Node to that container like CPU, RAM & Storage.
The Way the communication happens between the Worker Nodes is Services, its a LB which catches the request and directed to the pod or the application like database forward it to the respective pod. But in between another process name Kubeproxy which has a intelligent forwarding logic inside that make sure that communication also works in a performant way with low overhead.
In the above Screen Shot my-app is making request to the database, the service instead of randomly sending request to any replica it will forward to the replica that is running on the same node as the pod that initiated the request. It will avoid the network overhead of sending request to other machine.
Then it will gonna try to fix it which is the basic feature that k8s provide.
Here i specify i need two replicas of nginx deployment so when i apply this, when i actually create the deployment using this configuration file that’s what apply means k8s add here the status of your deployment and it will update that state continuously. Suppose a status will say only one replica is running, then k8s will compare the status with the specification and there will know there is problem there another replica needs to be created ASAP. Where does k8s get this status data ? It’s from etcd.
etcd holds the current status of any k8s component.
Format of k8s configuration file:--
What is YAML ?
i) YAML is a data serialization language. YAML is a superset of JSON: any valid JSON file is also a valid YAML file.
In YAML its line separation and spaces thru indentation
Sample YML file:- deployed: off/on (Boolean value)
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx-container
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: nginx-volume
mountPath: /usr/nginx/html
- name: sidecar-container
image: curlimages/curl
command: ["/bin/bash"]
args: ["-c", "echo hello from the sidecar container; sleep 30"]
Suppose you want to see list within a list you can write:—
Here in the above i can apply list like above in square bracket.
Suppose in YAML you want to specify multiline string:
Suppose if you want to interpret as a single line.
Below is the sample YAML file where a multiline script which executes the shell command
In a single YAML i can define multiple components and i can separate these components using three dashes. The below is a valid YAML
TLS Certificate:—
TLS Certificate is the way to establish trust between client and server or user and application. The user/ client initiates the request and the server/application gets that request and replies to it and internet is used to exchange all types of data. There is an encryption key which encrypts the plain textdata(username/pwd) so that the user can’t read it and the same key is used to decrypt the the encrypted data because the server either can’t read the encrypted data that’s where Asymmetric encryption comes to picture. Here separate keys has been used to encrypt and decrypt the data also called as Key pair where you encrypt the data with one (symmetric) and can decrypt the data only with the key pair.
How its works ??
Client uses Symmetric Key to encrypt the data where asymmetric key pair is generated on the server. From the server generated asymmetric key pair at their end end sends one key to the client. Now client will encrypt it’s own symmetric key with the key which server sent it and sends its own symmetric key to the server . Now server will then use the other key to decrypt the data that client sent and extract the client’s symmetric key. Here the hacker can’t decrypt the data becoz it doesn’t have the other key. The key which the server sends to the client is called as Public key but the key that server only leaves itself to decrypt the client’s data is called as private key or its not shared with client or anymore .
Here different key is used to encrypt and decrypt the data then its called as Asymmetric Encryption. Also called as Private and Public key pair. Here the hacker won’t be able to see the data between the client and server and its not able to impersonate the client. But how can the client validate the public key which the server sent it really belongs to its online banking website and not a fake hacker website ? That’s where Certificates come in. The real Website get the public key signed and certified by CA. The CA is an entity that gives website owners digital certificates that certifies the ownership of that specific website or domain. So basically the owner of the real website and its domain asks an authorized party to sign their certificate as a proof that their public key is really for the website. mybank.com or facebook.com etc and this Authorized party or CA will issue a certificate which has a public key embedded and includes the name of the real website and its domain and a signature of the CA. If a website has subdomains or any other domains including the language domains for the website, all of them should be included in the certificate for the website, all of them should be included in the certificate for a client to validate them.
When you access the real website from the browser, the browser which is the client will check the certificate, make sure the certificate is valid and issued for the domain name and issued signature from authorized CA.But question is how does the browser know that signed the certificate is a legitimate one ? Here the Operating system has a list of such CAs already packaged inside. That’s where browser can verify whether the CA is trusted or not.
Create Keys Request Certificate:-
With the help of simple command line tools, such as OpenSSL, that give you commands to do it.
##Usuing the below command we can generate a private key and then using that key, we can generate a
# certificate signing request with the public key inside which we will then send to the CA for signing
openssl req -new -key xyz-bank.key -out great-bank.csr -subj "/C=US/ST=NY/O=GreatBankOrg, Inc.
/CN=great-bank.com
Moving to k8s Cluster:-
My team members discussed whether to use Managed Kubernetes Service from one of the clouds or go for a self managed k8s cluster where self-managed k8s cluster has more control and flexibility. So my task is to set up k8s cluster from scratch on the bare servers and then decide Cluster Size and configuration option.
In the k8s official website the min. system req is at at least 2G.B RAM and 2 CPUs per node. Now in your aws create an admin user with auto generated password and because the admin user is going to be the only one in the group so we can skip creation of group so we can assign a permission to the user directly. And for that, we are going to go to attach policies directly.
What’s VPC and why it should be created ?
VPC in AWS is basically my private network where it’s also called Virtual representation of network structure. VPC is my own isolated network in the cloud. Its created because i don’t want other AWS users or other AWS accounts to have access to my resources. VPC will be created in a specific AWS region.
Whenever i create an EC2 instance, it has to run inside a VPC or whatever component i am launching. Region in AWS has multiple AZs. Where Availability Zone = 1 or more data centers where VMs run in one of those AZ’s. Under VPC we have subnets where Subnets are subnetworks of the VPC. The VPC spans the whole region where the subnet span individual availability zones. Subnet is nothing but a private network inside the network. VPC has a range of I.P address
So whenever i create a new EC2 instance the I.P address will be assigned to these range
So the internal I.P address is not for outside web traffic. There are two i.p adds internal one communication inside the VPC and a public IP address for accessing it through the browser for allowing internet connectivity with your VPC. For allowing internet connectivity with your VPC we also have internet gateway component. Internet Gateway connects the VPC to the internet to the outside world so that i can get traffic inside my VPC to your webservers, and can send traffic outside if i want to download some stuff from internet on your components, etc.
There are some components inside the VPC service called network ACLs. which are basically firewalls rules for subnets. So i can configure this per subnet and in addition to that i have security groups which are again firewall rules but VM level
.
Installation details in k8s in the control plane & Worker nodes:- -
In the Control plane node control plane applications will be running such as k8s processes which are used for cluster operation, cluster administration.
Below are those items need to be installed on Control Plane & Worker nodes
1)Container runtime
2)Kubelet application (Installed from package repository)
3)Control plane:— Api server, scheduler, controller manager, etcd and Kubeproxy , where all the Control plane components deployed as Pods. When deploying a pod, we need to send a request to the API server the scheduler component will then decide where the pod should be scheduled. And also the pod data will be written into and updated in etcd store.
Two things we need to consider. Kubernetes Manifests file for each application and Deploy the application securely.
Static Pod:- These static pods are managed directly by managed by Kubelet daemon with out any control plane. Once we have the container runtime and Kubelet running, we can actually schedule the static pod. Generally for the regular pods API server gets request to schedule a pod scheduler decides which node it should be scheduled on and then API server will contact the Kubelet on that selected node that scheduler selected and tell Kubelet on that node to schedule the pod. So that’s the way for scheduling regular pods, But in addition to getting these commands from API server, Kubelet can actually schedule pods on its own. Question is how does it happen ?
Kubelet watches a specific location /etc/kubernetes/manifests on that Node it is running. If it finds the pod manifests there it will schedule it as a static pod no master process required
How the Static pod is different then others ?
Kubelet(Not the controller Manager) watches static pod and restarts if it fails. To recognize a static pod at the end of their name it’s suffixed with the node name they are running on.
FIRST STEP WHEN INSTALLLING K8s Cluster ?
→ We need to Generate static pods manifests for the API-Server, Controller manager, scheduler and etcd applications and put these manifests files into /etc/kubernetes/manifests folder where where kubelet will find and schedule them.
when deploying these applications we need to ensure that they run securely which can be done with certificates. The API server is the center of all the things and all the communication. So almost every other component of k8s will talk to API server will need to provide a certificate to be able to authenticate itself with the API server.
Generate self-signed CA certificate for k8s(“cluster root CA”) which will then use to sign all the client and server certificates for each component in the cluster, Certificates are stored in /etc/kubernetes/pki.
API server will have server certificate for its endpoint where as scheduler and controller manager will both have their client certificate to talk to the API server. Similarly Kubelet and Etcd will have their server certificate and here as API server is the client it need its own Client certificate to talk to the etcd and kubelets applications so that it can authenticate itself with them for that its can use its existing service certificate or alternatively we should create new client certificates specifically for talking to etcd and kubelet applications. Kubelet also talks back to the API server also uses a separate client certificate for that. Whenever an application talks to the API server, API server application doesn’t know whether that application is a legitimate application, here every client needs to be authorized and the same goes with etcd when API server connects to it etcd decides is this client authorized to talk to me and to ensure proper identification each component will get certificate signed by same certificate authority so now when scheduler connects to API server this the certificate which is a proof that i am part of the same cluster as you are therefore i should be allowed to talk to you. So that’s how the whole set up public key infrastructure PKI should work in K8s. So all these component will talk to each other to do thier jobs.
Admins also need to talk to API server so we also need our own client certificate for the end user to authenticate with the API server. These certicates need to signed by root CA.
So here creating manifests file and certificates for each and components manually is a hectic job. For that we have have built-in tool in k8s is called as kubeadm.
Kubeadm(kubeadm init & kubeadm join) is built tool which does the bootstrapping along with best-practices for a K8s cluster. It provides “fast paths” for creating k8s cluster. Performs the necessary actions to get a minimum viable cluster. It cares only about bootstrapping, not about provisioning machines.
In the below diagram we can see only incoming traffic is allowed on the port 22.
Opening Ports in K8s:—
In the below k8s official documentation port details are mentioned which are required to configure the control plane and worker nodes. Link
Container Runtime Interface:—
K8s only need the container runtime part of Docker it doesn’t actually need any other parts, so more lightweight runtimes were emerged such as CRI-O and containerd. These container runtimes are lightweight and still allow us to pull and run Docker images. Docker images are compatible with other container runtimes so i can create docker images for my applications and running them in a cluster that uses a container D runtime, and that will work just fine. EKS, AKS and GKE user container D as a container runtime in their managed K8s clusters.
Installation steps for Container Runtime:-
1)Install & Configure Pre-requisites:-
i) AWS infrastructure provisioning:-
In order to configure the security groups in the control plane node below are the details.
6443:- This port is by default where API k8s server is running by default. Below is security gp configurations for the k8s control plane.
For worker below are the default configurations.
The first rule is for the kubelet which is running on the control plane and worker nodes and Kubelet port need to be open because Kube API server from control plane node will talk to Kubelet on each of the worker node. That’s why we need to make it accessible
In the Prerequisites disable some swap mem and then install the below commands.
sudo apt update -y
sudo vim /etc/hosts
sudo hostnamectl set-hostname worker2
sudo swapoff -a
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# Apply sysctl params without reboot
sudo sysctl --system
Here now we will install package manager from linux which will install containerd.
The config.toml file is a configuration file for the containerd (we have to generate this configuration file for our containerd process). The file must be placed at /etc/containerd/config.toml.
The default configuration can be generated via
containerd config default > /etc/containerd/config.toml.
To see the configuration file click the Link
containerd config default | sudo tee /etc/containerd/config.toml (tee cmd takes std input stream and writes it to both the std output stream and a file stream).
ubuntu@controlplane:~$ sudo ls /etc/containerd
ls: cannot access '/etc/containerd': No such file or directory
ubuntu@controlplane:~$ sudo mkdir /etc/containerd
ubuntu@controlplane:~$ sudo ls /etc/containerd/config.toml
/etc/containerd/config.toml
ubuntu@controlplane:~$
In the configuration file we can change SystemdCgroup = true from false Which can be done by below command.
sudo sed -i ‘s/SystemdCgroup \= false/SystemdCgroup \= true/g’ /etc/containerd/config.toml
Check the status:-
sudo systemctl status containerd
sudo systemctl status containerd
Active: active(running) since 2025-XX-XX
sudo systemctl restart containerd
Now Below is the script which we can create in the worker nodes to set up the configuration. Name the script as install-containerd.sh
install-containerd.sh
ubuntu@worker1:~$ cat install-containerd.sh
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# systemctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/systemctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
#Apply systemctl params without reboot
sudo sysctl --system
sudo apt-get update
sudo apt-get install -y install containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
Installing Kubelet, Kubeadm and Kubectl in the k8s cluster:--
Kubelet:- Its a process which is required to run pods on the control plane as well as worker node.
Kubeadm:-- Command line to used to initialize the cluster and create the cluster. We are going to use kubeadm tool once at the beginning to initiate the cluster. Install kubeadm on each node, execute
kubeadm init… once(Used to initialize the control plane Node). Control plane nodes are just empty servers we turned them into control plane node by executing kubeadm init on them. This will generate /etc/kubernetes folder and inside that we will put all the generated certificate all the k8s manifest file for k8s it will create all the necessary configurations for these applications like API server, Kubelet, etc. and so on. When the manifest files get generated and put into the /etc/kubernetes/manifests folder kubelet, which looks for any static pod manifest in that location will detect them and start those static pods with the suffix of the hostname. Kubelet will ask container d runtime to fetch all the images for the applications in those manifests and to eventually schedule them as containers. So at the end, we will have all the process up and running and K8s cluster will have been initialized with just one node so we are gonna have a one cluster basically. But kebeadm doesnot install kubelet as well
Kubectl:- Its a command line tool that we used to work with a cluster once it has been created. Its the main k8s tool that we will be working with to create k8s components, get information about these componenets etc.
Note:— I need to install same version for kubelet and kubectl as the k8s control plane.
Link for installing kubeadm in k8s cluster (1.28 version we are installing)
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
//downloads google cloud public signing key. this is what our package manager will need to connect to the repository. and finally we add the repository to our package manager’s list//
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
After then we can check which versions are already available using below command.
Commad for installing:—
sudo apt-get install -y kubelet=1.28.0-1.1 kubeadm=1.28.0-1.1 kubectl=1.28.0-1.1 (Here we are installing #1.28.0-1.1)
sudo apt-mark hold kubelet kubeadm kubectl (It will mark hold for each of the packages from being automatically updated).
Similarly install kubeadm, kubectl and kubelet in rest two worker nodes.
ubuntu@ip-172-31-36-119:~$ kubeadm --version
unknown flag: --version
To see the stack trace of this error execute with --v=5 or higher
ubuntu@ip-172-31-36-119:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.0", GitCommit:"855e7c48de7388eb330da0f8d9d2394ee818fb8d", GitTreeState:"clean", BuildDate:"2023-08-15T10:20:15Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}
ubuntu@ip-172-31-36-119:~$ kubectl version
Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?
ubuntu@ip-172-31-36-119:~$ kubelet --version
Kubernetes v1.28.0
ubuntu@ip-172-31-36-119:~$ sudo apt-get install -y kubelet=1.28.0-1.1
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
kubelet is already the newest version (1.28.0-1.1).
So basically we have kubeadm which will basically give some commands to schedule pods, and we have kubeadm and kubectl command line tools installed on each of the nodes.
Initializing Cluster with kubeadm:-
Here we need sudo command becoz sudo will give permission to write the generated configurations files into /etc/kubernetes folder which is only accessible for the root user so we need the root permission to execute the init command. sudo kubeadm init basically bootstrapping the whole cluster from the background.
ubuntu@controlplane:~$ sudo kubeadm init
I0118 15:31:01.721798 20180 version.go:256] remote version is
much newer: v1.32.1; falling back to: stable-1.28
#Below is the preflight mode is downloading all the images that are needed to start the k8s components
# and to set up a cluster in /etc/kubernetes/pki all the generated certificates will be placed
[init] Using Kubernetes version: v1.28.15
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
#Below is for generating self-signed CA to set up identities for each componenet in the cluster
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [controlplane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.33.153]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
#Below apiserver.crt and apiserver.key has server certificate and private key which is used for others
#clients to connect and authenticate api server and client certificates apiserver-etcd-client.key to talk to
#etcd and kubelet ca.crt & ca.key are self signed CA k8s certificates which will signs the other certicates
ubuntu@controlplane:~$ ls /etc/kubernetes/pki
apiserver-etcd-client.crt apiserver-kubelet-client.key ca.crt front-proxy-ca.crt front-proxy-client.key
apiserver-etcd-client.key apiserver.crt ca.key front-proxy-ca.key sa.key
apiserver-kubelet-client.crt apiserver.key etcd front-proxy-client.crt sa.pub
#the below etcd folder has its own certificates authority and it has its own server certificate becoz
#etcd doesn't talk to anyone else apart from thier peers
ubuntu@controlplane:~$ ls /etc/kubernetes/pki/etcd
ca.crt ca.key healthcheck-client.crt healthcheck-client.key peer.crt peer.key
server.crt server.key
#Basically for everyclient need to talk to API server kubeconfig files get created which we can see
#in /etc/kubernetes folder Here in /etc/kubernetes folder we have all this kubeconfig files for each of the
#clients which are connecting to the API server.
ubuntu@controlplane:~$ ls /etc/kubernetes
admin.conf controller-manager.conf kubelet.conf manifests pki scheduler.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
#Below details suggest it writes kubelet settings and restart the kubelet. kubeadm generated a
#kubelet for configuration and puts it in /var/lib/kubelet/ as well as generated environment file
# and then starts the kubelet service
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
#The below cmd will the check the kubelet service status
ubuntu@controlplane:~$ service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2025-01-18 15:31:47 UTC; 35min ago
Docs: https://kubernetes.io/docs/
Main PID: 21062 (kubelet)
Tasks: 12 (limit: 4676)
Memory: 34.1M (peak: 35.1M)
CPU: 23.600s
CGroup: /system.slice/kubelet.service
└─21062 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubel>
#The below is the logs which are generated
Jan 18 16:06:43 controlplane kubelet[21062]: E0118 16:06:43.624879 21062 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false re>
Jan 18 16:06:48 controlplane kubelet[21062]: E0118 16:06:48.626138 21062 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false re>
Jan 18 16:06:53 controlplane kubelet[21062]: E0118 16:06:53.627296 21062 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false re>
Jan 18 16:06:58 controlplane kubelet[21062]: E0118 16:06:58.627950 21062 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false re>
Jan 18 16:07:03 controlplane kubelet[21062]: E0118 16:07:03.629233 21062 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false re>
#Below are added as regular pods because the control plane pods are already running so
#they can take over
#scheduling pods. One of them need to be kubeproxy which we know again we need on each
#node so kubeproxy
#needs to be on the control plane as well another one is
#CoreDNS(Installs a DNS server and kube-proxy component
#via API server
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
#Manifests files will be generated and put into below manifest folder
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
#Below Kubelet has its own configuration in the below /var/lib/kubelet/ folder
ubuntu@controlplane:~$ sudo ls /var/lib/kubelet
config.yaml cpu_manager_state device-plugins kubeadm-flags.env
memory_manager_state pki plugins plugins_registry pod-resources pods
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
ubuntu@controlplane:~$ ls /etc/kubernetes/manifests
etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
So after certificates gets generated the kubeconfig files are ready to make it possible for applications to talk to API server, as well as kubelet starts, kubeadm will create the manifest files so that static pods can be created, which will be scheduled by kubelet and they will use kubeconfig files with the certificate information to talk to each other or talk to API server.
Kube-Proxy is a network proxy which runs on each node
#Lets find some intrsting facts in the apiserver manifests files
ubuntu@controlplane:~$ sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | less
#Here we can see server certificates for the API server and these are passed in to the kube API server
#application usuing these parameters
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
#Here below are the pointers for client certificate to connect to etcd r8 below
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
#Below is the endpoint of the etcd server and 2379 is the port where etcd server will run and be accessible
#
--etcd-servers=https://127.0.0.1:2379
#Below is the kubelet certificate location along with its key file location
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
ubuntu@controlplane:~$ sudo cat /etc/kubernetes/manifests/etcd.yaml | less
#below is the ca certificate
--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
#below is the server certificate of etcd
--cert-file=/etc/kubernetes/pki/etcd/server.crt
#below is the config for peers. So basically if we had more than one city in the cluster for high
#availability on other control plan nodes for example then we would have an etcd cluster and this
#would be the configuration for those etcd pods talking to each other and connecting each other as peers.
--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
--peer-client-cert-auth=true
--peer-key-file=/etc/kubernetes/pki/etcd/peer.key
--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
#Below is the location where data is actually stored
--data-dir=/var/lib/etcd
#Below is to check the configurations of the kubelet itself and check kubelet certificates etc
#everything is configured in /var/lib/kubelet/pki directory. because Kubelet runs as process not as a pod
ubuntu@controlplane:~$ sudo ls /var/lib/kubelet/pki
kubelet-client-2025-01-18-15-31-38.pem kubelet-client-current.pem kubelet.crt kubelet.key
ubuntu@controlplane:~$
sudo cat /var/lib/kubelet/config.yaml
#Below is the static pod path where by default is configured for kubelet at /etc/kubernetes/manifests
staticPodPath: /etc/kubernetes/manifests
So for troubleshooting or changing each parameters for how api server runs this is basically how you can override them and change them and if you change something in the configuration and then save that file kubelet will actually see that the changes has been made inside these manifests folder becoz it watches for any changes inside and will restart the pod with those changes.
Note: Kubelet runs as process not as a pod
But still there are some questions popping up in my mind ..
kubectl:- Command to connect to the cluster
Get information about cluster nodes:- kubectl get node
Kubeconfig:- Authenticate with cluster
#connected to the cluster and authenticate it usuing the admin.conf
ubuntu@controlplane:~$ sudo kubectl get node --kubeconfig /etc/kubernetes/admin.conf
NAME STATUS ROLES AGE VERSION
controlplane NotReady control-plane 3h20m v1.28.0
The Certificate authority data is the self signed CA that kubeadm generated for the cluster.
Here in the above screenshot the server: https://172.31.33.153:6443 its the private i.p address of the control plane node
The client certificates created for the admin and it was directly put in here because we don’t need it anywhere else as well as the controller manager and scheduler and kubelet they have their client certificates directly in those kubeconfig files.
So we have multiple cluster and multiple users here that we connect to using the same kubectl tool we could switch between those clusters and users using those contexts. Using context to choose one of the clusters we want to connect to from the cluster list and using one of the users from user list.
Now we can set kubeconfig file location as an environmental variable
export KUBECONFIG=/etc/kubernetes/admin.conf
ubuntu@controlplane:~$ sudo -i
root@controlplane:~# export KUBECONFIG=/etc/kubernetes/admin.conf
root@controlplane:~# kubectl get node
NAME STATUS ROLES AGE VERSION
controlplane NotReady control-plane 3h57m v1.28.0
root@controlplane:~#
We can use this script in the worker nodes to install containerd services,.
But problem is Every time you restart the machine you need to set the environmental variable.
#id -u (id of the current user) id -g(id of the current group)
ubuntu@controlplane:~$ mkdir -p ~/.kube
ubuntu@controlplane:~$ sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config
root@controlplane:~#
root@controlplane:~# ls -l ~/.kube/config
-rw------- 1 root root 5649 Jan 18 19:34 /root/.kube/config
ubuntu@controlplane:~\( echo \)(id -u)
1000
ubuntu@controlplane:~\( echo \)(id -g)
1000
#We are making current user and current gp owner of that ~/.kube/config
ubuntu@controlplane:~\( sudo chown \)(id -u):$(id -g) ~/.kube/config
#I can see ubuntu user and ubuntu group owning the config file
ubuntu@controlplane:~$ ls -l ~/.kube/config
-rw------- 1 ubuntu ubuntu 5649 Jan 18 19:45 /home/ubuntu/.kube/config
ubuntu@controlplane:~$ ls -l ~/.kube/config
-rw------- 1 ubuntu ubuntu 5649 Jan 18 19:55 /home/ubuntu/.kube/config
ubuntu@controlplane:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
controlplane NotReady control-plane 45h v1.28.0
ubuntu@controlplane:~$
sudo kubectl get node --kubeconfig /etc/kubernetes/admin.conf
Namespace:-
Its basically a virtual cluster inside a k8s cluster. When i create a cluster by default k8s gives you namespace out of the box.
ubuntu@controlplane:~$ kubectl create namespace my-namespaces
namespace/my-namespaces created
#"default" namespaces is used as a default when executing kubectl commands
ubuntu@controlplane:~$ kubectl get namespaces
NAME STATUS AGE
default Active 45h
kube-node-lease Active 45h
kube-public Active 45h
kube-system Active 45h
my-namespaces Active 14s
ubuntu@controlplane:~$
#I want to see pods running on the system namespace below is the command
#Here it will specify all the pods running in kube-system namespace.
#Below represents name of the static pods becoz it has suffix of the name of the node where they're scheduled on.
ubuntu@controlplane:~$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5dd5756b68-cpmwk 0/1 Pending 0 47h
coredns-5dd5756b68-j4hc2 0/1 Pending 0 47h
etcd-controlplane 1/1 Running 1 (119m ago) 47h
kube-apiserver-controlplane 1/1 Running 1 (119m ago) 47h
kube-controller-manager-controlplane 1/1 Running 1 (119m ago) 47h
kube-proxy-rd77v 1/1 Running 1 (119m ago) 47h
kube-scheduler-controlplane 1/1 Running 1 (119m ago) 47h
Another way of creating namespaces is using namespace configuration file.
Q. Why the need of k8s namespaces ?
In a complex application which has multiple deployments which create replica of many parts and you have
resources like services and configmaps, etc very soon the default namespaces will be filled with different components and it will be really difficult to have an overview of what’s in there, especially if you have multiple users creating stuff inside. Better way to use namespace in this case is to group resources into namespaces. For example you have a database where i deploy my dbase and all its required resources. And a monitoring namespace where i deploy the Prometheus and all the stuff that it needs, and elastic stack namespace where all the elastic search, kibana etc resources go and i can have NGINX-Ingress resources so just one way of logically grouping my resources inside the cluster.
Another used case where we use namespaces if you have multiple teams, Suppose two teams that use the same cluster and one team deploys an application which is called my app deployment(It’s name of the deployment they create) and deployment has its certain configuration. Suppose another team had a deployment that accidentally had the same name, but a different configuration if they applied it which will overwrite the first team’s deployment. So to avoid such kind of conflicts again we can use namespaces so that each team can work in their own namespace without with out disrupting the other.
Suppose in a single k8s cluster i want to host staging and development environment in the same cluster where for both the staging and development both Nginx-Ingress Controller and Elastic stack used for the both. Here instead of creating two separate cluster and deploying i can deploy it in one cluster and use it for both environments. Here in the below staging and development environment can use both resources.
Use Cases of Namespace in Blue/Green Deployment:-
Here we have two diff versions of production one that is active, which is production now, and another one that is going to be the next production version. The versions of the applications in those blue and green namespaces will be different. Here both the resources might need to use the same resources like, NGINX controller or Elastic Stack. Here they can both use this common shared resources without having to set up a separate cluster
Access and Resource Limit on Namespaces:-
Suppose we have two teams working in the same cluster and each one of them has their own namespace. So what i can do in this scenario is that i can give teams access to only their namespace so they can only be able to create updates, delete resources in thier own namespace, but they can’t do anything in the other namespaces i can minimize the risk of one team accidentally interfering with another team’s work. So each one have their own secured isolated environment.
ii) Limit the resources that each namespaces consumes like CPU, RAM, Storage as per NS. Per namespace i can define resource quota that limit how much CPU,RAM storage resources one namespace can use.
Limitations of Namespaces:-
Suppose i have a configmap in project A namespace which represents the database service i can’t use the configmap in the Project B instead i have to create the same configmap which also references the database service. Each namespace must define own configMap even if its same reference and same apply secrets suppose i have credentials to shared service here i need to create secrets for each namespace where i gonna need that.
But there some components in k8s which are not namespaced because they live globally in a cluster and i cant isolate them in a certain namespace e.g Volume, Persistent volume and Node. When i create a volume it will be accessible through out the whole cluster and i can list them using command:—
kubectl api-resources —namespaced=false /true
Configure Network Plugin Cilium:—
The Pod Network IP Address range should not overlap with Node IP Address Range. Our Node gets its IP address from the VPC, our private network on AWS. So by checking the VPC service we can see its IP address range, which is 172.31.0.0/16 (Node Ip range).
When we observe the cilium documentation we could see 10.0.0.0/8 is the default pod CIDR.
https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/
Here in the above we could see 10.0.0.0/8 is the default pod CIDR. Which means from 10.0.0.0 to 10.255.255.255 this many numbers of I.P address will be available which is around 16,777,216 Million I.P address range. Here pod network CIDR block and the node CIDR block do not overlap in our case. We can actually go ahead and use the default I.P range of psyllium.
To install cilium we will follow the quick installation method. The Cilium quick install actually consists of an CLI which connects to the same cluster that Kube ctl is connected to As long as we have the Kube ctl command set up the cilium quick install will work and it also lets us run different tasks besides just installing the CNI so once we have installed it we can actually run some validations commands to make sure everything is set up properly in terms of networking in k8s.
https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/
To install cilium CLI we will copy the comments from the docs and we are just going to run them on our control plane node.
cmds:—
CILIUM_CLI_VERSION=\((curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "\)(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/\({CILIUM_CLI_VERSION}/cilium-linux-\){CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-\({CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-\){CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
Cilium actually using the templating tool called Helm, which allows us to pass different parameters to the installation tool, which means we can use different values other than the default ones provided for us.
Below is the Helm documentation for cilium
cilium install --set ipam.operator.clusterPoolIPv4PodCIDRList="10.0.0.0/9" (Cilium in our cluster will use the CIDR block 10.0.0.0/9 as the pod CIDR range.
ubuntu@controlplane:~$ cilium install --set ipam.operator.clusterPoolIPv4PodCIDRList="10.0.0.0/9"
ℹ️ Using Cilium version 1.16.6
🔮 Auto-detected cluster name: kubernetes
🔮 Auto-detected kube-proxy has been installed
ubuntu@controlplane:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controlplane Ready control-plane 16d v1.28.0
1/1 Running 0 72s
ubuntu@controlplane:~$ kubectl get pods -n kube-system
coredns-5dd5756b68-cpmwk 1/1 Running 0 16d
coredns-5dd5756b68-j4hc2 1/1 Running 0 16d
#Below are the two new pods such as cilium and cilium operator on our cluster control plane node.
#Since we have only one node in the cluster we have one pod of both cilium and its operator that basically
#manage the network within the cluster,
ubuntu@controlplane:~$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
cilium-2tdtv 1/1 Running 0 72s
cilium-operator-68bddc6cc7-jrvwb 1/1 Running 0 72s
ubuntu@controlplane:~$ kubectl describe pod coredns-5dd5756b68-cpmwk -n kube-system
Name: coredns-5dd5756b68-cpmwk
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: coredns
Node: controlplane/172.31.33.153
Start Time: Tue, 04 Feb 2025 14:09:07 +0000
Labels: k8s-app=kube-dns
pod-template-hash=5dd5756b68
Annotations: <none>
Status: Running
IP: 10.0.0.109
IPs:
IP: 10.0.0.109
#Above is the IP Address range for the core DNS pod, which is in the range of the CIDR
#block that we defined for Cilium. But doing this for every pod in the cluster is
#too much work and also we dont have nice overview of list of pods with thier IP addresses.
#Cilium pods after cilium got deployed have Ip addresses from the range of the cilium's pod
#network CIDR block.
coredns-5dd5756b68-cpmwk 1/1 Running 0 16d 10.0.0.109 controlplane <none> <none>
coredns-5dd5756b68-j4hc2 1/1 Running 0 16d 10.0.0.132 controlplane <none> <none>
#Other pods such as the static pods as well as Kubeproxy and the Cilium pods have a different I.p adds
#I.P adds range of the cntrl plane node they are running o
etcd-controlplane 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
kube-apiserver-controlplane 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
kube-controller-manager-controlplane 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
kube-proxy-rd77v 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
kube-scheduler-controlplane 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
#Internal I.p Address which has same output
kube-scheduler-controlplane 1/1 Running 3 (43m ago) 16d 172.31.33.153 controlplane <none> <none>
ubuntu@controlplane:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
controlplane Ready control-plane 16d v1.28.0 172.31.33.153 <none> Ubuntu 24.04.1 LTS 6.8.0-1021-aws containerd://1.7.12
We have our control plane processes running and we have a pod network deployed in the cluster. Below is the example of one node cluster where we have a control plane node So its time to join
Now lets join the worker nodes and make them part of the cluster and installing the pod network plugin was needed to add the worker nodes to the cluster. Otherwise we wont be able to join them and create pod networks on them.
For worker nodes we already installed containerd, kubeadm, kubelet and kubectl on both the worker nodes.
Inorder to join the kubeadm to the cluster we can use the command:-
kubeadm join [api-server-endpoint] —token XXXX (cmd initializes a k8s worker Node and Joins it to the cluster and an automatically generated secret token so that control plane components can identify the worker and basically let it join the cluster).
ubuntu@controlplane:~$ kubeadm token create --help
-h, --help help for create
-print-join-command Instead of printing only the token, print the full 'kubeadm join' flag needed to join the cluster using the token.
-ttl duration The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire (default 24h0m0s)
#kubeadm join api-server-endpoint flags
kubeadm join 172.31.44.88:6443 --token 6js7ff.gbu7du7zearrlzg
#Suppose i forget the token
ubuntu@controlplane:~$ kubeadm token --help
Available Commands:
create Create bootstrap tokens on the server
delete Delete bootstrap tokens on the server
generate Generate and print a bootstrap token, but do not create it on the server
list List bootstrap tokens on the server
# print-join-command Instead of printing only the token,
#print the full 'kubeadm join' flag needed to join the cluster using the token.
#172.31.33.153- > api server ipadds 6443:port details
#4wj6gf.zxjfrnfqc06ozcrt - > newly generated token
#sha256:e1f0a5779609c172b92d37219fe1ebdfdc3b82686684ab - > CI certificate Hash
kubeadm join 172.31.33.153:6443 --token 4wj6gf.zxjfrnfqc06ozcrt
--discovery-token-ca-cert-hash sha256:e1f0a5779609c172b92d37219fe1ebdfdc3b82686684ab (copy this and)
#Run it in the worker node
To check the details in the control plane i can execute the command
ubuntu@controlplane:~$ kubectl get node
NAME STATUS ROLES AGE VERSION
controlplane READY control-plane 14h v1.28.0
worker1 READY <none> 4h55m v1.28.0
# Worker1 which is in ready state which is the pod of the cluster of same k8s version
ubuntu@controlplane:~$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNAL-VERSION CONTAINER-RUNTIME
controlplane READY control-plane 24h v1.28.0 172.31.24.70 <none> Ubuntu 22.04.04 LTS
worker1 READY <none> 4h55m v1.28.0 172.31.22.191 <none>
ubuntu@controlplane:~$ kubectl get node -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system cilium-c9cqn 1/1 RUNNING 1 4h56m 172.31.22.191 worker1
<none>
kube-system kube-proxy-v7zf7 1/1 RUNNING 1 4h56m 172.31.22.191 worker1
<none> <none>
In the above example the worker node with 172.31.22.191 internal ip address we have two pods that were automatically started in the worker node the first one is Kube proxy and the 2nd one is cilium where both are daemon sets .
Daemon sets automatically schedule a pod on every single node in the cluster
Similarly lets execute the same kubeadm join command on worker node 2. We don’t need to add a new token for each worker node. We can actually reuse the same one.
Below is the cmd for Checking the kubelet service status
service kubelet status
ubuntu@worker2:~$ service kubelet status
Kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor present: enabled)
Active: active (running) since Fri 2024-05-10 13:47:07 UTC; 15s ago
And again on worker two we have the two daemon set pods, Kube Proxy and cilium
kubectl get pod -A -o wide
ubuntu@controlplane:~$ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system cilium-xswnp 1/1 Running 0 5m1s 172.31.23.63 worker2
<none> <none>
kube-system kube-proxy-km2m8
<none> <none> 1/1 Running 0 5m1s 172.31.23.63 worker2
As we already learnt with cilium which is a pod networking layer inside k8s the cilium pods on all the nodes should now be talking to each other and forming this network group so that we have a pod network in our cluster. And as we see, they are running in the cluster but we don’t know whether they are taking to each other nor not. Did they actually discover or find each other and did they form a group
# kubectl get pods -A | grep cilium this will print only the cilium pods with their names.
# list of pods running in the cluster
ubuntu@controlplane:~$ kubectl get pods -A | grep cilium
kube-system cilium-c9cqn 1/1 Running 1 5h4m
kube-system cilium-d6jb8 1/1 Running 1 21h
kube-system cilium-operator-6f744d4558-mbcxz 1/1 Running 1 21h
kube-system cilium-xswnp 1/1 Running 1 7m7s
ubuntu@controlplane:~$ kubectl -n kube-system exec cilium-xswnp -- cilium-dbg status
Cluster health: 1/3 reachable (2024-05-10T14:15:44Z)
Name
kubernetes/controlplane IP Node Endpoints
kubernetes/worker2 172.31.24.70 unreachable unreachable
kubernetes/worker2 1172.31.23.63 unreachable unreachable
Modules Health: Stopped(0) OK(11) Unknown(3)
As the above details we can see the networking plugin which gets deployed on every node so together these individual networking networking pods deployed across different nodes make 1 shared pod network.
In our case this networking application is cilium which listens on multiple ports for different features and for them to connect to each other they need to be able to talk to each other on this port.
cilium-operator-68bddc6cc7-jrvwb 1/1 Running 0 72s
In a VM to my AZure devops org
orgsett —> agent pool —>
capa —> add cap
,/run.sh —> in vm
piplin → crat piplin —> classic edi —>
Market place is the plugin section
token —> admin —> sec —→ user —> generate token
out Suppose container/pod1 is impacting pod2 in that scenario immediately Kubernatis put pod 2 in a another container.
There are several types of secrets in Kubernetes, including:
Opaque: Default secret type in Kubernetes, it can be used to store any arbitrary data
Suppose pod Where we have a master Node and another will be the Slave node.
