View Certificate Details
In this tutorial, we are going to discuss on how we can view certificate details in an existing Kubernetes cluster.
So you join a new team to help them manage their Kubernetes environment. Your new administrator to this team is being told that there are multiple issues related to certificates in the environment.
So you’re asked to perform a health check of all the certificates in the entire cluster. What do you do?
First of all it’s important to know how the cluster was set up. There are different solutions available of deploying a Kubernetes cluster and they use different methods to generate and manage certificates.
If you were to deploy a Kubernetes cluster from scratch you generate all the certificates by yourself as we did in the previous tutorial or else if you were to rely on an automated provisioning tool like kubeadm, it takes care of automatically generating and configuring the cluster for you.
While you deploy all the components as native services on the nodes in the hard way, the kubeadm tool deploys these as PODs. So it’s important to know where to look at to view the right information.
In this tutorial, we are going to look at a cluster provisioned by kubeadm as an example. In order to perform a health check, Start by identifying all the certificates used in the system.
So the idea is to create a list of certificate files used, their paths, the names configured on them, the alternative names configured if any the organization the certificate account belongs to the issue of the certificate and the expiration date on the certificate.
Get list of certificate files
So how do you get these? Start with the certificate files used. For this, in an environment setup by kubeadm. Look for the Kube API Server definition file under /etc/kubernetes/manifests folder.
The command used to start the API server has information about all the certificates it uses. Identify the certificate file used for each purpose and note it down.
Next, take each certificate and look inside it to find more details about that certificate.
For example, we will start with the API Server certificate file. Run the openssl x509 command and provide the certificate file as input to decode the certificate and view details.
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
Start with a name on the certificate under the subject section. In this case its kube-apiserver. Then the alternative names. The Kube API Server has many.
So you must ensure all of them are there and then check the validity section of the certificate to identify the expiry date and then the issuer of the certificate. This should be the CA who issued the certificate.
Kubeadm names the Kubernetes CA as Kubernetes itself. Follow the same procedure to identify information about all the other certificates.
Things to look for check to make sure you have the right names, the right alternate names make sure the certificates are part of the correct organization and most importantly you are issued by the right issuer and that the certificates are not expired.
The certificate requirements are listed in detail in the Kubernetes Documentation page.
Inspect Service Logs
When you run into issues you want to start looking at logs. If you set up the cluster from scratch by yourself and the services are configured as native services in the OS, you want to start looking at the service logs using the operating systems logging functionality.
In case you setup the cluster with kubeadm, then the various components are deployed as PODs. So you can look at the logs using kubectl logs command followed by the pod name.
Sometimes if the core components such as the Kubernetes API Server or the ETCD server are down, the kubectl commands wont function. In that case you have to go one level down to docker to fetch the logs.
List all the containers using the docker ps –a command. And then view the logs using docker logs command followed by the container ID.