Deploy Metrics Server For PostgreSQL on K8S with Cronjob

Enterprise PostgreSQL Solutions

Comments are off

Deploy Metrics Server For PostgreSQL on K8S with Cronjob


Recently I had an opportunity to look into deploying PostgreSQL and pgpool on Kubernetes. The deployment is straightforward, but I also need to obtain the metrics information such as CPU and memory usages that each deployed pod is using. There are several ways to do this, but today I am sharing my way, which utilizes k8s’s native metrics server + cronjob to achieve.

This blog assumes you have already installed Kubernetes and deployed some services already (for my case, I have both pgpool and PostgreSQL running). You may need to refer to other posts about how to deploy them on Kubernetes.

initial setups

We have a pgpool pod running inside a deployment, a standalone pod running PostgreSQL as primary node and another deployment set up to scale up or down replica as needed (initially scaled to 2 replica pods). All these are deployed in a simple k8s cluster, minikube in my machine. We will be adding a cronjob block that can periodically checks CPU usage of each primary and replica nodes and perform custom tasks should threshold exceeded. They can be visualized as:

$ kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
postgres-primary                               1/1     Running   0          1m
postgres-replica-deployment-69f995884b-72v6g   1/1     Running   0          9s
postgres-replica-deployment-69f995884b-kkkn5   1/1     Running   0          9s
pgpool-59b77dbc76-6nr45                        1/1     Running   0          1m
$ kubectl get deployment
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
postgres-replica-deployment   2/2     2            2           16m
pgpool                        1/1     1            1           1m

set up k8s metric server

To get the metric data from k8s, we need to setup metric server first:

$ wget

for demonstration purpose, I do add one line into components.yaml to disable strict TLS checking. Simply add --kubelet-insecure-tls under spec-args under the pod template for metrics-server


apiVersion: apps/v1
kind: Deployment
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
      k8s-app: metrics-server
      maxUnavailable: 0
        k8s-app: metrics-server
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls ####### <====== line added here to disable TLS checks


check metrics server is installed:

$ kubectl get deployment metrics-server -n kube-system
metrics-server   1/1     1            1           15d

then we should be able to get metrics from each pod

$ kubectl top pods
NAME                                           CPU(cores)   MEMORY(bytes)
postgres-primary                               9m           63Mi
postgres-replica-deployment-69f995884b-72v6g   7m           48Mi
postgres-replica-deployment-69f995884b-kkkn5   7m           46Mi
pgpool-59b77dbc76-6nr45                        2m           143Mi

Note that the unit of CPU usage is in millicores, which equals 1000 per CPU core you have physically. For example, my PC has a 8 core CPU, so I have in total 8000 millicores available.

allocate max CPU usage per pod

To get correct reading of CPU usage, we need to specify a maximum CPU resource a pod can use. I find the value of 400m is suitable for my needs:

Follow this example manifest to set CPU limits to 400m:

apiVersion: v1
kind: Pod
  name: mypod
    - name: mycontainer
      image: myimage:latest
          cpu: 400m

create a cpumon deployment for CPU monitoring

Now, we have everything needed to define a CPU monitoring cronjob on k8s. This is called cpumon in this example. We will deploy it next:

  1. Prepare a config file for running k8s cluster (minikube), and example TLS certificate, CA certificate and private key files. All these are needed for cpumon to communicate with k8s controller to get metrics information. You can find the config file for your cluster with this command:
$ kubectl config view
  1. create a new folder called scripts and specs
  2. save the contents of the output into a separate file, called my-cluster.yml and put it in specs folder
  3. make a copy of CA certificate, client certificate and keys as specified in the output and put them in specs folder as well. For most cases they are located in /home/$USER/.minikube/ca.crt, /home/$USER/.minikube/profiles/minikube/client.crt, and /home/$USER/.minikube/profiles/minikube/client.key
  4. modify the my-cluster.yml file, and change all the path prefix to /k8sconfig. For example:
apiVersion: v1
- cluster:
    certificate-authority-data: DATA+OMITTED
  name: kind-kind
- cluster:
    certificate-authority: /k8sconfig/ca.crt
    - extension:
        last-update: Thu, 20 Apr 2023 10:37:33 PDT
        version: v1.30.1
      name: cluster_info
  name: minikube
- context:
    cluster: kind-kind
    user: kind-kind
  name: kind-kind
- context:
    cluster: minikube
    - extension:
        last-update: Thu, 20 Apr 2023 10:37:33 PDT
        version: v1.30.1
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
- name: kind-kind
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED
- name: minikube
    client-certificate: /k8sconfig/client.crt
    client-key: /k8sconfig/client.key

6. prepare a simple script that will be run when cron triggers and place it in scripts folder. Remember to run chmod 755 on the script. For example:


# this is the max cpu assigned to each pod

echo "cpumon: total cpu millicore = $total_millicore"
for pod in $(kubectl get pods | grep "primary\|replica" | awk '{print $1}'); do
        pod_millicore=$(kubectl top pods $pod | awk 'NR!=1{print $2+0}')
        pod_cpu_percent=$(awk "BEGIN {printf \"%.2f\", ${pod_millicore}/${total_millicore} * 100}")
        echo "cpumon: $pod uses $pod_cpu_percent% CPU (${pod_millicore}/${total_millicore} millicores used)"
exit 0
  1. create configMap object for scripts and specs
$ kubectl create configmap cpumon-scripts --from-file=./scripts
$ kubectl create configmap k8s-config --from-file=./specs
  1. create the cpumon cronjob with this example manifest below. This cpumon is run within bitnami/kubectl utility, which contains kubectl utility to access pod information from k8s controller. The files in specs and scripts will be passed into the pod as volume via configMap. Note that environment variable KUBECONFIG has to point to the my-cluster.yml that we have just created.

Save this manifest as cpumon.yml

apiVersion: batch/v1
kind: CronJob
  name: cpumon
  schedule: "* * * * *" # run every 1 minutes
          - name: cpumon
            image: bitnami/kubectl
            command: ["/cpumon/"]
            - name: KUBECONFIG
              value: "/k8sconfig/my-cluster.yml"
            - name: cpumon-scripts
              mountPath: "/cpumon"
            - name: k8s-config
              mountPath: "/k8sconfig"
          - name: cpumon-scripts
              name: cpumon-scripts
              defaultMode: 0755
          - name: k8s-config
              name: k8s-config
          restartPolicy: Never
      ttlSecondsAfterFinished: 1800
  1. deploy cpumon by
kubectl apply -f cpumon.yml

examine cronjob runs

Every minute, a pod will be started to run the script and exit. We can see their execution history by looking at the pod status below:

NAME                                           READY   STATUS      RESTARTS   AGE
postgres-primary                               1/1     Running     0          142m
cpumon-28075474-hjx4l                          0/1     Completed   0          2m42s
cpumon-28075475-qw9b7                          0/1     Completed   0          102s
cpumon-28075476-8nlgv                          0/1     Completed   0          42s
postgres-replica-deployment-69f995884b-72v6g   1/1     Running     0          99m
postgres-replica-deployment-69f995884b-kkkn5   1/1     Running     0          99m
pgpool-59b77dbc76-6nr45                        1/1     Running     0          127m

We can pick any of these instances of cpumon to check their log output.

$ kubectl logs cpumon-28075476-8nlgv
cpumon: total cpu millicore = 400
cpumon: postgres-primary  uses 2.00% CPU (8/400 millicores used)
cpumon: postgres-replica-deployment-69f995884b-72v6g  uses 2.00% CPU (8/400 millicores used)
cpumon: postgres-replica-deployment-69f995884b-kkkn5  uses 2.00% CPU (8/400 millicores used)

Having this basic infrastructure setup, we can then customize further to perform certain actions when certain threshold is exceeded. For example, scale up or down replica deployment. This would be the topic for next time.