Authored by Stas Kondratiev. Edited by Nanette Ray. Reviewed by Ben Hummerstone.
To gather metrics about your Kubernetes cluster in Azure, you can collect telemetry from the nodes and store it in Graphite, InfluxDb, or another time-series database. This article describes how to use collectd, a popular daemon that collects system and application metrics and can store them in many ways.
Collectd gained popularity for its performance and light weight (it’s written in C). With more than 90 plugins, collectd is a flexible tool for system monitoring including containers.
The collectd agent runs in a Docker image, meaning you run collectd in a completely containerized environment. By writing the telemetry to a Graphite-enabled database, you can then monitor metrics and events in your cluster.
This article assumes that you have:
- A Graphite-enabled server to receive and aggregate the collectd metrics. It’s beyond the scope of this article to describe how to deploy this type of server, which takes careful planning. One approach is to use a Linux virtual machine on Azure.
- A Kubernetes cluster deployed in Azure using Azure Container Service.
- A private Azure Container Registry to store your custom Docker images.
- A local installation of Docker, used to create a custom Docker image.
Create a collectd Docker image
For maximum flexibility and portability, you can create a Docker image to run your collectd agent. These setup steps are based on Alpine Linux, a lightweight Linux distribution, but the Docker image can be created on either Windows or Linux as the local operating system.
- Create a new folder named collectd-agent, then change to that directory by using the command:
- In this folder, create a file named Dockerfile and add the following text to it:
# A simple collectd agent on Linux Alpine FROM alpine:3.5 RUN apk update && apk add collectd CMD [ "/bin/sh", "-c", "collectd; while :; do sleep 300; done" ]
This file describes a Docker image based on Alpine Linux that installs collectd and sets it up as an entry point.
- To build a Docker image based on this Dockerfile, run the following command in shell. For <your registry name>, substitute the name of the Azure Container Registry you previously created.
docker build -t <your registry name>-microsoft.azurecr.io/collectd-agent:1.0 .
- To push the Docker image to the Azure Container Registry, run the following command, substituting the name of your Azure Container Registry for <your registry name>:
docker push <your registry name>-microsoft.azurecr.io/collectd-agent:1.0
NOTE: You may have to log on to Azure Container Registry if you haven’t before. You will be prompted for your logon credentials.
Deploy to Kubernetes cluster
To deploy the collectd agent to the Kubernetes cluster, YAML files are used to specify the configuration details. You create a service that provides the IP address of the Graphite-enabled server and then deploy it to the Kubernetes cluster.
You also define a ConfigMap, a YAML file that loads the plugins used by collectd. For this example, we load the following three plugins, but you can add more as needed:
- CPU plugin. Collects the amount of time spent by a CPU in various states, such as executing user code, executing system code, waiting for I/O operations, and being idle.
- Memory plugin. Collects physical memory usage by the operating system, such as used, buffered, cached, and free.
- Write Graphite plugin. Stores values in Carbon, the storage layer of a Graphite installation.
To run the monitoring agent, you create DaemonSets. Kubernetes uses DaemonSets to run a single instance of a container on each host in the cluster.
- Create a file named graphite-svc.yml that contains the following content specifying a Kubernetes service and an external endpoint for a Graphite server. This is a convenient way to decouple the external installation of a Graphite server from your Kubernetes cluster. For the ip parameter, specify the IP address of your Graphite server.
# graphite service apiVersion: v1 kind: Service metadata: name: graphite-svc spec: clusterIP: 10.0.90.1 # can be any cluster IP you want ports: - port: 2003 targetPort: 2003 protocol: TCP --- apiVersion: v1 kind: Endpoints metadata: name: graphite-svc subsets: - addresses: - ip: <your graphite server IP address> ports: - port: 2003 protocol: TCP
- To deploy the service and endpoint to your Kubernetes cluster, run the following command:
kubectl apply -f graphite-svc.yml
- Create a ConfigMap file named collectd-config.yml with the following content specifying the collectd configuration and plugins:
kind: ConfigMap apiVersion: v1 metadata: name: collectd-config namespace: default data: node-collectd.conf: |- FQDNLookup false LoadPlugin syslog <Plugin syslog> LogLevel info </Plugin> LoadPlugin cpu LoadPlugin memory LoadPlugin write_graphite <Plugin "cpu"> Interval 5 ReportByState false ReportByCpu false </Plugin> <Plugin "memory"> Interval 30 ValuesAbsolute false ValuesPercentage true </Plugin> <Plugin write_graphite> <Node "wgnode1"> Host "10.0.90.1" Port "2003" Protocol "tcp" LogSendErrors true StoreRates true AlwaysAppendDS false EscapeCharacter "_" </Node> </Plugin> #Last line (collectd requires ‘\n’ at the last line)
- If you didn’t do so earlier, create a secret for your Azure Container Registry using the following command:
kubectl create secret docker-registry acrregkey --docker-server=<your registry name>-microsoft.azurecr.io --docker-username=<name> --docker-password=<password> --docker-email=<your email>
To generate the password for your container registry, use Azure Portal.
- Create a Kubernetes DaemonSet file named collectd-node-agent.yml that contains the following content describing the collectd agent. For the image parameter, specify your registry name.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: collectd-node-agent labels: app: collectd-node-agent spec: template: metadata: labels: name: collectd-node-agent spec: hostNetwork: true imagePullSecrets: - name: acrregkey containers: - name: collectd image: <your registry name>-microsoft.azurecr.io/collectd-agent:1.0 securityContext: privileged: true volumeMounts: - name: collectd-config mountPath: /etc/collectd - name: proc mountPath: /mnt/proc readOnly: true - name: root mountPath: /hostfs readOnly: true - name: etc mountPath: /mnt/etc readOnly: true - name: run mountPath: /var/run/docker.sock volumes: - name: collectd-config configMap: name: collectd-config items: - key: node-collectd.conf path: collectd.conf - name: proc hostPath: path: /proc - name: root hostPath: path: / - name: etc hostPath: path: /etc - name: run hostPath: path: /var/run/docker.sock
- To deploy the DaemonSet to your Kubernetes cluster, run the following command:
kubectl apply -f collectd-node-agent.yml
The collectd agent starts to collect telemetry from the nodes in your Kubernetes cluster and writes the data to your Graphite-enabled server.
The following resources provide helpful tips: