Custom metrics with AKS

When working with microservices, we always hear that monitoring and observability are key metrics to keep our ecosystem cohesive, functional, and maintain sanity about what’s happening. We talked about this in the #FalaDev podcast I participated in with several guests.

After all, in distributed systems, the complexity is not in the unit itself, but in how these units interact with each other. And if we don’t know what’s happening in our ecosystem, we can’t diagnose, understand, and much less respond to incidents in a timely manner.

But how can we solve these problems? The answer is quite simple: we need to start monitoring our applications.

Azure Monitor For Containers#

For this type of situation, we have tools like Azure Monitor. This is a tool provided by Azure for monitoring several of its products, one of which is AKS.

The idea of this article is to understand a bit more about Azure Monitor first, then we’ll create a small application that will extract some custom metrics from some of our services. So let’s start by understanding how it works!

How does Azure Monitor For Containers work?#

As this great article by Thomas Stringer says, Azure Monitor is a solution that includes several facets, one of which is data visualization obtained through metrics capture by what we call an agent.

An agent is a small service that runs inside our cluster as a daemonset, that is, a service that creates a pod in each node of our cluster to be able to capture machine resource metrics, such as CPU, RAM, etc.

By default, this feature comes disabled when we create a new AKS cluster, so we need to enable it. We’ll learn how to do that in the next chapter.

Once the feature is enabled, a DaemonSet called omsagent is installed in the cluster and a deployment called omsagent-rs is created in each node of the cluster. This deployment is responsible for aggregating metrics and sending them to what we call a Log Analytics workspace, that is, the place where all our metrics will be stored for us to read.

Once we have all the services running, we’ll be able to get the metrics of our cluster by accessing either Azure’s own dashboard or a tool called Azure Data Explorer.

Monitoring a cluster#

First, we need to create an AKS cluster to monitor. For that, we need to register two extensions in our Azure CLI (if you don’t have Azure CLI installed yet, then install it on your machine).

Let’s check if we already have the providers installed with the following commands:

1
az provider show -n Microsoft.OperationsManagement -o table && \
2
az provider show -n Microsoft.OperationalInsights -o table

If we have outputs of this type:

1
Namespace                       RegistrationPolicy    RegistrationState
2
------------------------------  --------------------  -------------------
3
Microsoft.OperationsManagement  RegistrationRequired  Registered
4

5
Namespace                      RegistrationPolicy    RegistrationState
6
-----------------------------  --------------------  -------------------
7
Microsoft.OperationalInsights  RegistrationRequired  Registered

This means the providers are installed and working (see the Registered). However, if we need to install them, we’ll have to run the following commands:

1
az provider register --namespace Microsoft.OperationsManagement && \
2
az provider register --namespace Microsoft.OperationalInsights

The process may take a few minutes to complete. Run the first command again to make sure the provider was registered. Now let’s create a new Resource Group and store both its value and the name of our new cluster in a variable:

1
export RESOURCE_GROUP=aksmonitor
2
export CLUSTER_NAME=aksmonitor
3
az group create -n $RESOURCE_GROUP -l eastus

Then we can create a new AKS cluster enabled for monitoring through the command:

1
az aks create \
2
  -g $RESOURCE_GROUP \
3
  -n $CLUSTER_NAME \
4
  --node-count 1 \
5
  --generate-ssh-keys \
6
  --enable-addons monitoring,http_application_routing

If you already have an AKS cluster created, you can enable monitoring through the command az aks enable-addons -a monitoring -n $CLUSTER_NAME -g $RESOURCE_GROUP

Creating a test#

Let’s create some test pods so we can monitor system usage. First, we’ll create a simple deployment that will expose a small Node.js API. For that, we’ll create a new file simple_api.yaml and add our instructions:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: simple-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: simple-api
9
  template:
10
    metadata:
11
      labels:
12
        app: simple-api
13
    spec:
14
      containers:
15
      - name: simple-api
16
        image: khaosdoctor/scalable-node-api:2.0.0
17
        resources:
18
          limits:
19
            memory: "128Mi"
20
            cpu: "100m"
21
        ports:
22
        - containerPort: 8080
23
          name: http
24
        env:
25
          - name: PORT
26
            value: "8080"
27
---
28
apiVersion: v1
29
kind: Service
30
metadata:
31
  name: simple-api
32
spec:
33
  type: LoadBalancer
34
  selector:
35
    app: simple-api
36
  ports:
37
  - port: 80
38
    targetPort: http

This service will give us an external IP address that can be obtained using kubectl get svc in the EXTERNAL-IP column:

1
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)
2
simple-api   LoadBalancer   10.0.50.182   xxx.xxx.xxx.xxx   80:30287/TCP

Access the address after a few moments to see the pod running. In this version we’ll have a 2-second delay generating a small load so we can see resource consumption. Let’s go to our Azure portal and select our AKS cluster, then go to the “Insights” menu:

Let’s use kubectl’s scale command to be able to scale the pods and see that we have constant monitoring:

If we run a stress test, we’ll see a gradual increase in memory and CPU usage:

Additionally, we have several workbooks which are pre-made dashboards that allow us to view some data:

We can also get the logs of our containers through the logs menu on the left side of the panel:

Here we’ll have an initial view with a modal asking if we want to see some ready-made queries. Let’s go to the “Audit” tab and then click “Run” below “List containers logs per namespace”:

See an example of a query in Kusto format:

1
// List container logs per namespace
2
// View container logs from all the namespaces in the cluster.
3
ContainerLog
4
| join(KubePodInventory
5
    | where TimeGenerated > startofday(ago(1h)))//KubePodInventory Contains namespace information
6
    on ContainerID
7
| where TimeGenerated > startofday(ago(1h))
8
| project TimeGenerated, Namespace, LogEntrySource, LogEntry

We’ll have a list of all the logs generated by our application:

But what if we want to generate more information and search for more data? How do we create a custom metric in Azure Monitor?

Creating custom metrics#

To understand a bit about what we’ll do next, we need to understand how Kubernetes works with metrics.

Prometheus#

Currently, Prometheus is the market leader when it comes to storage and collection of metrics for distributed applications. Prometheus works in a scraping model, that is, from time to time it accesses a URL defined in its registered endpoints and retrieves the data in a specific format. The abstraction layer that runs between a service and Prometheus is called an exporter.

Exporters fetch metrics from APIs and applications and format them so they can be consumed by Prometheus correctly. For this reason, they are usually run inside the same pod, in another container, accessing the application locally. As this image from Thomas Stringer’s blog shows us:

Exporters#

When we’re working with a technology that doesn’t yet have a ready-made exporter, or when we want to scrape metrics from applications we wrote ourselves (which is our case here), we can write our own through a list of available clients.

The work of an exporter is basically to run a loop where it:

1. Starts an HTTP server
2. Fetches metrics from the target application
3. Processes the metrics and formats them
4. Returns the metrics to Prometheus when needed
5. Sleeps for a certain time before starting again

The application we’ll be using is a simple voting application written in Go. You can see the source code in this repository. We have an image hosted in my personal Docker Hub.

Since I created the application from scratch, I built a small exporter in Node.js for it that you can check in this repository with this image. Basically, the application only has a single index.js file that starts a Koa server using a Prometheus client library.

1
const Koa = require('koa')
2
const app = new Koa()
3
const axios = require('axios').default
4

5
const prometheus = require('prom-client')
6
const PrometheusRegistry = prometheus.Registry
7
const registry = new PrometheusRegistry()
8

9
const PREFIX = `go_vote_api_`
10
const pollingInterval = process.env.POLLING_INTERVAL_MS || 5000
11
registry.setDefaultLabels({ service: 'go_vote_api', hostname: process.env.POD_NAME || process.env.HOSTNAME || 'unknown' })
12

13
// METRICS START
14

15
const totalScrapesCounter = new prometheus.Counter({
16
  name: `${PREFIX}total_scrapes`,
17
  help: 'Number of times the service has been scraped for metrics'
18
})
19
registry.registerMetric(totalScrapesCounter)
20

21
const scrapeResponseTime = new prometheus.Summary({
22
  name: `${PREFIX}scrape_response_time`,
23
  help: 'Response time of the scraped service in ms'
24
})
25
registry.registerMetric(scrapeResponseTime)
26

27
const localResponseTime = new prometheus.Summary({
28
  name: `${PREFIX}exporter_response_time`,
29
  help: 'Response time of the exporter in ms'
30
})
31
registry.registerMetric(localResponseTime)
32

33
const totalVotes = new prometheus.Gauge({
34
  name: `${PREFIX}total_votes`,
35
  help: 'Total number of votes computed until now',
36
  async collect () {
37
    const total = await scrapeApplication()
38
    this.set(total)
39
  }
40
})
41
registry.registerMetric(totalVotes)
42

43
// --Utility Function-- //
44

45
async function scrapeApplication () {
46
  const id = Date.now().toString(16)
47
  console.log(`Scraping ${process.env.SCRAPE_URL}:${process.env.SCRAPE_PORT}/${process.env.SCRAPE_PATH} [scrape id: ${id}]`)
48
  const start = Date.now()
49
  const metrics = await axios.get(`${process.env.SCRAPE_URL}:${process.env.SCRAPE_PORT}/${process.env.SCRAPE_PATH}`)
50
  scrapeResponseTime.observe(Date.now() - start)
51
  totalScrapesCounter.inc()
52
  console.log(`Scraped data [scrape id: ${id}]`)
53
  return metrics.data.total
54
}
55

56
// --Servers start-- //
57

58
app.use(async (ctx, next) => {
59
  console.log(`Received scrape request: ${ctx.method} ${ctx.url} @ ${new Date().toUTCString()}`)
60
  const start = Date.now()
61
  await next()
62
  localResponseTime.observe(Date.now() - start)
63
})
64

65
app.use(async ctx => {
66
  ctx.set('Content-Type', registry.contentType)
67
  ctx.body = await registry.metrics()
68
})
69

70
// start loop
71
if (pollingInterval > 0) {
72
  setInterval(async () => {
73
    const total = await scrapeApplication()
74
    totalVotes.set(total)
75
  }, pollingInterval)
76
}
77

78
console.log(`Listening on ${process.env.SCRAPER_PORT || 9837}`)
79
app.listen(process.env.SCRAPER_PORT || 9837)

What this application is doing is registering a series of metrics in a standard Prometheus registry. The metrics we’re capturing are:

• Total number of votes
• Number of times we fetch the metrics
• Response time of the exporter and also the API on the /total route

Obviously, these metrics are not as important as other metrics that can be obtained through direct instrumentation in the application. For this reason, a good practice is to have a /metrics route that serves the application’s instrumentation metrics, such as CPU, RAM, and others, in addition to the exporter.

If we access the exporter, we’ll have an output like this:

Extracting metrics from the application#

To extract metrics from the application, we’ll run the two containers side by side. This way, we don’t burden the original application with searching for and parsing metrics, and we maintain connection speed since they’re on the same network. Our previous deployment file will change a bit:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: vote-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: vote-api
9
  template:
10
    metadata:
11
      labels:
12
        app: vote-api
13
    spec:
14
      containers:
15
      - name: vote-api
16
        image: khaosdoctor/go-vote-api
17
        resources:
18
          limits:
19
            memory: "128Mi"
20
            cpu: "200m"
21
        ports:
22
        - containerPort: 8080
23
          name: http
24
      - name: vote-api-exporter
25
        image: khaosdoctor/go-vote-api-exporter
26
        resources:
27
          limits:
28
            memory: "128Mi"
29
            cpu: "100m"
30
        ports:
31
        - containerPort: 9837
32
          name: exporter
33
        env:
34
          - name: SCRAPE_PORT
35
            value: "8080"
36
          - name: SCRAPE_PATH
37
            value: total
38
          - name: SCRAPE_URL
39
            value: "http://localhost"
40
      - name: vote-api-voter
41
        image: curlimages/curl
42
        command: ["/bin/sh"]
43
        args: [
44
          "-c",
45
          "while true; do wget -O- http://localhost:8080/votes/Lucas; sleep 3; done"
46
        ]
47
        resources:
48
          limits:
49
            memory: "128Mi"
50
            cpu: "100m"
51
---
52
apiVersion: v1
53
kind: Service
54
metadata:
55
  name: vote-api
56
spec:
57
  type: LoadBalancer
58
  selector:
59
    app: vote-api
60
  ports:
61
  - port: 80
62
    targetPort: http
63
---
64
apiVersion: v1
65
kind: Service
66
metadata:
67
  name: vote-api-exporter
68
spec:
69
  selector:
70
    app: vote-api
71
  ports:
72
  - port: 9837
73
    targetPort: exporter

What we’re doing is running three containers alongside the application. One of them is the exporter and the other is a simple application that will be voting every 3 seconds to simulate vote increases. Note that we’re setting localhost as the scraping URL because all containers are on the same local network.

We can check the logs of each created container later with the command kubectl logs deploy/vote-api -c <container-name>. To see our exporter in action, just run kubectl port-forward svc/vote-api-exporter 9837:9837 and access localhost:9837 on our machine:

Note that now we have more labels, such as the hostname that wasn’t being fetched before.

Preparing Azure Monitor#

Now that we have our API ready, let’s prepare Azure Monitor to be able to fetch metrics. For this, we’ll create a simple ConfigMap that will configure our agent within the Node. Microsoft itself has a standard configuration template that we can download with the command below:

1
$ curl -Lo agent-config.yaml https://aka.ms/container-azm-ms-agentconfig

Save the file and see that it’s well commented. It’s a long file, but the part we’re interested in is here:

1
        # When monitor_kubernetes_pods = true, replicaset will scrape Kubernetes pods for the following prometheus annotations:
2
        # - prometheus.io/scrape: Enable scraping for this pod
3
        # - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
4
        #     set this to `https` & most likely set the tls config.
5
        # - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
6
        # - prometheus.io/port: If port is not 9102 use this annotation
7
        monitor_kubernetes_pods = false

Let’s change monitor_kubernetes_pods to true. This will cause any deployments with the prometheus.io/scrape and prometheus.io/scheme annotations to be scraped by the agent as if Prometheus were scraping metrics. Now let’s create the configuration with kubectl apply -f agent.yaml.

Now let’s add the annotations to our deployment file:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: vote-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: vote-api
9
  template:
10
    metadata:
11
      labels:
12
        app: vote-api
13
      annotations:
14
        prometheus.io/scrape: "true"
15
        prometheus.io/path: /
16
        prometheus.io/port: "9837"
17
    spec:
18
      containers:
19
      - name: vote-api
20
        image: khaosdoctor/go-vote-api
21
        resources:
22
          limits:
23
            memory: "128Mi"
24
            cpu: "200m"
25
        ports:
26
        - containerPort: 8080
27
          name: http
28
      - name: vote-api-exporter
29
        image: khaosdoctor/go-vote-api-exporter
30
        resources:
31
          limits:
32
            memory: "128Mi"
33
            cpu: "100m"
34
        ports:
35
        - containerPort: 9837
36
          name: exporter
37
        env:
38
          - name: SCRAPE_PORT
39
            value: "8080"
40
          - name: SCRAPE_PATH
41
            value: total
42
          - name: SCRAPE_URL
43
            value: "http://localhost"
44
      - name: vote-api-voter
45
        image: curlimages/curl
46
        command: ["/bin/sh"]
47
        args: [
48
          "-c",
49
          "while true; do wget -O- http://localhost:8080/votes/Lucas; sleep 3; done"
50
        ]
51
        resources:
52
          limits:
53
            memory: "128Mi"
54
            cpu: "100m"
55
---
56
apiVersion: v1
57
kind: Service
58
metadata:
59
  name: vote-api
60
spec:
61
  type: LoadBalancer
62
  selector:
63
    app: vote-api
64
  ports:
65
  - port: 80
66
    targetPort: http
67
---
68
apiVersion: v1
69
kind: Service
70
metadata:
71
  name: vote-api-exporter
72
spec:
73
  selector:
74
    app: vote-api
75
  ports:
76
  - port: 9837
77
    targetPort: exporter

Now we can fetch metrics through our portal. Let’s click on Logs as we did before, and now we can search within the InsightsMetrics table using the following query:

1
InsightsMetrics
2
| where Namespace == "prometheus"
3
| where Name in ("go_vote_api_total_votes")
4
| summarize sum(Val) by TimeGenerated, Name
5
| order by TimeGenerated asc

This will give us all the votes that were computed, separated by the time they were generated:

If we click on Chart, we’ll have a chart that can be configured to show the metric’s growth:

Conclusion#

Extracting metrics is important and necessary for us to have a better view of our system. With Azure Monitor, it becomes much easier to make these measurements because we don’t need to install anything external like Prometheus, and we also don’t need to manage databases or anything else.