Custom metrics with AKS

Whenever we work with microservices, we always hear that monitoring and observability are key metrics for keeping our ecosystem cohesive, functional, and for not going crazy trying to figure out what’s happening. We talked about this in the #FalaDev podcast I joined alongside several guests.

After all, in distributed systems, the complexity doesn’t live in the unit itself, but in how these units interact with one another. And if we don’t know what’s happening in our ecosystem, we can’t diagnose, understand, or respond to incidents in a timely manner.

So how do we solve these problems? The answer is pretty simple: we have to start monitoring our applications.

Azure Monitor For Containers#

For this kind of situation we have tools like Azure Monitor. It’s a tool made available by Azure to monitor several of its products, and one of them is AKS.

The idea of this article is to first understand a bit more about Azure Monitor, then build a small application that will extract some custom metrics from a few of our services. So let’s start by understanding how it works!

How does Azure Monitor For Containers work?#

According to this amazing article by Thomas Stringer, Azure Monitor is a solution with several facets, one of which is data visualization, obtained through metrics captured by what we call an agent.

An agent is a small service that runs inside our cluster as a daemonset, which means a service that creates a pod on every node in our cluster to capture machine resource metrics, such as CPU, RAM, and so on.

By default, this feature comes disabled when we create a new AKS cluster, so we have to enable it. We’ll learn how to do that in the next chapter.

Once the feature is enabled, a DaemonSet called omsagent is installed on the cluster, and a deployment called omsagent-rs is created on each of the cluster’s nodes. This deployment is responsible for aggregating metrics and sending them to what we call a Log Analytics workspace, the place where all our metrics get stored so we can read them.

Once all the services are running, we’ll be able to fetch our cluster’s metrics either through the Azure portal itself, or through a tool called Azure Data Explorer.

Monitoring a cluster#

First, we need to create an AKS cluster to monitor, and for that we need to register two extensions in our Azure CLI (if you don’t have the Azure CLI installed yet, go ahead and install it on your machine).

Let’s check if we already have the providers installed with the following commands:

1
az provider show -n Microsoft.OperationsManagement -o table && \
2
az provider show -n Microsoft.OperationalInsights -o table

If we get an output like this:

1
Namespace                       RegistrationPolicy    RegistrationState
2
------------------------------  --------------------  -------------------
3
Microsoft.OperationsManagement  RegistrationRequired  Registered
4

5
Namespace                      RegistrationPolicy    RegistrationState
6
-----------------------------  --------------------  -------------------
7
Microsoft.OperationalInsights  RegistrationRequired  Registered

It means the providers are installed and running (see the Registered status), but if we need to install them, we’ll have to run the following commands:

1
az provider register --namespace Microsoft.OperationsManagement && \
2
az provider register --namespace Microsoft.OperationalInsights

The process can take a few minutes to complete, so run the first command again to make sure the provider was registered. Now let’s create a new Resource Group and store both its value and our new cluster’s name in a variable:

1
export RESOURCE_GROUP=aksmonitor
2
export CLUSTER_NAME=aksmonitor
3
az group create -n $RESOURCE_GROUP -l eastus

Then we can create a new AKS cluster with monitoring enabled through the command:

1
az aks create \
2
  -g $RESOURCE_GROUP \
3
  -n $CLUSTER_NAME \
4
  --node-count 1 \
5
  --generate-ssh-keys \
6
  --enable-addons monitoring,http_application_routing

If you already have an AKS cluster created, you can enable monitoring through the command az aks enable-addons -a monitoring -n $CLUSTER_NAME -g $RESOURCE_GROUP

Creating a test#

Let’s create a few test pods so we can monitor system usage. First we’ll create a simple deployment that exposes a small Node.js API. For that, let’s create a new file called simple_api.yaml and write our instructions:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: simple-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: simple-api
9
  template:
10
    metadata:
11
      labels:
12
        app: simple-api
13
    spec:
14
      containers:
15
      - name: simple-api
16
        image: khaosdoctor/scalable-node-api:2.0.0
17
        resources:
18
          limits:
19
            memory: "128Mi"
20
            cpu: "100m"
21
        ports:
22
        - containerPort: 8080
23
          name: http
24
        env:
25
          - name: PORT
26
            value: "8080"
27
---
28
apiVersion: v1
29
kind: Service
30
metadata:
31
  name: simple-api
32
spec:
33
  type: LoadBalancer
34
  selector:
35
    app: simple-api
36
  ports:
37
  - port: 80
38
    targetPort: http

This service will give us an external IP address that we can grab using kubectl get svc in the EXTERNAL-IP column:

1
NAME         TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)
2
simple-api   LoadBalancer   10.0.50.182   xxx.xxx.xxx.xxx   80:30287/TCP

Access the address after a few moments to see the pod running. In this version, we have a 2-second delay generating a bit of load so we can see resource consumption. Let’s head to the Azure portal, select our AKS cluster, then go to the “Insights” menu:

Let’s use the kubectl scale command to scale our pods and see that we have constant monitoring:

If we run a stress test, we’ll see a gradual increase in memory and CPU usage:

We also have several workbooks, which are ready-made dashboards that let us see some data:

We can also get our containers’ logs through the logs menu on the left of the panel:

Here we’ll get an initial view with a modal asking whether we want to see some ready-made queries. Let’s go to the “Audit” tab and click “Run” under “List containers logs per namespace”:

Here’s an example query in Kusto format:

1
// List container logs per namespace
2
// View container logs from all the namespaces in the cluster.
3
ContainerLog
4
| join(KubePodInventory
5
    | where TimeGenerated > startofday(ago(1h)))//KubePodInventory Contains namespace information
6
    on ContainerID
7
| where TimeGenerated > startofday(ago(1h))
8
| project TimeGenerated, Namespace, LogEntrySource, LogEntry

We’ll get a list of every log our application has generated:

But what if we want to generate more information and fetch more data? How do we create a custom metric in Azure Monitor?

Creating custom metrics#

To understand what we’re going to do next, we need to understand how Kubernetes works with metrics.

Prometheus#

Nowadays, Prometheus is the market leader when it comes to storing and collecting metrics for distributed applications. Prometheus works on a scraping model, meaning that every so often it hits a URL defined on its registered endpoints and fetches the data in a specific format. The abstraction layer that runs between a service and Prometheus is called an exporter.

Exporters fetch metrics from APIs and applications and format them so Prometheus can consume them correctly, which is why they usually run inside the same pod, in a separate container, accessing the application locally. As this image from Thomas Stringer’s blog shows us:

Exporters#

When we’re working with a technology that doesn’t have a ready-made exporter yet, or when we want to scrape metrics from applications we wrote ourselves (which is our case here), we can write our own using a list of available clients.

An exporter’s job is basically to run a loop where it:

1. Starts an HTTP server
2. Fetches metrics from the target application
3. Processes and formats the metrics
4. Returns the metrics to Prometheus when requested
5. Sleeps for a set amount of time before starting over

The application we’re going to use is a simple voting application written in Go. You can check the source code in this repository. We have an image hosted on my personal Docker Hub.

Since I built the application from scratch, I also built a small Node.js exporter for it, which you can check out in this repository, with this image. Basically the application only has a single index.js file that starts a Koa server using a Prometheus client library.

1
const Koa = require('koa')
2
const app = new Koa()
3
const axios = require('axios').default
4

5
const prometheus = require('prom-client')
6
const PrometheusRegistry = prometheus.Registry
7
const registry = new PrometheusRegistry()
8

9
const PREFIX = `go_vote_api_`
10
const pollingInterval = process.env.POLLING_INTERVAL_MS || 5000
11
registry.setDefaultLabels({ service: 'go_vote_api', hostname: process.env.POD_NAME || process.env.HOSTNAME || 'unknown' })
12

13
// METRICS START
14

15
const totalScrapesCounter = new prometheus.Counter({
16
  name: `${PREFIX}total_scrapes`,
17
  help: 'Number of times the service has been scraped for metrics'
18
})
19
registry.registerMetric(totalScrapesCounter)
20

21
const scrapeResponseTime = new prometheus.Summary({
22
  name: `${PREFIX}scrape_response_time`,
23
  help: 'Response time of the scraped service in ms'
24
})
25
registry.registerMetric(scrapeResponseTime)
26

27
const localResponseTime = new prometheus.Summary({
28
  name: `${PREFIX}exporter_response_time`,
29
  help: 'Response time of the exporter in ms'
30
})
31
registry.registerMetric(localResponseTime)
32

33
const totalVotes = new prometheus.Gauge({
34
  name: `${PREFIX}total_votes`,
35
  help: 'Total number of votes computed until now',
36
  async collect () {
37
    const total = await scrapeApplication()
38
    this.set(total)
39
  }
40
})
41
registry.registerMetric(totalVotes)
42

43
// --Utility Function-- //
44

45
async function scrapeApplication () {
46
  const id = Date.now().toString(16)
47
  console.log(`Scraping ${process.env.SCRAPE_URL}:${process.env.SCRAPE_PORT}/${process.env.SCRAPE_PATH} [scrape id: ${id}]`)
48
  const start = Date.now()
49
  const metrics = await axios.get(`${process.env.SCRAPE_URL}:${process.env.SCRAPE_PORT}/${process.env.SCRAPE_PATH}`)
50
  scrapeResponseTime.observe(Date.now() - start)
51
  totalScrapesCounter.inc()
52
  console.log(`Scraped data [scrape id: ${id}]`)
53
  return metrics.data.total
54
}
55

56
// --Servers start-- //
57

58
app.use(async (ctx, next) => {
59
  console.log(`Received scrape request: ${ctx.method} ${ctx.url} @ ${new Date().toUTCString()}`)
60
  const start = Date.now()
61
  await next()
62
  localResponseTime.observe(Date.now() - start)
63
})
64

65
app.use(async ctx => {
66
  ctx.set('Content-Type', registry.contentType)
67
  ctx.body = await registry.metrics()
68
})
69

70
// start loop
71
if (pollingInterval > 0) {
72
  setInterval(async () => {
73
    const total = await scrapeApplication()
74
    totalVotes.set(total)
75
  }, pollingInterval)
76
}
77

78
console.log(`Listening on ${process.env.SCRAPER_PORT || 9837}`)
79
app.listen(process.env.SCRAPER_PORT || 9837)

What this application does is register a set of metrics on a default Prometheus registry. The metrics we’re grabbing are:

• Total number of votes
• How many times we’ve fetched the metrics
• Response time of the exporter and also of the API on the /total route

Obviously these metrics aren’t as important as other metrics you could get by instrumenting the application directly. A good practice for that is to have a /metrics route that serves the application’s own instrumentation metrics, like CPU, RAM, and others, in addition to the exporter.

If we hit the exporter, we get an output like this:

Extracting metrics from the application#

To extract metrics from the application, what we’ll do is run both containers side by side. That way we don’t burden the original application with fetching and parsing metrics, and we still keep the connection fast since they’re on the same network. Our previous deployment file will change a bit:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: vote-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: vote-api
9
  template:
10
    metadata:
11
      labels:
12
        app: vote-api
13
    spec:
14
      containers:
15
      - name: vote-api
16
        image: khaosdoctor/go-vote-api
17
        resources:
18
          limits:
19
            memory: "128Mi"
20
            cpu: "200m"
21
        ports:
22
        - containerPort: 8080
23
          name: http
24
      - name: vote-api-exporter
25
        image: khaosdoctor/go-vote-api-exporter
26
        resources:
27
          limits:
28
            memory: "128Mi"
29
            cpu: "100m"
30
        ports:
31
        - containerPort: 9837
32
          name: exporter
33
        env:
34
          - name: SCRAPE_PORT
35
            value: "8080"
36
          - name: SCRAPE_PATH
37
            value: total
38
          - name: SCRAPE_URL
39
            value: "http://localhost"
40
      - name: vote-api-voter
41
        image: curlimages/curl
42
        command: ["/bin/sh"]
43
        args: [
44
          "-c",
45
          "while true; do wget -O- http://localhost:8080/votes/Lucas; sleep 3; done"
46
        ]
47
        resources:
48
          limits:
49
            memory: "128Mi"
50
            cpu: "100m"
51
---
52
apiVersion: v1
53
kind: Service
54
metadata:
55
  name: vote-api
56
spec:
57
  type: LoadBalancer
58
  selector:
59
    app: vote-api
60
  ports:
61
  - port: 80
62
    targetPort: http
63
---
64
apiVersion: v1
65
kind: Service
66
metadata:
67
  name: vote-api-exporter
68
spec:
69
  selector:
70
    app: vote-api
71
  ports:
72
  - port: 9837
73
    targetPort: exporter

What we’re doing is spinning up three containers alongside the application: one is the exporter, and the other is a simple application that keeps voting every 3 seconds to simulate the vote count increasing. Notice that we’re setting localhost as the scrape URL, because all the containers are on the same local network.

We can check the logs of each created container afterward with the command kubectl logs deploy/vote-api -c <container-name>. If we want to see our exporter in action, we just need to run kubectl port-forward svc/vote-api-exporter 9837:9837 and hit localhost:9837 on our machine:

Notice that now we have more labels, like hostname, which wasn’t fetched before.

Setting up Azure Monitor#

Now that our API is ready, let’s set up Azure Monitor so it can fetch the metrics. To do that, we’ll create a simple ConfigMap that configures our agent inside the Node. Microsoft itself has a default configuration template we can download with the command below

1
$ curl -Lo agent-config.yaml https://aka.ms/container-azm-ms-agentconfig

Save the file, and you’ll see it’s pretty heavily commented. It’s a long file, but the part we care about is this one:

1
        # When monitor_kubernetes_pods = true, replicaset will scrape Kubernetes pods for the following prometheus annotations:
2
        # - prometheus.io/scrape: Enable scraping for this pod
3
        # - prometheus.io/scheme: If the metrics endpoint is secured then you will need to
4
        #     set this to `https` & most likely set the tls config.
5
        # - prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
6
        # - prometheus.io/port: If port is not 9102 use this annotation
7
        monitor_kubernetes_pods = false

Let’s change monitor_kubernetes_pods to true. This will make any deployment with the prometheus.io/scrape and prometheus.io/scheme annotations get scraped by the agent as if it were Prometheus fetching metrics. Now let’s apply the configuration with kubectl apply -f agent.yaml.

Now let’s add the annotations to our deployment file:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: vote-api
5
spec:
6
  selector:
7
    matchLabels:
8
      app: vote-api
9
  template:
10
    metadata:
11
      labels:
12
        app: vote-api
13
      annotations:
14
        prometheus.io/scrape: "true"
15
        prometheus.io/path: /
16
        prometheus.io/port: "9837"
17
    spec:
18
      containers:
19
      - name: vote-api
20
        image: khaosdoctor/go-vote-api
21
        resources:
22
          limits:
23
            memory: "128Mi"
24
            cpu: "200m"
25
        ports:
26
        - containerPort: 8080
27
          name: http
28
      - name: vote-api-exporter
29
        image: khaosdoctor/go-vote-api-exporter
30
        resources:
31
          limits:
32
            memory: "128Mi"
33
            cpu: "100m"
34
        ports:
35
        - containerPort: 9837
36
          name: exporter
37
        env:
38
          - name: SCRAPE_PORT
39
            value: "8080"
40
          - name: SCRAPE_PATH
41
            value: total
42
          - name: SCRAPE_URL
43
            value: "http://localhost"
44
      - name: vote-api-voter
45
        image: curlimages/curl
46
        command: ["/bin/sh"]
47
        args: [
48
          "-c",
49
          "while true; do wget -O- http://localhost:8080/votes/Lucas; sleep 3; done"
50
        ]
51
        resources:
52
          limits:
53
            memory: "128Mi"
54
            cpu: "100m"
55
---
56
apiVersion: v1
57
kind: Service
58
metadata:
59
  name: vote-api
60
spec:
61
  type: LoadBalancer
62
  selector:
63
    app: vote-api
64
  ports:
65
  - port: 80
66
    targetPort: http
67
---
68
apiVersion: v1
69
kind: Service
70
metadata:
71
  name: vote-api-exporter
72
spec:
73
  selector:
74
    app: vote-api
75
  ports:
76
  - port: 9837
77
    targetPort: exporter

Now we can fetch the metrics through our portal. Let’s click on Logs, just like we did before, and now we can query inside the InsightsMetrics table with the following query:

1
InsightsMetrics
2
| where Namespace == "prometheus"
3
| where Name in ("go_vote_api_total_votes")
4
| summarize sum(Val) by TimeGenerated, Name
5
| order by TimeGenerated asc

This gives us every vote that’s been computed, broken down by the time they were generated:

If we click Chart, we’ll get a graph we can configure to see the metric grow:

Conclusion#

Extracting metrics is important and necessary so we can get a better view of our system. With Azure Monitor it’s a lot easier to take these measurements, because we don’t need to install anything external like Prometheus, and we don’t need to manage databases or anything else either.