Enterprise Java

Discover applications running on Kubernetes with Prometheus

Prometheus supports scraping multiple instances of an application. Applications that run in orchestrated environments require to be discovered dynamically, since their IP addresses will change. Prometheus can be configured to use the Kubernetes API to discover changes in the list of running instances dynamically.

The following shows a minimalistic Prometheus example of instrumenting an application with multiple pod instances.

Prometheus configuration

We configure Prometheus to discover the pods of our config-example application.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
global:
  scrape_interval:     15s
 
  external_labels:
    monitor: 'example-monitor'
 
scrape_configs:
- job_name: 'example-metric'
  scrape_interval: 5s
 
  metrics_path: /metrics/
  scheme: https
  basic_auth:
    username: admin
    password: adminadmin
  tls_config:
    insecure_skip_verify: true
 
 
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - default
 
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: config-example
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https
    action: replace

We need to adjust the app label (here config-example) and the port name (https) under which the monitoring endpoint is available.

Instrumented application

This example uses the MicroProfile Metrics endpoint that runs on top of Open Liberty. The application exposes the metrics in the Prometheus format under port 9443 and path /metrics/.

Here you can see an example service and deployment:

01
02
03
04
05
06
07
08
09
10
11
12
13
14
kind: Service
apiVersion: v1
metadata:
  name: config-example
  labels:
    app: config-example
spec:
  selector:
    app: config-example
  ports:
  - port: 9443
    name: https
  - port: 9080
    name: http
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: config-example
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: config-example
    spec:
      containers:
      - name: config-example
        image: sdaschner/config-example:1
    # ...

You can find the full example including the running application on GitHub.

Prometheus setup & RBAC

In order to make the example work for a RBAC Kubernetes setup, the Prometheus service account needs to have the correct permissions. Therefore, we create a service account and corresponding cluster roles as included in the YAML definitions here. Similarly, we specify the service account when creating the Prometheus deployment.

We create a deployment and service for our Prometheus instance.

01
02
03
04
05
06
07
08
09
10
11
12
kind: Service
apiVersion: v1
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  selector:
    app: prometheus
  ports:
  - port: 9090
    name: http
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
        version: v1
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.7.1
        ports:
        - containerPort: 9090
        volumeMounts:
        - name: prometheus-config-volume
          mountPath: /etc/prometheus/prometheus.yml
          subPath: prometheus.yml
      volumes:
      - name: prometheus-config-volume
        configMap:
          name: prometheus-config
      restartPolicy: Always

The configuration shown earlier can be injected into the instance from a config map.

1
2
3
4
5
6
7
8
9
kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      # content as shown earlier ...

For a full explanation on the Prometheus configuration, have a look at the documentation.

Accessing the targets

The running Prometheus instance can now discover both running config example applications which run as Kubernetes pods:

1
gt; kubectl get pods NAME READY STATUS RESTARTS AGE config-example-69974cbc96-dqd96 1/1 Running 0 4m config-example-69974cbc96-zstg7 1/1 Running 0 4m grafana-8694db9d4f-nvn5s 1/1 Running 0 3m prometheus-594dd9cdb8-95ftz 1/1 Running 0 3m

We can see the actual targets including their IP address under the Prometheus configuration.

Kubernetes with Prometheus

Have a look at the full example on GitHub. You might also see the following video how to implement business metrics in Java EE applications using MicroProfile.

For more complex microservice examples that define many more applications, this approach using the plain Prometheus configuration might be a bit cumbersome. To minimize boilerplate code, developers can use solutions that abstracts away the lower-level configuration, such as Prometheus Operator. In the next post we’ll see how Prometheus Operator facilitates the configuration of applications.

Happy monitoring!

Published on Java Code Geeks with permission by Sebastian Daschner, partner at our JCG program. See the original article here: Discover applications running on Kubernetes with Prometheus

Opinions expressed by Java Code Geeks contributors are their own.

Sebastian Daschner

Sebastian Daschner is a self-employed Java consultant and trainer. He is the author of the book 'Architecting Modern Java EE Applications'. Sebastian is a Java Champion, Oracle Developer Champion and JavaOne Rockstar.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments
Back to top button