Kubernetes Monitoring with Heapster, InfluxDB and Grafana

Kubernetes provides detailed insights about resource usage in the cluster. This is enabled by using Heapster, cAdvisor, InfluxDB and Grafana.

Heapster is installed as a cluster-wide pod. It gathers monitoring and events data for all pods on each node by talking to the Kubelet. Kubelet itself fetches this data from cAdvisor. This data is persisted in InfluxDB and then visualized using Grafana.
Resource Usage Monitoring provide more details about monitoring resources in Kubernetes.

Heapster, InfluxDB and Grafana are Kubernetes addons. They are enabled by default if you are running the cluster on Amazon Web Services or Google Cloud. But need to be explicitly enabled if the cluster is started using minikube or kops addons.

Start a Kubernetes cluster on Amazon Web Services as:


More details about starting a Kubernetes cluster are available at Getting Started with Kubernetes 1.4.

By default, it creates a 4-node Kubernetes cluster in us-west-2a region. More details about the cluster can be seen using the command kubectl cluster-info and it shows the results as:

Kubernetes master is running at
Elasticsearch is running at
Heapster is running at
Kibana is running at
KubeDNS is running at
kubernetes-dashboard is running at
Grafana is running at
InfluxDB is running at
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Note the URL for the Grafana service. Open this URL in a browser window. You’ll be prompted for an invalid certificate warning but this can be safely ignored at this time. In production system, appropriate certificates should be installed.

Then you’ll be prompted for credentials. These can be obtained using kubectl config view command. It will show the output as:

- name: aws_kubernetes-basic-auth
    password: ZeH4JpQzAtGDEBdb
    username: admin

Use the value from username and password fields.

This shows the default dashboard:


It consists of two dashboards – one for cluster and another for pods.


For this blog, a 4-node Couchbase cluster was created following the steps outlined in Create a Couchbase Cluster using Kubernetes.

A cluster-wide dashboard shows CPU, Memory, Filesystem and Network usage across all the hosts and looks like:


CPU, memory, filesystem and network usage for all nodes may be seen:


Details for each node may be seen by selecting the node:


CPU, memory, filesystem and network usage for each node is displayed:


Pods dashboard shows CPU, memory, filesystem and network usage for each pod:


A different pod may be chosen:


A complete list of all services running in the Kubernetes can be seen using kubectl get services --all-namespaces command. It shows the output as: get svc --all-namespaces
NAMESPACE     NAME                       CLUSTER-IP     EXTERNAL-IP        PORT(S)             AGE
default       couchbase-master-service    aef06961eb8f3...   8091/TCP            1h
default       kubernetes                           443/TCP             1h
kube-system   elasticsearch-logging                 9200/TCP            1h
kube-system   heapster                          80/TCP              1h
kube-system   kibana-logging                    5601/TCP            1h
kube-system   kube-dns                            53/UDP,53/TCP       1h
kube-system   kubernetes-dashboard                80/TCP              1h
kube-system   monitoring-grafana                  80/TCP              1h
kube-system   monitoring-influxdb                 8083/TCP,8086/TCP   1h

A complete list of all the pods running in the Kubernetes cluster can be seen using kubectl get pods --all-namespaces. It shows the output as: get pods --all-namespaces
default couchbase-master-rc-q9awd 1/1 Running 17 56m
default couchbase-worker-rc-b1qkc 1/1 Running 15 54m
default couchbase-worker-rc-j1c5z 1/1 Running 17 52m
default couchbase-worker-rc-ju7z3 1/1 Running 15 52m
kube-system elasticsearch-logging-v1-18ylh 1/1 Running 0 1h
kube-system elasticsearch-logging-v1-fupap 1/1 Running 0 1h
kube-system 1/1 Running 0 1h
kube-system 1/1 Running 0 1h
kube-system 1/1 Running 15 1h
kube-system 1/1 Running 17 1h
kube-system heapster-v1.2.0-1374379659-jms8e 4/4 Running 0 1h
kube-system kibana-logging-v1-fcg4b 1/1 Running 3 1h
kube-system kube-dns-v20-wpip4 3/3 Running 0 1h
kube-system 1/1 Running 0 1h
kube-system 1/1 Running 0 1h
kube-system 1/1 Running 15 1h
kube-system 1/1 Running 17 1h
kube-system kubernetes-dashboard-v1.4.0-yxxgx 1/1 Running 0 1h
kube-system monitoring-influxdb-grafana-v4-7asy4 2/2 Running 0 1h

Some references:

Arun Gupta

Arun is a technology enthusiast, avid runner, author of a best-selling book, globe trotter, a community guy, Java Champion, JavaOne Rockstar, JUG Leader, Minecraft Modder, Devoxx4Kids-er, and a Red Hatter.
Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments
Back to top button