Setting up Kubernetes Addons
2023/06/11
After creating a Kubernetes cluster, we would want to add a set of standard cluster addons using Helm for DevOps:
cert-manager: certificate management
cluster-autoscaler: to dynamically autoscale cluster by adding or reducing nodes.
metrics-server: for monitoring and and HPA (HorizontalPodAutoscaler) to work.
kubernetes-dashboard: basic cluster monitoring (if Lens is not available)
Loki (scalable): to aggregate and index all logs in the cluster, with retention policy; the logs are searchable in Grafana. Additionally:
promtail to aggregate logs
Note: Elasticsearch charts (hence ELK) have been deprecated in favor of their licensed ECK; plus Loki is much easier to run and maintain
kube-prometheus-stack: for cluster monitoring with many useful preconfigured cluster Prometheus metrics in Grafana dashboards. Additionally:
prometheus-adapter for custom metrics API, e.g. for HPA to scale using custom-defined metrics.
prometheus-pushgateway to push application metrics
prometheus-blackbox-exporter to probe endpoints for uptime monitoring
Additionally, install Lens for GUI monitoring and access to the cluster. Get a free license to use.
Installations
cert-manager
# cert manager
helm repo add jetstack https://charts.jetstack.io
helm upgrade -i cert-manager jetstack/cert-manager -n cert-manager --create-namespace --version 'v1.12.1' --set installCRDs=true
cluster-autoscaler
Helm chart here. This component has some custom settings dependending on which cloud provider is used, but the main gist is:
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm upgrade -i cluster-autoscaler autoscaler/cluster-autoscaler -n kube-system
metrics-server
Helm chart here. Some kubernetes providers have this preinstalled, but the gist is:
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm upgrade -i metrics-server metrics-server/metrics-server -n kube-system --version '3.10.0'
kubernetes-dashboard
Manifest here - this installs more reliably than its Helm chart. First, prepare manifest to create a service account for dashboard access:
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
Then run:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
kubectl apply -f ./cluster/dashboard-admin-user.yaml
Preferably, install Lens for GUI monitoring and access to the cluster (it connects using kubeconfig). Get a free license to use.
Loki
Helm chart here. Loki is a Grafana project for log aggregation - it is scalable (to TBs), cheap, and simple to maintain. The logs will show up in Grafana dashboards - which is a must-have for Kubernetes clusters.
First, prepare the Helm override file to:
configure storage for scalable mode (s3/GCP/Azure etc., or use minio to try first)
# for loki scalable mode https://grafana.com/docs/loki/latest/fundamentals/architecture/deployment-modes/#simple-scalable-deployment-mode
# need to configure storage https://github.com/grafana/loki/blob/main/production/helm/loki/values.yaml#L253
# or try with minio first
minio:
enabled: true
loki:
storage:
type: s3
s3:
s3: null
endpoint: null
region: null
secretAccessKey: null
accessKeyId: null
s3ForcePathStyle: false
insecure: false
# configure retention https://grafana.com/docs/loki/latest/operations/storage/retention/
# fields: https://grafana.com/docs/loki/latest/configuration/
compactor:
shared_store: filesystem
retention_enabled: true
limits_config:
retention_period: 744h
auth_enabled: false
We'll also install promtail as log aggregating agent.
Then run:
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade -i loki grafana/loki -n logging --create-namespace --version '5.6.4' -f ./cluster/loki-values.yaml
helm upgrade -i promtail grafana/promtail -n logging --version '6.11.3'
kube-prometheus-stack
Helm chart here. This includes kube-state-metrics, node-exporter, and Grafana: they gather kubernetes metrics from all the cluster nodes, and preconfigure many useful cluster metric dashboards.
Prepare the Helm override file to set/change default password, and add Loki as data source for log search in Grafana:
grafana:
adminPassword: prom-operator
# configure anonymous view-access
grafana.ini:
auth.anonymous:
enabled: true
org_name: Main Org.
org_role: Viewer
# auth:
# disable_login_form: true
persistence:
enabled: true
## Configure additional grafana datasources (passed through tpl)
## ref: http://docs.grafana.org/administration/provisioning/#datasources
additionalDataSources:
- name: Loki
type: loki
access: proxy
url: http://loki-gateway.logging.svc.cluster.local
version: 1
isDefault: false
We will also install 3 additional Helm charts. The first is prometheus-adapter for custom metrics API, e.g. for HPA to scale using custom-defined metrics.
The second is Prometheus Pushgateway, e.g. to push application metrics. Specify Helm override file to configure ServiceMonitor with matching label so it is scraped by kube-prometheus-stack:
serviceMonitor:
enabled: true
additionalLabels:
release: prometheus
The third is Blackbox Exporter to probe endpoints for uptime monitoring. Specify the Helm override file to configure targets and ServiceMonitor with matching label so it is scraped by kube-prometheus-stack:
config:
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
follow_redirects: true
preferred_ip_protocol: "ip4"
valid_status_codes:
- 200
serviceMonitor:
enabled: true
defaults:
labels:
# match kube-prometheus-stack scrape config
release: prometheus
interval: 30s
scrapeTimeout: 30s
module: http_2xx
scheme: http
targets: # lowercase only
- name: github
url: http://github.com/status
- name: gitlab
url: https://status.gitlab.com
Now, install all of these by running:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade -i prometheus prometheus-community/kube-prometheus-stack -n monitoring --create-namespace --version '46.8.0' -f ./cluster/prometheus-values.yaml
# adapter for k8s HPA custom metrics
helm upgrade -i prom-adapter prometheus-community/prometheus-adapter -n monitoring --version '4.2.0'
# pushgateway for app metrics
helm upgrade -i prom-pushgateway prometheus-community/prometheus-pushgateway -n monitoring --version '2.2.0' -f ./cluster/pushgateway-values.yaml
# blackbox exporter for uptime monitoring
helm install blackbox prometheus-community/prometheus-blackbox-exporter -n monitoring --version '7.10.0' -f ./cluster/blackbox-values.yaml
Grafana Dashboards can also be provisioned via ConfigMaps. See the repo linked above for more examples as the ConfigMap file is large.
Accessing Dashboards
After installing the addons, access all the dashboards as follows:
just open the app, it will use
~/.kube/config
to connect

get token:
kubectl -n kubernetes-dashboard create token admin-user
run
kubectl proxy
and visit http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Grafana for cluster and logging monitoring
data sources include kube-state-metrics, node-exporter, prometheus, and custom-added loki for logs
run
kubectl port-forward -n monitoring svc/prometheus-grafana 6060:80
and visit http://localhost:6060 to find the preconfigured dashboards(one-time) import this Loki Kubernetes Logs and this Blackbox exporter dashboards



Prometheus for cluster monitoring
run
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
and visit http://localhost:9090

Last updated
Was this helpful?