Setting up a private Kubernetes cluster with k0sctl

2023/05/28

In my continuing quest for the easiest, fastest way to set up a private Kubernetes cluster (e.g. on home lab server), a solution finally exists: k0sctl. This is an automation to run k0s on nodes that you have SSH access to, e.g. your laptop.

As an optional brief background info, see the previous private Kubernetes setup guide with plain k0s.

Prerequisites

First, setup the machines that will become nodes in the Kubernetes cluster as Ubuntu servers. See this previous guide:

pageUbuntu GPU Server Setup

Check the node requirements by k0sctl, e.g. the host account must be root or have passwordless root access. To do the latter on the node:

1. SSH into the node

2. open sudoers file: sudo nano /etc/sudoers

3. below the line includedir /etc/sudoers.d, add: <username> ALL=(ALL) NOPASSWD: ALL (replace <username>)

Installation

Install k0sctl and kubectl on your laptop (e.g. MacOS) with SSH access to the nodes:

brew install k0sproject/tap/k0sctl
brew install kubectl

Create Kubernetes cluster

Then, follow the k0sctl guide:

1. Create initial config file.

k0sctl init > k0sctl.yaml

2. Configure nodes in the hosts fields, e.g. choose the appropriate roles. Additionally:

   1. Default setup with kuberouter has issues, use Calico instead

   2. Add the builtin openEBS for persistent storage (PVC)

   3. Add other Helm charts: cert-manager, Prometheus stack, Grafana

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
    - ssh:
        address: 192.168.12.34
        user: root
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: controller+worker
    - ssh:
        address: 192.168.12.35
        user: root
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: worker
  k0s:
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s-cluster
      spec:
        network:
          provider: calico
        extensions:
          storage:
            type: openebs_local_storage
          helm:
            repositories:
              - name: openebs-cstor
                url: https://openebs.github.io/cstor-operators
            charts:
              - name: openebs-cstor
                chartname: openebs-cstor/cstor
                version: "3.1.0"
                values: |
                  csiNode:
                    kubeletDir: /var/lib/k0s/kubelet/
                namespace: openebs

3. Apply to deploy the cluster.

k0sctl apply --config k0sctl.yaml

4. Once cluster is set up, get the kubeconfig for kubectl to access the cluster.

mkdir -p ~/.kube
k0sctl kubeconfig --config cluster/k0sctl.yaml > ~/.kube/config && chmod go-r ~/.kube/config
# see nodes
kubectl get nodes
# make openebs-hostpath the default storage class
kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

That's all! See the k0sctl doc for more operations, e.g. add nodes, upgrade, reset.