K3S High Availability using v1.21.3+k3s1

Homelab
1

Moral of The Story: Read the damn release notes !! :flushed: => K3S v1.21.3+k3s1

I just spent all afternoon calling K3S every name in the book besides a viable Kubernetes engine.

Here’s the story, without all the colorful metaphors.

Cluster Components

  • pqdb - K3S PostgreSQL External Database
  • k3s-lb - Nginx Stream Load Balancer (TCP and UDP)
  • k3s-server{1,2} - server nodes (servers, non-pod targets)
  • k3s-agent{1…5} - agent nodes (workers, pod-targets)

In older versions of K3S < 1.21.3, you didn’t have to specify K3S_TOKEN=123XYZ for adding additional server nodes, but you did for agents. Well that’s changed in 1.21.3+. Also, you MUST have both your database and load balancer up and running “before” joining your server nodes (should be obvious, but it’s not always). The Rancher Docs are far from clear on this minor detail :japanese_goblin:

So the process goes something like this (if you want a fancy doc, I can write it up).

# On Server-1, execute the following

export K3S_DATASTORE_ENDPOINT='postgres://somebody:something@192.168.0.XX:5432/k3s'

sudo curl -sfL https://get.k3s.io | sh -s - server --node-taint CriticalAddonsOnly=true:NoExecute --tls-san 192.168.0.XX

# Important - get the K3S_TOKEN !!!
sudo cat /var/lib/rancher/k3s/server/node-token

12345ABCDE

Exit Server-1 and ssh in to Server-2

# On Server-2, execute the following

export K3S_DATASTORE_ENDPOINT='postgres://somebody:something@192.168.0.XX:5432/k3s'

sudo curl -sfL https://get.k3s.io | K3S_TOKEN=12345ABCDE sh -s - server --node-taint CriticalAddonsOnly=true:NoExecute --tls-san 192.168.0.XX

Exit Server-2 and ssh back in to Server-1, then check node status

# Back on on Server-1, execute the following

radio@k3s-server1:~$ sudo k3s kubectl get nodes
[sudo] password for me: 
NAME          STATUS   ROLES                  AGE    VERSION
k3s-server1   Ready    control-plane,master   11m    v1.21.3+k3s1
k3s-server2   Ready    control-plane,master   25s    v1.21.3+k3s1

Additionally, when adding you’re k3s-agent(x) nodes, you will also need to use that same token, otherwise, you get join errors referencing token errors.

Anyway, I hope this helps prevent others from having the same frustration.

1 Like
2

And here’s what the landscape looks like after adding the agent nodes. Next step is to add some work !!

radio@k3s-server1:~$ sudo k3s kubectl get nodes

NAME          STATUS   ROLES                  AGE     VERSION
k3s-server1   Ready    control-plane,master   101m    v1.21.3+k3s1
k3s-agent1    Ready    <none>                 4m22s   v1.21.3+k3s1
k3s-server2   Ready    control-plane,master   91m     v1.21.3+k3s1
ks3-agent3    Ready    <none>                 49s     v1.21.3+k3s1
k3s-agent2    Ready    <none>                 87s     v1.21.3+k3s1
k3s-agent4    Ready    <none>                 15s     v1.21.3+k3s1

And now after labeling the agents. A “keen-eye” would have spotted that I misspelled agent3.

# To label a node, use:
# sudo k3s kubectl label nodes ks3-agent1 kubernetes.io/role=agent

# After Roles assigned

radio@k3s-server1:~$ sudo k3s kubectl get nodes

NAME          STATUS   ROLES                  AGE    VERSION
k3s-agent2    Ready    agent                  25m    v1.21.3+k3s1
k3s-server2   Ready    control-plane,master   114m   v1.21.3+k3s1
ks3-agent3    Ready    agent                  24m    v1.21.3+k3s1
k3s-server1   Ready    control-plane,master   125m   v1.21.3+k3s1
k3s-agent4    Ready    agent                  23m    v1.21.3+k3s1
k3s-agent1    Ready    agent                  28m    v1.21.3+k3s1
1 Like