Remove Nodes from existing Kubernetes Cluster.

Step 1

Only when deleting a Control Plane node, it is necessary to delete Etcd and load balancing configuration beforehand like follows.

[root@ctrl ~]# kubectl get nodes

NAME               STATUS   ROLES           AGE    VERSION

dlp-1.srv.world    Ready    control-plane   117m   v1.31.9

dlp.srv.world      Ready    control-plane   3h3m   v1.31.9

node01.srv.world   Ready    <none>          178m   v1.31.9

node02.srv.world   Ready    <none>          177m   v1.31.9

node03.srv.world   Ready    <none>          38s    v1.31.9

[root@ctrl ~]# kubectl get pods -n kube-system | grep etcd

etcd-dlp-1.srv.world                       1/1     Running   0          120m

etcd-dlp.srv.world                         1/1     Running   2          3h5m

as an example, delete [dlp-1.srv.world] node

access to the Etcd on a Control Plane which is not the delete target and

remove configuration for delete target

[root@ctrl ~]# kubectl -n kube-system exec -it etcd-dlp.srv.world -- sh

sh-5.2#

sh-5.2# <span class="color1">etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list </span>

41771c7b1d23e5, started, dlp-1.srv.world, https://10.0.0.31:2380, https://10.0.0.31:2379, false

dd4b95995dc266b1, started, dlp.srv.world, https://10.0.0.30:2380, https://10.0.0.30:2379, false

<span class="color2"># remove a member which you like to delete</span>

sh-5.2# <span class="color1">etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member remove 41771c7b1d23e5 </span>

Member   41771c7b1d23e5 removed from cluster 63678238411c70a3

sh-5.2# <span class="color1">exit </span>

exit

[root@ctrl ~]# vi /etc/nginx/nginx.conf

<span class="color2"># remove the target Control Plane</span>

stream {

    upstream k8s-api {

        server 10.0.0.30:6443;

        <span class="color1">###</span>server 10.0.0.31:6443;

    }

    server {

        listen 6443;

        proxy_pass k8s-api;

    }

}

[root@ctrl ~]# systemctl reload nginx

Step 2

Remove a node from the cluster.

From this point, the procedure is the same for both the Control Plane and Worker.

[root@ctrl ~]# kubectl get nodes

NAME               STATUS   ROLES           AGE     VERSION

dlp-1.srv.world    Ready    control-plane   124m    v1.31.9

dlp.srv.world      Ready    control-plane   3h9m    v1.31.9

node01.srv.world   Ready    <none>          3h4m    v1.31.9

node02.srv.world   Ready    <none>          3h3m    v1.31.9

node03.srv.world   Ready    <none>          6m49s   v1.31.9

prepare to remove a target node

--ignore-daemonsets => ignore pods in DaemonSet

--delete-emptydir-data => ignore pods that has emptyDir volumes

--force => also remove pods that was created as a pod, not as deployment or others

[root@ctrl ~]# kubectl drain dlp-1.srv.world --ignore-daemonsets --delete-emptydir-data --force

node/dlp-1.srv.world cordoned

Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-m9cc9, kube-system/kube-proxy-4r2xt

node/dlp-1.srv.world drained

verify a few minutes later

[root@ctrl ~]# kubectl get nodes dlp-1.srv.world

NAME              STATUS                     ROLES           AGE    VERSION

dlp-1.srv.world   Ready,SchedulingDisabled   control-plane   125m   v1.31.9

run delete method

[root@ctrl ~]# kubectl delete node dlp-1.srv.world

node "dlp-1.srv.world" deleted

[root@ctrl ~]# kubectl get nodes

NAME               STATUS   ROLES           AGE     VERSION

dlp.srv.world      Ready    control-plane   3h11m   v1.31.9

node01.srv.world   Ready    <none>          3h6m    v1.31.9

node02.srv.world   Ready    <none>          3h5m    v1.31.9

node03.srv.world   Ready    <none>          8m46s   v1.31.9

Step 3

On the removed Node, Reset kubeadm settings.

[root@dlp-1 ~]# kubeadm reset

[reset] Reading configuration from the cluster...

[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

W0520 12:50:42.776277   36142 reset.go:123] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get node registration: failed to get corresponding node: nodes "dlp-1.srv.world" not found

W0520 12:50:42.776743   36142 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.

[reset] Are you sure you want to proceed? [y/N]: y

[preflight] Running pre-flight checks

W0520 12:50:44.299139   36142 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory

[reset] Deleted contents of the etcd data directory: /var/lib/etcd

[reset] Stopping the kubelet service

[reset] Unmounting mounted directories in "/var/lib/kubelet"

[reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki]

[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/super-admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.

If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)

to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.

Please, check the contents of the $HOME/.kube/config file.