目前我的k8s环境容器是使用的containerd,删除POD以后一直terminating,无法销毁,然后查看terminating的POD有以下事件报错
Normal Killing 8m16s kubelet Stopping container orion-admission-hooks Warning FailedKillPod 8m15s kubelet error killing pod: failed to "KillPodSandbox" for "654ae46c-95bd-485d-950f-eaaefe2a7ca0" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to remove network namespace for sandbox \"0a7f8f58125be5de252499661bed130f238748de61bc998c705ed0780db32507\": failed to remove netns: unlinkat /run/netns/cni-19690d53-50e9-8185-d204-92e90d76d565: device or resource busy" Warning FailedKillPod 3m5s (x26 over 8m14s) kubelet error killing pod: failed to "KillPodSandbox" for "654ae46c-95bd-485d-950f-eaaefe2a7ca0" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to check network namespace closed: remove netns: unlinkat /var/run/netns/cni-19690d53-50e9-8185-d204-92e90d76d565: device or resource busy"
导致这个问题的原因是在CentOS7.4引入了一个新的参数来控制内核的行为。 /proc/sys/fs/may_detach_mounts 默认设置为0;当系统有容器运行的时候,需要将该值设置为1。
我们需要修改内核方面的配置
echo 1 > /proc/sys/fs/may_detach_mounts
使用上述方法以后POD就销毁了,问题解决,但是上述方式重启以后就失效了,我们需要添加到sysctl.conf配置中,并使其生效
echo "fs.may_detach_mounts = 1" >> /etc/sysctl.conf sysctl -p