三、容器环境操作
容器选择
安装Docker或Containered
Kubernetes 默认容器运行时(CRI)为 Docker，所以需要先在各个节点中安装 Docker。另可选安装containerd
从kubernetes 1.24开始，dockershim已经从kubelet中移除，但因为历史问题docker却不支持kubernetes主推的CRI（容器运行时接口）标准，所以docker不能再作为kubernetes的容器运行时了，即从kubernetesv1.24开始不再使用docker了。
但是如果想继续使用docker的话，可以在kubelet和docker之间加上一个中间层cri-docker。cri-docker是一个支持CRI标准的shim（垫片）。一头通过CRI跟kubelet交互，另一头跟docker api交互，从而间接的实现了kubernetes以docker作为容器运行时。但是这种架构缺点也很明显，调用链更长，效率更低。
在安装Docker前需要确保操作系统内核版本为 3.10以上，因此需要CentOS7 ，CentOS7内核版本为3.10.
推荐使用containerd作为kubernetes的容器运行。
containerd配置起来比较麻烦
拉取镜像时需ctr ctrctl或安装nerdctl,推送镜像不方便
下载镜像的时候增加–all-platforms参数,譬如：ctr i pull –all-platforms,否则推送时出错，而加上–all-platforms太费带宽和空间，本来几秒的事，搞了几分钟，600多M
推送镜像的时候增加 用户和密码和–plain-http ctr i push –plain-http=true -u admin:xxxxxx ,不想-u user:password 每次必须使用 ctr pull/ctr push， 可以使用nerdctl
nerdctl 构建的机制和 docker 是完全不同的。
docker 首先会检查本地是否有 Dockerfile 中 FROM 的镜像。如果有，直接使用。没有则通过网络下载镜像;
nerdctl 会根据 Dockerfile FROM参数指定镜像的域名去网上找这个镜像，找到后确认和本地同名镜像校验无误之后，才会使用本地的镜像构建新镜像。
harbor而且要求一定要https，http不行.
方式一 安装docker
docker不支持centos6
推荐在harbor安装docker,k8s安装containered
如果之前安装过低版本docker，卸载
yum remove -y docker \
docker-client \
docker-client-latest \
docker-ce-cli \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine在master和node上操作
1 切换镜像源
方式一
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
方式二
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's/download.docker.com/mirrors.aliyun.com\/docker-ce/g' /etc/yum.repos.d/docker-ce.repo
yum makecache fast2 查看当前镜像源中支持的docker版本
yum list docker-ce --showduplicates
yum list docker-ce-cli --showduplicates3 安装特定版本的docker-ce
kubernetes各版本对应支持的docker版本列表
https://github.com/kubernetes/kubernetes/releases
kubernetes v1.28.2 支持 docker 24.0.5 ,安装docker-ce-24.0.6-1.el7
必须指定–setopt=obsoletes=0，否则yum会自动安装更高版本
yum install –setopt=obsoletes=0 docker-ce-24.0.6-1.el7 -y
Installing:
docker-ce x86_64 3:24.0.6-1.el7 docker-ce-stable 24 M
Installing for dependencies:
audit-libs-python x86_64 2.8.5-4.el7 base 76 k
checkpolicy x86_64 2.5-8.el7 base 295 k
container-selinux noarch 2:2.119.2-1.911c772.el7_8 extras 40 k
containerd.io x86_64 1.6.24-3.1.el7 docker-ce-stable 34 M
docker-buildx-plugin x86_64 0.11.2-1.el7 docker-ce-stable 13 M
docker-ce-cli x86_64 1:24.0.6-1.el7 docker-ce-stable 13 M
docker-ce-rootless-extras x86_64 24.0.6-1.el7 docker-ce-stable 9.1 M
docker-compose-plugin x86_64 2.21.0-1.el7 docker-ce-stable 13 M
fuse-overlayfs x86_64 0.7.2-6.el7_8 extras 54 k
fuse3-libs x86_64 3.6.1-4.el7 extras 82 k
libcgroup x86_64 0.41-21.el7 base 66 k
libsemanage-python x86_64 2.5-14.el7 base 113 k
policycoreutils-python x86_64 2.5-34.el7 base 457 k
python-IPy noarch 0.75-6.el7 base 32 k
setools-libs x86_64 3.3.8-4.el7 base 620 k
slirp4netns x86_64 0.4.3-4.el7_8 extras 81 k
Transaction Summary
================================================================================================================================================================================================================================================================4 添加一个配置文件
Docker在默认情况下使用的Cgroup Driver为cgroupfs，而kubernetes推荐使用systemd来代替cgroupfs;
第二行配置第三方docker仓库
mkdir - p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors":["http://hub-mirror.c.163.com"],
"exec-opts":["native.cgroupdriver=systemd"],
"data-root": "/var/lib/docker",
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 5,
"log-driver":"json-file",
"log-opts": {
"max-size": "300m",
"max-file": "2"
},
"live-restore": true
}
EOF5 启动docker，并设置为开机自启
systemctl daemon-reload
systemctl restart docker && systemctl enable docker
6 检查docker状态和版本
docker version
Client: Docker Engine - Community
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:35:25 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:34:28 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0
systemctl start docker # 启动docker服务
systemctl stop docker # 停止docker服务
systemctl restart docker # 重启docker服务
#测试
sudo docker run hello-world在所有节点安装cri-docker
yum install -y libcgroup
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4-3.el8.x86_64.rpm
rpm -ivh cri-dockerd-0.3.4-3.el8.x86_64.rpm
vim /usr/lib/systemd/system/cri-docker.service
----
#修改第10行内容
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 --container-runtime-endpoint fd://
----
systemctl start cri-docker
systemctl enable cri-docker方式二 安装containerd
containerd 安装
wget https://github.com/containerd/containerd/releases/download/v1.7.11/cri-containerd-cni-1.7.11-linux-amd64.tar.gz
wget https://github.com/containerd/containerd/releases/download/v1.7.1/containerd-1.7.1-linux-amd64.tar.gz
tar xvf containerd-1.7.1-linux-amd64.tar.gz
mv bin/* /usr/local/bin/
tar xzf cri-containerd-cni-1.7.11-linux-amd64.tar.gz
#生成containerd 配置文件
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
配置 containerd cgroup 驱动程序 systemd（所有节点）
当 systemd 是选定的初始化系统时,要将 systemd 设置为 cgroup 驱动
kubernets 自v1.24.0 后，就不再使用 docker.shim，替换采用 containerd 作为容器运行时端点
vi /etc/containerd/config.toml
#SystemdCgroup的值改为true
SystemdCgroup = true
#由于国内下载不到registry.k8s.io的镜像，修改sandbox_image的值为：
#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml
sed -i "s#registry.k8s.io/pause:3.8#registry.aliyuncs.com/google_containers/pause:3.9#g" /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.tomlvi /etc/containerd/config.toml
address = "/run/containerd/containerd.sock"
socket_path = "/var/run/nri/nri.sock"启动containerd服务
mkdir -p /usr/local/lib/systemd/system
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
mv containerd.service /usr/lib/systemd/system/
cat /usr/lib/systemd/system/containerd.service
cat > /usr/lib/systemd/system/containerd.service << EOF
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
#uncomment to fallback to legacy CRI plugin implementation with podsandbox support.
#Environment="DISABLE_CRI_SANDBOXES=1"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOFmodprobe overlay
modprobe: FATAL: Module overlay not found.lsmod | grep overlay
重新编译内核支持overlay
systemctl daemon-reload
systemctl enable --now containerd
systemctl status containerd验证安装
ctr version
Client:
Version: v1.7.1
Revision: 1677a17964311325ed1c31e2c0a3589ce6d5c30d
Go version: go1.20.4
Server:
Version: v1.7.1
Revision: 1677a17964311325ed1c31e2c0a3589ce6d5c30d
UUID: 0a65fe08-25a6-4bda-a66f-c6f52e334e70安装runc
安装runc
以下步骤所有节点都执行。
准备文件
wget https://github.com//opencontainers/runc/releases/download/v1.1.7/runc.amd64
chmod +x runc.amd64查找containerd安装时已安装的runc所在的位置，如果不存在runc文件，则直接进行下一步
which runc
/usr/bin/runc
替换上一步的结果文件
cp runc.amd64 /usr/bin/runc验证runc安装
runc -v
runc version 1.1.7
commit: v1.1.7-0-g860f061b
spec: 1.0.2-dev
go: go1.20.3
libseccomp: 2.5.4安装CNI插件
安装CNI插件
下载地址：https://github.com/containernetworking/plugins/releases
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz安装crictl
安装 kubernetes 社区提供的 containerd 客户端工具 crictl
根据 https://www.downloadkubernetes.com/ 确定即将安装的Kubernetes版本， 本次即将安装 Kubernetes v1.28.0。 客户端工具 crictl 的版本号需和即将安装的 Kubernetes 版本号一致。
crictl 命令基本和docker一样的用法
下载地址：https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md
#wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.27.1/crictl-v1.27.1-linux-amd64.tar.gz
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz
#tar -xf crictl-v1.27.1-linux-amd64.tar.gz -C /usr/local/bin
tar -xf crictl-v1.28.0-linux-amd64.tar.gz -C /usr/local/bin
# 编辑配置文件
cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 30
debug: false
pull-image-on-create: false
EOF
systemctl daemon-reload && systemctl restart containerd && systemctl status containerd
crictl --version
crictl version v1.28.0配置Containerd运行时镜像加速器
Containerd通过在启动时指定一个配置文件夹，使后续所有镜像仓库相关的配置都可以在里面热加载，无需重启Containerd。
在/etc/containerd/config.toml配置文件中插入如下config_path：
config_path = "/etc/containerd/certs.d"
说明
/etc/containerd/config.toml非默认路径，您可以根据实际使用情况进行调整。
若已有plugins."io.containerd.grpc.v1.cri".registry，则在下面添加一行，注意要有Indent。若没有，则可以在任意地方写入。
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"之后需要检查配置文件中是否有原有mirror相关的配置，如下：
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]若有原有mirror相关的配置，则需要清理。
执行systemctl restart containerd重启Containerd。
若启动失败，执行journalctl -u containerd检查为何失败，通常是配置文件仍有冲突导致，您可以依据报错做相应调整。
在步骤一中指定的config_path路径中创建docker.io/hosts.toml文件。
在文件中写入如下配置。
mkdir /etc/containerd/certs.d/docker.io -pv
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://xxx.mirror.aliyuncs.com"]
capabilities = ["pull", "resolve"]
EOFsystemctl restart containerd
ctr和crictl
ctr是由containerd提供的一个客户端工具。
crictl是CRI兼容的容器运行时命令接口，和containerd无关，由kubernetes提供，可以使用它来检查和调试k8s节点上的容器运行时和应用程序。
crictl pull docker.io/library/nginx
crictl images
crictl rmi docker.io/library/nginx
ctr image pull docker.io/library/nginx:alpine
ctr image ls
ctr image check
ctr image tag docker.io/library/nginx:alpine harbor.k8s.local/course/nginx:alpine
ctr container create docker.io/library/nginx:alpine nginx
ctr container ls
ctr container info nginx
ctr container rm nginx
No Responses (yet)
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.