查看etc目录，主要为containerd服务管理配置文件及cni虚拟网卡配置文件 # ls etc cni crictl.yaml systemd # ls etc/systemd/ system # ls etc/systemd/system/ containerd.service
查看opt目录，主要为gce环境中使用containerd配置文件及cni插件 # ls opt cni containerd # ls opt/containerd/ cluster # ls opt/containerd/cluster/ gce version # ls opt/containerd/cluster/gce cloud-init cni.template configure.sh env
查看usr目录，主要为containerd运行时文件，包含runc # ls usr local # ls usr/local/ bin sbin # ls usr/local/bin containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr # ls usr/local/sbin runc
# Copyright The containerd Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License.
Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity # Comment TasksMax if your systemd version does not supports it. # Only systemd 226 and above support this version. TasksMax=infinity OOMScoreAdjust=-999
[Install] WantedBy=multi-user.target
2.2.2.3 复制containerd运行时文件至系统
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
查看宿主机/usr/local/bin目录，里面没有任何内容。 # ls /usr/local/bin/
查看解压后usr/local/bin目录，里面包含containerd运行时文件 # ls usr/ local # ls usr/local/ bin sbin # ls usr/local/bin/ containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2 containerd-stress crictl critest ctd-decoder ctr
复制containerd文件至/usr/local/bin目录中，本次可仅复制containerd一个文件也可复制全部文件。 # cp usr/local/bin/containerd /usr/local/bin/ # ls /usr/local/bin/ containerd
2.2.2.4 添加containerd.service文件至系统
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
查看解压后的etc/system目录 # ls etc cni crictl.yaml systemd
containerd is a high performance container runtime whose daemon can be started by using this command. If none of the *config*, *publish*, or *help* commands are specified, the default action of the **containerd** command is to start the containerd daemon in the foreground.
A default configuration is used if no TOML configuration is specified or located at the default file location. The *containerd config* command can be used to generate the default configuration for containerd. The output of that command can be used and modified as necessary as a custom configuration.
COMMANDS: config information on the containerd config publish binary to publish events to containerd oci-hook provides a base for OCI runtime hooks to allow arguments to be injected. help, h Shows a list of commands or help for one command
GLOBAL OPTIONS: --config value, -c value path to the configuration file (default: "/etc/containerd/config.toml") --log-level value, -l value set the logging level [trace, debug, info, warn, error, fatal, panic] --address value, -a value address for containerd's GRPC server --root value containerd root directory --state value containerd state directory --help, -h show help --version, -v print the version
# systemctl enable containerd Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service. # systemctl start containerd
1 2 3 4 5 6 7 8 9 10 11 12
# systemctl status containerd ● containerd.service - containerd container runtime Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled) Active: active (running) since 五 2022-02-1813:02:37 CST; 7s ago Docs: https://containerd.io Process: 60383 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 60384 (containerd) Tasks: 8 Memory: 20.0M CGroup: /system.slice/containerd.service └─60384 /usr/local/bin/containerd ......
ctr is an unsupported debug and administrative client for interacting with the containerd daemon. Because it is unsupported, the commands, options, and operations are not guaranteed to be backward compatible or stable from release to release of the containerd project.
COMMANDS: plugins, plugin provides information about containerd plugins version print the client and server versions containers, c, container manage containers content manage content events, event display containerd events images, image, i manage images leases manage leases namespaces, namespace, ns manage namespaces pprof provide golang pprof outputs for containerd run run a container snapshots, snapshot manage snapshots tasks, t, task manage tasks install install a new package oci OCI tools shim interact with a shim directly help, h Shows a list of commands or help for one command
GLOBAL OPTIONS: --debug enable debug output in logs --address value, -a value address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS] --timeout value total timeout for ctr commands (default: 0s) --connect-timeout value timeout for connecting to containerd (default: 0s) --namespace value, -n value namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE] --help, -h show help --version, -v print the version
COMMANDS: check check existing images to ensure all content is available locally export export images import import images list, ls list images known to containerd mountmount an image to a target path unmount unmount the image from the target pull pull an image from a remote push push an image to a remote delete, del, remove, rm remove one or more images by reference tag tag an image label set and clear labels for an image convert convert an image
OPTIONS: --help, -h show help
3.2 查看镜像
1 2
# ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS
把已下载的容器镜像挂载至当前文件系统 # ctr images mount docker.io/library/nginx:alpine /mnt sha256:af2fcce448e2e4451a5f4796a9bf9cb5c9b5f88e0d6d10029cada42fb9d268ac /mnt [root@localhost ~]# ls /mnt bin dev docker-entrypoint.d docker-entrypoint.sh etc home lib media mnt opt proc root run sbin srv sys tmp usr var
1 2
卸载 # umount /mnt
3.5 镜像导出
1 2
把容器镜像导出 # ctr i export --all-platforms nginx.img docker.io/library/nginx:alpine
ctr is an unsupported debug and administrative client for interacting with the containerd daemon. Because it is unsupported, the commands, options, and operations are not guaranteed to be backward compatible or stable from release to release of the containerd project.
COMMANDS: plugins, plugin provides information about containerd plugins version print the client and server versions containers, c, container manage containers content manage content events, event display containerd events images, image, i manage images leases manage leases namespaces, namespace, ns manage namespaces pprof provide golang pprof outputs for containerd run run a container snapshots, snapshot manage snapshots tasks, t, task manage tasks install install a new package oci OCI tools shim interact with a shim directly help, h Shows a list of commands or help for one command
GLOBAL OPTIONS: --debug enable debug output in logs --address value, -a value address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS] --timeout value total timeout for ctr commands (default: 0s) --connect-timeout value timeout for connecting to containerd (default: 0s) --namespace value, -n value namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE] --help, -h show help --version, -v print the version
COMMANDS: create create container delete, del, remove, rm delete one or more existing containers info get info about a container list, ls list containers label set and clear labels for a container checkpoint checkpoint a container restore restore a container from checkpoint
USAGE: ctr run [commandoptions] [flags] Image|RootFS ID [COMMAND] [ARG...]
OPTIONS: --rm remove the container after running --null-io send all IO to /dev/null --log-uri value log uri --detach, -d detach from the task after it has started execution --fifo-dir value directory used for storing IO FIFOs --cgroup value cgroup path (To disable use of cgroup, set to "" explicitly) --platform value run image for specific platform --cni enable cni networking for the container --runc-binary value specify runc-compatible binary --runc-root value specify runc-compatible root --runc-systemd-cgroupstart runc with systemd cgroup manager --uidmap container-uid:host-uid:length run inside a user namespace with the specified UID mapping range; specified with the format container-uid:host-uid:length --gidmap container-gid:host-gid:length run inside a user namespace with the specified GID mapping range; specified with the format container-gid:host-gid:length --remap-labels provide the user namespace ID remapping to the snapshotter via label options; requires snapshotter support --cpus value set the CFS cpu quota (default: 0) --cpu-shares value set the cpu shares (default: 1024) --snapshotter value snapshotter name. Empty value stands for the default value. [$CONTAINERD_SNAPSHOTTER] --snapshotter-label value labels added to the new snapshot for this container. --config value, -c value path to the runtime-specific spec config file --cwd value specify the working directory of the process --env value specify additional container environment variables (e.g. FOO=bar) --env-file value specify additional container environment variables in a file(e.g. FOO=bar, one per line) --label value specify additional labels (e.g. foo=bar) --mount value specify additional container mount (e.g. type=bind,src=/tmp,dst=/host,options=rbind:ro) --net-host enable host networking for the container --privileged run privileged container --read-onlyset the containers filesystem as readonly --runtime value runtime name (default: "io.containerd.runc.v2") --runtime-config-path value optional runtime config path --tty, -t allocate a TTY for the container --with-ns value specify existing Linux namespaces to join at container runtime (format '<nstype>:<path>') --pid-file value file path to write the task's pid --gpus value add gpus to the container --allow-new-privs turn off OCI spec's NoNewPrivileges feature flag --memory-limit value memory limit (in bytes) for the container (default: 0) --device value file path to a device to add to the container; or a path to a directory tree of devices to add to the container --cap-add value add Linux capabilities (Set capabilities with 'CAP_' prefix) --cap-drop value drop Linux capabilities (Set capabilities with 'CAP_' prefix) --seccomp enable the default seccomp profile --seccomp-profile value file path to custom seccomp profile. seccomp must be set to true, before using seccomp-profile --apparmor-default-profile value enable AppArmor with the default profile with the specified name, e.g. "cri-containerd.apparmor.d" --apparmor-profile value enable AppArmor with an existing custom profile --rdt-class value name of the RDT classtoassociatethecontainerwith. SpecifiesaClassofService (CLOS) forcacheandmemorybandwidthmanagement. --rootfsusecustomrootfsthatisnotmanagedbycontainerdsnapshotter --no-pivotdisableuseofpivot-root (linuxonly) --cpu-quotavalueLimitCPUCFSquota (default: -1) --cpu-periodvalueLimitCPUCFSperiod (default: 0) --rootfs-propagationvaluesetthepropagationofthecontainerrootfs
1 2
说明： 使用`ctr run`命令可以创建一个静态容器并使其运行。一步到位运行容器。
4.2 查看容器
container表示静态容器，可用c缩写代表container
1 2
# ctr container ls CONTAINER IMAGE RUNTIME
或
1 2
# ctr c ls CONTAINER IMAGE RUNTIME
4.3 查看任务
task表示容器里跑的进程, 可用t缩写代表task
1 2
# ctr task ls TASK PID STATUS
或
1 2
# ctr t ls TASK PID STATUS
4.4 创建静态容器
1
# ctr c create docker.io/library/nginx:alpine nginx1
1 2 3
# ctr container ls CONTAINER IMAGE RUNTIME nginx1 docker.io/library/nginx:alpine io.containerd.runc.v2
启动task，即表时在容器中运行了进程，即为动态容器。 # ctr task start -d nginx1 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
1 2
说明： -d表示daemon或者后台的意思，否则会卡住终端
1 2 3 4
查看容器所在宿主机进程，是以宿主机进程的方式存在的。 # ctr task ls TASK PID STATUS nginx1 3395 RUNNING
ifconfig 查看网卡信息 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
curl127.0.0.1 访问本地提供的web服务 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 00000000--:--:----:--:----:--:--0<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p>
<p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you forusing nginx.</em></p> </body> </html> 10061510061500429k 0--:--:----:--:----:--:--600k
1 2
说明： 为exec进程设定一个id，可以随意输入，只要保证唯一即可，也可使用$RANDOM变量。
4.7 直接运行一个动态容器
1 2 3
# ctr run -d --net-host docker.io/library/nginx:alpine nginx2 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
COMMANDS: attach attach to the IO of a running container checkpoint checkpoint a container delete, del, remove, rm delete one or more tasks exec execute additional processes in an existing container list, ls list tasks kill signal a container (default: SIGTERM) pause pause an existing container ps list processes for container resume resume a paused container startstart a container that has been created metrics, metric get a single data point of metrics for a task with the built-in Linux runtime
OPTIONS: --help, -h show help
1 2
使用kill命令停止容器中运行的进程，既为停止容器 # ctr tasks kill nginx2
1 2 3 4 5
查看容器停止后状态，STATUS为STOPPED # ctr tasks ls TASK PID STATUS nginx1 3395 RUNNING nginx2 4061 STOPPED
COMMANDS: create, c create a new namespace list, ls list namespaces remove, rm remove one or more namespaces label set and clear labels for a namespace
OPTIONS: --help, -h show help
1 2 3 4 5
列出已有namespace # ctr namespace ls NAME LABELS default k8s.io
1 2 3 4 5 6 7 8
创建namespace # ctr namespace create kubemsb
[root@localhost ~]# ctr namespace ls NAME LABELS default k8s.io kubemsb 此命名空间为新添加的
1 2 3 4 5 6 7 8 9
删除namespace # ctr namespace rm kubemsb kubemsb
再次查看是否删除 [root@localhost ~]# ctr namespace ls NAME LABELS default k8s.io
1 2 3
查看指定namespace中是否有用户进程在运行 # ctr -n kubemsb tasks ls TASK PID STATUS
必须在scripts目录中执行，需要依赖exec-plugins.sh文件，再次进入scripts目录 [root@localhostcni]# cd scripts/
查看执行脚本文件 [root@localhostscripts]# ls docker-run.sh exec-plugins.sh priv-net-run.sh release.sh
执行脚本文件，基于/etc/cni/net.d/目录中的*.conf配置文件生成容器网络 [root@localhostscripts]# CNI_PATH=/home/cni-plugins ./priv-net-run.sh echo "Hello World" Hello World
1 2 3 4 5 6 7 8 9
在宿主机上查看是否生成容器网络名为cni0的网桥 # ip a s ...... 5: cni0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 36:af:7a:4a:d6:12 brd ff:ff:ff:ff:ff:ff inet 10.66.0.1/16 brd 10.66.255.255 scope global cni0 valid_lft forever preferred_lft forever inet6 fe80::34af:7aff:fe4a:d612/64 scope link valid_lft forever preferred_lft forever
1 2 3 4 5 6
在宿主机上查看其路由表情况 # ip route default via 192.168.10.2 dev ens33 proto dhcp metric 100 10.66.0.0/16 dev cni0 proto kernel scope link src 10.66.0.1 192.168.10.0/24 dev ens33 proto kernel scope link src 192.168.10.164 metric 100 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
7.2 为Containerd容器配置网络功能
7.2.1 创建一个容器
1 2 3 4 5 6 7 8 9 10 11 12 13 14
# ctr images ls REF TYPE DIGEST SIZE PLATFORMS LABELS
修改Docker服务文件，以便使用已安装的containerd。 # vim /etc/systemd/system/multi-user.target.wants/docker.service
修改前： [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 此处 ExecReload=/bin/kill-s HUP $MAINPID
修改后：
[Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd --containerd /run/containerd/containerd.sock --debug 此处 ExecReload=/bin/kill-s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always
1 2 3 4 5
设置docker daemon启动并设置其开机自启动 # systemctl daemon-reload # systemctl enable docker Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. # systemctl start docker
使用docker运行容器 # docker run -d nginx:latest ...... 219a9c6727bcd162d0a4868746c513a277276a110f47e15368b4229988003c13
1 2 3 4
使用docker ps命令查看正在运行的容器 # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 219a9c6727bc nginx:latest "/docker-entrypoint.…"14 seconds ago Up 13 seconds 80/tcp happy_tu
1 2 3 4 5 6 7
使用ctr查看是否添加一个新的namespace，本案例中发现添加一个moby命名空间，即为docker使用的命名空间。 # ctr namespace ls NAME LABELS default k8s.io kubemsb moby
