Skip to content

k8s_安装4_私仓harbor

四、harbor仓库

k8s经常使用的镜像地址

从 Kubernetes 1.25 开始，我们的容器镜像注册中心已经从 k8s.gcr.io 更改为 registry.k8s.io .
registry.aliyuncs.com/google_containers是定时同步kubernetes的镜像到阿里镜像仓库服务的，但只是K8S组件的镜像，阿里云镜像仓库有谷歌和RedHat的镜像，但是不全。
当我们下载k8s.gcr.io，gcr.io镜像和quay.io镜像，可以把k8s.gcr.io，gcr.io， quay.io镜像换成阿里云或其它国内镜像加速地址下，如下所示：

k8s中相关镜像 

#k8s.io  不能访问
registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2

docker pull registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2
docker pull registry.k8s.io/dns/k8s-dns-node-cache:1.22.28
docker pull k8s.mirror.nju.edu.cn/dns/k8s-dns-node-cache:1.22.28

# gcr.io
gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
docker pull m.daocloud.io/gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
docker pull m.daocloud.io/gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.2

# k8s.gcr.io
k8s.gcr.io/pause:3.2
registry.aliyuncs.com/google_containers/pause:3.2

#docker.io
docker.io/flannel/flannel-cni-plugin:v1.2.0
https://registry.cn-hangzhou.aliyuncs.com

#quay.io 可以下载
quay.io/external_storage/nfs-client-provisioner:latest
docker pull quay.nju.edu.cn/jetstack/cert-manager-webhook:v1.13.1

#docker.elastic.co 可以下载，有时慢
docker.elastic.co/beats/filebeat:8.11.0

#没有登录，需先登录
Error response from daemon: pull access denied for registry.aliyuncs.com/google_containers/cert-manager-webhook, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
docker login registry.cn-hangzhou.aliyuncs.com
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/cert-manager-webhook:v1.13.2

私仓

Docker比较流行使用的三种私有仓库
Docker Registry，Nexus，Harbor

Docker Registry

通过使用 Docker Compose，我们可以轻松地在本地部署 Docker Registry.
Harbor完全是在Registry上的封装，目前比Registry功能主要的强化在于：

  • 提供UI界面
  • 提供基于角色管理的用户权限
  • 提供用户操作记录审计确认
  • 提供镜像
  • 提供对Helm Chart等的支持

Nexus

Java 开发中的那个 Maven 私服，对对对，就是这个 Nexus。Nexus它也可以应用于 docker 仓库。

优势

docker就可以安装，可以代理其它仓库并缓存到本地。
nexus3覆盖更全面，啥都可以做，是一个混合仓库maven、yum、npm的私服

repositories说明：(参考maven已有repositories介绍，因为刚开始进来maven相关的库是已经存在的)

  • maven-public：它的类型是group(分组)。即：仓库所有的访问读取入口，都是从 public 开始，读取会分别从 snapshots、releases、central 中都去找，只要其中一个找到就读取回来），就是本地仓库找不到，就去配置的网络仓库中去找【网络仓库地址，会在docker-public中配置阿里云加速镜像地址】
  • maven-releases 发布后的jar包，放到release中
  • maven-snapshots 测试的jar包，放到snapshots中
  • maven-central 它的类型是proxy(代理)。代理不存数据，是只读的。这个类似配置maven仓库时我们配置的一个aliyun仓库，帮我们去代理到aliyun仓库

nexus中有个默认自带的仓库列表，里面包含了各种各样的仓库。
这些仓库主要分为三类，代理仓库、宿主仓库和仓库组。

  • 代理仓库：代理仓库主要是让使用者通过代理仓库来间接访问外部的第三方远程仓库的。代理仓库会从被代理的仓库中下载构件，缓存在代理仓库中以供maven用户使用。

  • 宿主仓库： 宿主仓库主要是给我们自己用的，主要有2点作用：
    　　　　将私有的一些构件通过nexus中网页的方式上传到宿主仓库中给其他同事使用
    　　　　将自己开发好的一些构件发布到nexus的宿主仓库中以供其他同事使用。

  • 仓库组：（默认maven-public）仓库组中可以有多个代理仓库和宿主仓库，而maven用户只用访问一个仓库组就可以间接地访问这个组内所有的仓库，仓库组中多个仓库是有顺序的，当maven用户从仓库组下载构件时，仓库组会按顺序依次在组内的仓库中查找组件，查找到了立即返回给本地仓库，所以一般情况我们会将速度快的放在前面。仓库组内部实际上是没有构件内容的，他只是起到一个请求转发的作用，将maven用户下载构件的请求转发给组内的其他仓库处理。

Harbor 私仓

它是Docker Registry的更高级封装，它除了提供友好的Web UI界面，角色和用户权限管理，用户操作审计等功能外，它还整合了K8s的插件(Add-ons)仓库，即Helm通过chart方式下载，管理，安装K8s插件，而chartmuseum可以提供存储chart数据的仓库【注:helm就相当于k8s的yum】。另外它还整合了两个开源的安全组件，一个是Notary，另一个是Clair，Notary类似于私有CA中心，而Clair则是容器安全扫描工具，它通过各大厂商提供的CVE漏洞库来获取最新漏洞信息，并扫描用户上传的容器是否存在已知的漏洞信息，这两个安全功能对于企业级私有仓库来说是非常具有意义的。
harbor 的优势很明显，特别是可以自建文件夹进行分组这点就非常好。其实说实话，作为一个私有的镜像仓库，harbor 已经做得很好的了，唯一的缺点是它无法帮你下载镜像。
相比Nexus要费资源。使用 Harbor 必须要先安装 docker 以及 docker-compose。

安装参考:
https://agou-ops.cn/post/containerdharbor%E7%A7%81%E6%9C%89%E4%BB%93https/

Harbor相关地址

官网：​ ​https://goharbor.io/​​

Github地址：​ ​https://github.com/goharbor/harbor​​

操作文档：​ ​https://goharbor.io/docs/​

Harbor安装有多种方式：

在线安装：从Docker Hub下载Harbor相关镜像，因此安装软件包非常小
离线安装：安装包包含部署的相关镜像，因此安装包比较大

只在仓库机器上执行
Harbor安装前提条件
Harbor被部署为几个Docker容器。因此，您可以在任何支持Docker的Linux发行版上部署它。目标主机需要安装Docker，并安装Docker Compose。
安装Harbor前，需要安装Docker engine，Docker Compose和Openssl。

私有库默认是不支持删除镜像的，需要修改config.yml配置文件，在storage节点下加入 delete: enabled: true
删tag不会回收空间

Harbor 高可用方式

http://t.csdnimg.cn/k43Ao

  1. 安装两台 Harbor 仓库，他们共同使用一个存储（一般常见的便是 NFS 共享存储）需要额外配置 Redis 和 PostgreSQL 以及 NFS 服务
  2. 安装两台 Harbor 仓库，并互相配置同步关系。

1.docker环境的安装

docker-ce安装

首先先把服务停止了，不要直接卸载

systemctl stop docker

重命名数据目录

把默认的docker目录改一下名称。

mv /var/lib/docker /var/lib/docker-bak

卸载旧版本

Docker 的旧版本被称为 docker 或 docker-engine。如果这些已安装，请卸载它们以及关联 的依赖关系。

yum remove docker docker-common docker-selinux docker-engine -y

安装docker-ce

## 阿里云源
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce

docker version
Client: Docker Engine – Community
Version: 26.1.0
API version: 1.45
Go version: go1.21.9
Git commit: 9714adc
Built: Mon Apr 22 17:09:57 2024
OS/Arch: linux/amd64
Context: default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors":["https://docker.nju.edu.cn/"],
"insecure-registries":["repo.k8s.local"],
"exec-opts":["native.cgroupdriver=systemd"],
"data-root": "/var/lib/docker",
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 5,
"log-driver":"json-file",
"log-opts": {
"max-size": "300m",
"max-file": "2"
},
"experimental": true,
"live-restore": true
}
EOF

报错
Failed to chown socket at step GROUP: No such process
可能是文件被锁定，用户组没添加成功 

lsattr /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/services
----i----------- /etc/passwd
----i----------- /etc/shadow
----i----------- /etc/group
----i----------- /etc/gshadow
----i----------- /etc/services
chattr -i /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/services
groupadd docker

恢复数据目录

安装完docker-ce后，系统会创建新的docker目录，删除新的，如何把备份的改回docker名称。
切记：不要启动docker！！

rm -rf /var/lib/docker
mv /var/lib/docker-bak /var/lib/docker

启动docker服务

systemctl enable containerd

docker -v
Docker version 26.1.0, build 9714adc

ctr version
Client:
  Version:  1.6.31
  Revision: e377cd56a71523140ca6ae87e30244719194a521
  Go version: go1.21.9

Server:
  Version:  1.6.31
  Revision: e377cd56a71523140ca6ae87e30244719194a521
  UUID: 07d08019-3523-4f01-90db-367b21874598

cat /etc/containerd/config.toml

报错1
Error response from daemon: Unknown runtime specified docker-runc
需要针对容器里面的docker-runc改一下名称，用runc替换docker-runc。
grep -rl ‘docker-runc’ /var/lib/docker/containers/ | xargs sed -i ‘s/docker-runc/runc/g’
systemctl restart docker

报错1
level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
使用
systemctl start containerd
不要用
systemctl start docker

2.安装docker-compose：

https://github.com/docker/compose/releases
安装方式

  • 二进制安装docker-compose
  • yum方式安装docker-compose
  • python3+pip 安装

二进制方式安装docker-compose：

简单快速

wget https://github.com/docker/compose/releases/download/v2.12.2/docker-compose-linux-x86_64
wget https://github.com/docker/compose/releases/download/v2.22.0/docker-compose-linux-x86_64
sudo cp -arf docker-compose-linux-x86_64 /usr/bin/docker-compose
sudo chmod +x /usr/bin/docker-compose
ln -s /usr/bin/docker-compose /usr/local/bin/docker-compose

卸载
sudo rm /usr/bin/docker-compose

yum方式安装docker-compose：

安装复杂

相关包安装

yum -y install libjpeg zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
yum install -y libffi-devel zlib1g-dev

openssl升级

centos7默认的openssl版本和python3.10以上的版本不兼容
openssl version

OpenSSL 1.0.2k-fips  26 Jan 2017
wget https://www.openssl.org/source/openssl-1.1.1q.tar.gz
tar -zxvf openssl-1.1.1q.tar.gz
cd openssl-1.1.1q
./config --prefix=/usr/local/openssl 
make -j && make install

ln -s /usr/local/openssl/lib/libcrypto.so.1.1  /usr/lib64/libcrypto.so.1.1
ln -s /usr/local/openssl/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
/usr/local/openssl/bin/openssl version
OpenSSL 1.1.1q  5 Jul 2022

备份原来的openssl文件，可通过whereis openssl查询位置

mv /usr/bin/openssl /usr/bin/openssl.old
mv /usr/include/openssl /usr/include/openssl.old

用新的文件替换旧的文件，执行命令如下：

ln -sf /usr/local/openssl/bin/openssl /usr/bin/openssl
ln -s /usr/local/openssl/include/openssl /usr/include/openssl

openssl version
OpenSSL 1.1.1q  5 Jul 2022
python3+pip 安装
yum install epel-release 
yum install -y python3 python3-pip python3-devel

pip3 –version

pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)
pip 国内镜像加速
mkdir ~/.pip/
vi ~/.pip/pip.conf
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
trusted-host = mirrors.aliyun.com         
disable-pip-version-check = true        
timeout = 120
pip升级
pip3 install --upgrade pip --trusted-host mirrors.aliyun.com

pip3 –version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with ‘-m pip’ instead of running pip directly.
pip 21.3.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

docker-compose 安装
pip3 install docker-compose --trusted-host mirrors.aliyun.com

ln -s /usr/local/python3/bin/docker-compose /usr/bin/docker-compose 

docker-compose version
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
docker-compose 卸载

pip uninstall docker-compose

安装错误

setuptools-rust

File "/tmp/pip-build-6plwm63o/bcrypt/setup.py", line 11, in <module>
        from setuptools_rust import RustExtension
    ModuleNotFoundError: No module named 'setuptools_rust'
pip3 install setuptools-rust
The wheel package is not available.
pip3 install wheel

cffi

Could not find a version that satisfies the requirement cffi>=1.4.1 (from versions: )
pip3 install cffi
Successfully installed cffi-1.15.1 pycparser-2.21

Rust

his package requires Rust >=1.48.0.
pip3 install Rust

zlib,bzip

configure: error: zlib development files not found
yum install zlib zlib-devel
yum install bzip2 bzip2-devel bzip2-libs

libjpeg

The headers or library files could not be found for jpeg,
    a required dependency when compiling Pillow from source.

yum install libjpeg zlib libtiff

3.harbor 安装

准备文件

wget https://github.com/goharbor/harbor/releases/download/v2.8.4/harbor-offline-installer-v2.8.4.tgz
tar xf harbor-offline-installer-v2.8.4.tgz
mv harbor /usr/local/
cd /usr/local/
cd harbor
cp harbor.yml.tmpl harbor.yml

设定访问域名，端口，admin密码，db密码，存储目录

vi harbor.yml
hostname: repo.k8s.local
http:
    port:5100
#https:
  # https port for harbor, default is 443
  #port: 443
  # The path of cert and key files for nginx
  #certificate: /your/certificate/path
  #private_key: /your/private/key/path

harbor_admin_password: Harbor12345

database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123

# The default data volume
data_volume: /data_harbor

初始化

其实这个prepare的作用是用来初始化将harbor.yml转化为/usr/local/harbor/docker-compose.yml文件

./prepare

prepare base dir is set to /usr/local/harbor
Unable to find image 'goharbor/prepare:v2.8.4' locally
v2.8.4: Pulling from goharbor/prepare
b73ab88bdeef: Pull complete 
0a4647ff4f26: Pull complete 
ac87c0a6beec: Pull complete 
58290933e402: Pull complete 
2dd75ae2b8d6: Pull complete 
0432d14b35c2: Pull complete 
1d94d426c05b: Pull complete 
9242105872e7: Pull complete 
7079d6fb028f: Pull complete 
e3e737964616: Pull complete 
Digest: sha256:80837b160b3a876748c4abb68b35389485b4ddcd5de39bb82a7541d3f3051cae
Status: Downloaded newer image for goharbor/prepare:v2.8.4
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

开始安装

./install.sh
[Step 0]: checking if docker is installed ...

Note: docker version: 24.0.6

[Step 1]: checking docker-compose is installed ...

Note: Docker Compose version v2.21.0

[Step 2]: loading Harbor images ...
a074a02dfff1: Loading layer [==================================================>]  37.79MB/37.79MB
a1845a3d89a2: Loading layer [==================================================>]  9.188MB/9.188MB
3f06bc32288c: Loading layer [==================================================>]  3.584kB/3.584kB
245244bd15d4: Loading layer [==================================================>]   2.56kB/2.56kB
42ca8ea5af72: Loading layer [==================================================>]  47.58MB/47.58MB
8d1a6771e613: Loading layer [==================================================>]  48.37MB/48.37MB
Loaded image: goharbor/harbor-jobservice:v2.8.4
9e404a035c29: Loading layer [==================================================>]  84.62MB/84.62MB
8a45a3e2d467: Loading layer [==================================================>]  3.072kB/3.072kB
50103680c597: Loading layer [==================================================>]   59.9kB/59.9kB
7da34aa8a12d: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.8.4
5d6d0147b133: Loading layer [==================================================>]  89.19MB/89.19MB
f7f30f0432f2: Loading layer [==================================================>]  3.584kB/3.584kB
b895ffa154de: Loading layer [==================================================>]  3.072kB/3.072kB
9fb8c7a01498: Loading layer [==================================================>]   2.56kB/2.56kB
8a232dc48045: Loading layer [==================================================>]  3.072kB/3.072kB
839e0de14204: Loading layer [==================================================>]  3.584kB/3.584kB
3f683bb644b2: Loading layer [==================================================>]  20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.8.4
627fc8f29b12: Loading layer [==================================================>]  115.9MB/115.9MB
b4faf8a74f36: Loading layer [==================================================>]   25.2MB/25.2MB
22c2b4c49c70: Loading layer [==================================================>]   5.12kB/5.12kB
98c144348806: Loading layer [==================================================>]  6.144kB/6.144kB
6f34146f1977: Loading layer [==================================================>]  3.072kB/3.072kB
8dd9b9af7425: Loading layer [==================================================>]  2.048kB/2.048kB
04498149158d: Loading layer [==================================================>]   2.56kB/2.56kB
7600d3f327f6: Loading layer [==================================================>]   2.56kB/2.56kB
e30935897ec8: Loading layer [==================================================>]   2.56kB/2.56kB
b91c1501abe9: Loading layer [==================================================>]  9.728kB/9.728kB
Loaded image: goharbor/harbor-db:v2.8.4
736147cbb70a: Loading layer [==================================================>]  81.13MB/81.13MB
Loaded image: goharbor/nginx-photon:v2.8.4
3ee113d617fa: Loading layer [==================================================>]  72.75MB/72.75MB
8f8c635f3d64: Loading layer [==================================================>]  38.64MB/38.64MB
50ede47ef7b6: Loading layer [==================================================>]  19.94MB/19.94MB
bbe4550fbed9: Loading layer [==================================================>]  65.54kB/65.54kB
6a6c08954476: Loading layer [==================================================>]   2.56kB/2.56kB
4fcee09b3045: Loading layer [==================================================>]  1.536kB/1.536kB
cd9e13a0fadf: Loading layer [==================================================>]  12.29kB/12.29kB
5c4cf244ed4a: Loading layer [==================================================>]  2.123MB/2.123MB
2f207d2f7a63: Loading layer [==================================================>]  419.8kB/419.8kB
Loaded image: goharbor/prepare:v2.8.4
e4e75b52265a: Loading layer [==================================================>]  9.188MB/9.188MB
6ca0b8687881: Loading layer [==================================================>]  3.584kB/3.584kB
2efe438491fa: Loading layer [==================================================>]   2.56kB/2.56kB
6c8c2dc9cf24: Loading layer [==================================================>]  59.31MB/59.31MB
70aa7368b062: Loading layer [==================================================>]  5.632kB/5.632kB
1ad1e6d7b7f2: Loading layer [==================================================>]  116.7kB/116.7kB
fdf3c64c43d4: Loading layer [==================================================>]  44.03kB/44.03kB
af312371ff9e: Loading layer [==================================================>]  60.26MB/60.26MB
2ef0db7a0b49: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.8.4
0cbd65e4d842: Loading layer [==================================================>]  6.699MB/6.699MB
dfd5a1cf5002: Loading layer [==================================================>]  4.096kB/4.096kB
793940424064: Loading layer [==================================================>]  3.072kB/3.072kB
44888bf86da0: Loading layer [==================================================>]    196MB/196MB
561960448b05: Loading layer [==================================================>]   14.1MB/14.1MB
deb1d83b4cbd: Loading layer [==================================================>]  210.9MB/210.9MB
Loaded image: goharbor/trivy-adapter-photon:v2.8.4
dd52e9dde638: Loading layer [==================================================>]  81.13MB/81.13MB
8cfe6bb78139: Loading layer [==================================================>]    6.1MB/6.1MB
8aebde8774f2: Loading layer [==================================================>]  1.233MB/1.233MB
Loaded image: goharbor/harbor-portal:v2.8.4
e2bb7b6f858e: Loading layer [==================================================>]  6.172MB/6.172MB
b58529f5727f: Loading layer [==================================================>]  4.096kB/4.096kB
15b87640160a: Loading layer [==================================================>]  3.072kB/3.072kB
f8cc13293a41: Loading layer [==================================================>]  17.57MB/17.57MB
51175195e0e4: Loading layer [==================================================>]  18.36MB/18.36MB
Loaded image: goharbor/registry-photon:v2.8.4
8c3c80de8e46: Loading layer [==================================================>]  6.167MB/6.167MB
ba247990a26d: Loading layer [==================================================>]  9.143MB/9.143MB
78c730633955: Loading layer [==================================================>]  15.88MB/15.88MB
901f70ff7f25: Loading layer [==================================================>]  29.29MB/29.29MB
e91438791db8: Loading layer [==================================================>]  22.02kB/22.02kB
eb4f8ee41ee3: Loading layer [==================================================>]  15.88MB/15.88MB
Loaded image: goharbor/notary-server-photon:v2.8.4
d3bc6746e3a0: Loading layer [==================================================>]  6.167MB/6.167MB
e9dc9957d190: Loading layer [==================================================>]  9.143MB/9.143MB
6548f7c0890e: Loading layer [==================================================>]  14.47MB/14.47MB
8ab2eab06c9c: Loading layer [==================================================>]  29.29MB/29.29MB
475002e6da05: Loading layer [==================================================>]  22.02kB/22.02kB
70a417415ad1: Loading layer [==================================================>]  14.47MB/14.47MB
Loaded image: goharbor/notary-signer-photon:v2.8.4
bfe8ceaf89b9: Loading layer [==================================================>]  6.172MB/6.172MB
cf503352618a: Loading layer [==================================================>]  4.096kB/4.096kB
21e09698bb69: Loading layer [==================================================>]  17.57MB/17.57MB
7afc834ab33e: Loading layer [==================================================>]  3.072kB/3.072kB
14752e2b2fbf: Loading layer [==================================================>]  31.13MB/31.13MB
8daa88c089ea: Loading layer [==================================================>]  49.49MB/49.49MB
Loaded image: goharbor/harbor-registryctl:v2.8.4
4b4afa104b42: Loading layer [==================================================>]  9.188MB/9.188MB
4ef2e0c082a7: Loading layer [==================================================>]  26.03MB/26.03MB
8eb9f5ee0436: Loading layer [==================================================>]  4.608kB/4.608kB
d449c6ac0cd4: Loading layer [==================================================>]  26.82MB/26.82MB
Loaded image: goharbor/harbor-exporter:v2.8.4

[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/src/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

Note: stopping existing Harbor instance ...

[Step 5]: starting Harbor ...
[+] Running 1/1

--Harbor has been installed and started successfully.----

测试

在外部机器，本机，k8s节点中修改host文件，增加指向
192.168.244.6 repo.k8s.local

vi /etc/hosts

192.168.244.6 repo.k8s.local

在浏览器访问 http://repo.k8s.local:5100/

输入初始密码登录
admin/Harbor12345

忘记密码

docker ps

f11056420a99   goharbor/harbor-core:v2.8.4          "/harbor/entrypoint.…"   3 weeks ago    Up About an hour (healthy)                                                                                        harbor-core  
f69e921a4464   goharbor/harbor-db:v2.8.4            "/docker-entrypoint.…"   3 weeks ago    Up About an hour (healthy)                                                                                        harbor-db

方式一 查看yaml密码

cat /usr/local/harbor/harbor.yml|grep password

harbor_admin_password: Harbor12345
  # The password for the root user of Harbor DB. Change this before any production use.
  password: local_ROOT_!!88

方式二 容器内查看

第一步、 进入容器
docker exec -it "" bash
docker exec -it "f11056420a99" bash

第二步、查看密码
env | grep HARBOR_ADMIN_PASSWORD

HARBOR_ADMIN_PASSWORD=Harbor12345

重置 admin 密码
docker exec -it "" bash
docker exec -it "f69e921a4464" bash

psql -U postgres -d registry
> select * from harbor_user;
> update harbor_user set salt='', password='' where user_id = "<admin user_id>";  
exit

重新启动Harbor私有镜像仓库后，密码就会自动重置为之前安装时配置的Harbor12345

重新启动Harbor私有镜像仓库

# docker-compose down
#./prepare 
# docker-compose up -d

docker-compose ps
no configuration file provided: not found
最常见的原因是没有在docker-compose.yml文件的路径下执行该命令。

cd /usr/local/harbor/
docker-compose ps

web登录时会一直提示用户名密码错误

如果之前清理过镜像可能把镜像删除了。

docker-compose down
./prepare 
prepare base dir is set to /usr/local/harbor
Unable to find image 'goharbor/prepare:v2.8.4' locally
docker-compose up -d

项目用户角色说明

受限访客：受限访客没有项目的完全读取权限。他们可以拉取镜像但不能推送，而且他们看不到日志或项目的其他成员。例如，你可以为来自不同组织的共享项目访问权限的用户创建受限访客。
访客：访客对指定项目具有只读权限。他们可以拉取和重新标记镜像，但不能推送。
开发者：开发者拥有项目的读写权限。
维护者：维护者拥有超越“开发者”的权限，包括扫描镜像、查看复制作业以及删除镜像和helm charts的能力。
项目管理员：创建新项目时，你将被分配给项目的“ProjectAdmin”角色。“ProjectAdmin”除了读写权限外，还有一些管理权限，如添加和删除成员、启动漏洞扫描等。

新建用户

用户管理/创建用户

k8s_user1
k8s_Uus1

k8s_pull
k8s_Pul1

项目

将项目设为公开，并将用户加入项目
library 公开
项目/成员/+用户
k8s_user1
维护者
k8s_pull
访客

项目/新建
k8s 公开
项目/成员/+用户
k8s_user1
维护者
k8s_pull
访客

项目命名规则
方式一 全地址做对应,前面增加私仓地址
docker.io/library/nginx:1.21.4
docker tag docker.io/library/nginx:1.21.4 repo.k8s.local/docker.io/library/nginx:1.21.4

方式二 忽略域名,目录对应
docker.io/library/nginx:1.21.4
docker tag docker.io/library/nginx:1.21.4 repo.k8s.local/library/nginx:1.21.4

方式三 忽略域名和目录，镜像对应
按属性放入目录,如基础服务放入google_containers,组件放library ,应用放app
repo.k8s.local/library/nginx:1.21.4
docker tag docker.io/library/nginx:1.21.4 repo.k8s.local/library/nginx:1.21.4

关闭：

cd /usr/local/harbor/

docker-compose down -v   
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml, compose.yml, compose.yaml

cd /usr/local/harbor
docker-compose down -v

开启：

cd /usr/local/harbor/
docker-compose up -d

Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... done
Creating registry      ... done
Creating registryctl   ... done
Creating harbor-db     ... done
Creating redis         ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED         STATUS                   PORTS                                       NAMES
7ee76b324f96   goharbor/harbor-jobservice:v2.8.4    "/harbor/entrypoint.…"   3 minutes ago   Up 3 minutes (healthy)                                               harbor-jobservice
e1f9af0dfec1   goharbor/nginx-photon:v2.8.4         "nginx -g 'daemon of…"   3 minutes ago   Up 3 minutes (healthy)   0.0.0.0:5100->8080/tcp, :::5100->8080/tcp   nginx
55212a4181c5   goharbor/harbor-core:v2.8.4          "/harbor/entrypoint.…"   3 minutes ago   Up 3 minutes (healthy)                                               harbor-core
bfd166244ad3   goharbor/redis-photon:v2.8.4         "redis-server /etc/r…"   3 minutes ago   Up 3 minutes (healthy)                                               redis
beb5c0c77832   goharbor/harbor-registryctl:v2.8.4   "/home/harbor/start.…"   3 minutes ago   Up 3 minutes (healthy)                                               registryctl
17bd8d7f8a02   goharbor/harbor-db:v2.8.4            "/docker-entrypoint.…"   3 minutes ago   Up 3 minutes (healthy)                                               harbor-db
c7c665923196   goharbor/registry-photon:v2.8.4      "/home/harbor/entryp…"   3 minutes ago   Up 3 minutes (healthy)                                               registry
a08329e11be2   goharbor/harbor-portal:v2.8.4        "nginx -g 'daemon of…"   3 minutes ago   Up 3 minutes (healthy)                                               harbor-portal
d8716d3159a0   goharbor/harbor-log:v2.8.4           "/bin/sh -c /usr/loc…"   3 minutes ago   Up 3 minutes (healthy)   127.0.0.1:1514->10514/tcp                   harbor-log

注册成服务

cat > /lib/systemd/system/harbor.service <<EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /usr/local/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

开机启动

systemctl daemon-reload 
systemctl status harbor
systemctl stop harbor
systemctl start harbor
systemctl enable harbor

防火墙

iptables -A INPUT -p tcp -m tcp --dport 5100 -j ACCEPT
iptabes-save

配置对Harbor的HTTPS访问

注意一旦启用https 那么http会强跳到https,http不再可用
要配置HTTPS，必须创建SSL证书。您可以使用由受信任的第三方CA签名的证书，也可以使用openssl进行自签名证书。
本节介绍如何使用 ​ ​OpenSSL​​​创建CA，以及如何使用CA签署服务器证书和客户端证书。您可以使用其他CA工具进行自签名
在生产环境中，一般是应该从CA获得证书，例如：在阿里云购买域名之后就可以下载相关域名的CA证书了。但是在测试或开发环境中，对于这种自己定义的内网域名，就可以自己生成自己的CA证书。

1. 生成CA证书私钥 ca.key。Generate a CA certificate private key.

openssl genrsa -out ca.key 4096

2. 根据上面生成的CA证书私钥，再来生成CA证书 ca.crt。 Generate the CA certificate.

设置 ​​-subj​​ 选项中的值来反映的组织，例如：省份、地市、域名等等信息。如果使用FQDN 【 (Fully Qualified Domain Name)全限定域名：同时带有主机名和域名的名称。】连接Harbor主机，则必须将其指定为通用名称（​​CN​​​）属性，可以看到示例写的是​​yourdomain.com​​。

openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=repo.k8s.local" \
 -key ca.key \
 -out ca.crt

 # 参数说明：
-new 指生成证书请求
-x509 表示直接输出证书
-key 指定私钥文件
-days 指定证书过期时间为3650天
-out 导出结束后证书文件
-subj 输入证书拥有者信息
生成服务器证书 Generate a Server Certificate

证书通常包含一个​​.crt​​​文件和一个​​.key​​​文件，例如​​yourdomain.com.crt​​​和​​yourdomain.com.key​​。

在这里，因为我上面设置的服务器域名为repo.k8s.local​​​，所以将要生成的证书为repo.k8s.local.crt​​​ 和 repo.k8s.local.key​​。

1.生成CA证书私钥
openssl genrsa -out harbor.key 4096
2.生成证书签名请求（CSR）

注意下使用repo.k8s.local.key 和 repo.k8s.local.csr

openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=repo.k8s.local" \
    -key harbor.key \
    -out harbor.csr
3.生成一个x509 v3扩展文件。Generate an x509 v3 extension file.
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=repo.k8s.local
DNS.2=repo.k8s
EOF
4. 使用该​​v3.ext​​​文件为您的Harbor主机生成证书 yourdomain.com.crt。
openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in harbor.csr \
    -out repo.k8s.local.crt

将服务器证书​​yourdomain.com.crt​​​和密钥​​yourdomain.com.key​​复制到Harbor主机上的certficates文件夹中。

mkdir /usr/local/harbor/certificate
cp repo.k8s.local.crt /usr/local/harbor/certificate/
cp harbor.key /usr/local/harbor/certificate/

双机时也复制到从机

scp  /usr/local/harbor/certificate/* [email protected]:.

编辑配置，打开https支持

vi  /usr/local/harbor/harbor.yml
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /usr/local/harbor/certificate/repo.k8s.local.crt 
  private_key: /usr/local/harbor/certificate/harbor.key
cd /usr/local/harbor/
./prepare 
#重新生成/usr/local/harbor/docker-compose.yml

systemctl daemon-reload 
systemctl stop harbor
systemctl start harbor
systemctl status harbor

netstat -lntp
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      31217/docker-proxy  
tcp        0      0 127.0.0.1:1514          0.0.0.0:*               LISTEN      30700/docker-proxy  
tcp        0      0 0.0.0.0:5100            0.0.0.0:*               LISTEN      31235/docker-proxy

防火墙

iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
iptabes-save

http://repo.k8s.local:5100

https://repo.k8s.local

在浏览器访问，使用nat 5443指到192.168.244.6:443
https://repo.k8s.local:5443/

docker
将服务器证书​​yourdomain.com.crt​​​的编码格式转换为​​yourdomain.com.cert​​​，提供Docker使用

openssl x509 -inform PEM -in repo.k8s.local.crt -out repo.k8s.local.cert

mkdir -p /etc/docker/certs.d/repo.k8s.local/
cp repo.k8s.local.cert /etc/docker/certs.d/repo.k8s.local/
cp harbor.key /etc/docker/certs.d/repo.k8s.local/repo.k8s.local.key
cp ca.crt /etc/docker/certs.d/repo.k8s.local/

双机时也复制到从机

scp  /etc/docker/certs.d/repo.k8s.local/* [email protected]:.

测试docker登录

docker login repo.k8s.local
Username: k8s_user1
Password: k8s_Uus1

修改http为https
vi /etc/docker/daemon.json 

#"insecure-registries":["repo.k8s.local:5100"],
"insecure-registries":["https://repo.k8s.local"],

错误
ctr: failed to resolve reference "xxx.local/library/docker/getting-started
需要拷贝一份上面harbor的ca到系统ca目录并更新
cp ca.crt /usr/local/share/ca-certificates/
/usr/sbin/update-ca-certificates

错误
Error response from daemon: missing client certificate harbor.cert for key harbor.key
注意 harbor.key key文件名和 cert一致

错误

Error response from daemon: Get "https://repo.k8s.local/v2/": Get "https://repo.k8s.local:5433/service/token?account=k8s_user1&client_id=docker&offline_token=true&service=harbor-registry": dial tcp 192.168.244.6:5433: connect: connection refused

harbor配制文件/usr/local/harbor/harbor.yml
修正注释掉 external_url: https://repo.k8s.local:5433

错误
Error response from daemon: Get "https://repo.k8s.local/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority

错误
Job harbor.service/start failed with result ‘dependency’.

systemctl staart harbor
systemctl status harbor

ctr测试推送

ctr -n k8s.io images ls  |grep busybox
repo.k8s.local:5100/google_containers/busybox:9.9                                                                                       application/vnd.docker.distribution.manifest.v2+json      sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee 2.1 MiB   linux/amd64   

ctr -n k8s.io i tag  repo.k8s.local:5100/google_containers/busybox:9.9 repo.k8s.local/google_containers/busybox:9.9

ctr -n k8s.io i push --user k8s_user1:k8s_Uus1 repo.k8s.local/google_containers/busybox:9.9 
ctr: failed to do request: Head "https://repo.k8s.local/v2/google_containers/busybox/blobs/sha256:a416a98b71e224a31ee99cff8e16063554498227d2b696152a9c3e0aa65e5824": tls: failed to verify certificate: x509: certificate signed by unknown authority

#解决办法1.指定 -k 参数跳过证书校验。
ctr -n k8s.io i push --user k8s_user1:k8s_Uus1 -k repo.k8s.local/google_containers/busybox:9.9 

# 解决办法2.指定CA证书、Harbor 相关证书文件路径。
#将harbor的ca.crt 复制到当前节点
scp /root/src/harbor/ca.crt [email protected]:.
ctr -n k8s.io i push --user k8s_user1:k8s_Uus1 --tlscacert ca.crt repo.k8s.local/google_containers/busybox:9.9

准备测试创建pod文件
vi test-harbor.yaml

# test-harbor.yaml
apiVersion: v1
kind: Pod
metadata:
  name: harbor-registry-test
spec:
  containers:
  - name: test
    image: repo.k8s.local/google_containers/busybox:9.9 
    args:
    - sleep
    - "3600"
  imagePullSecrets:
  - name: harbor-auth

创建pod
kubectl apply -f test-harbor.yaml

pod/harbor-registry-test created

查看
kubectl describe pod harbor-registry-test

Name:             harbor-registry-test
Namespace:        default
Priority:         0
Service Account:  default
Node:             node01.k8s.local/192.168.244.5
Start Time:       Mon, 16 Oct 2023 17:32:15 +0800
Labels:           
Annotations:      
Status:           Running
IP:               10.244.1.4
IPs:
  IP:  10.244.1.4
Containers:
  test:
    Container ID:  containerd://5fa473b370c56d71f0466e14960a974ead4559106495a79eb9b2a21a7ecee52f
    Image:         repo.k8s.local/google_containers/busybox:9.9
    Image ID:      repo.k8s.local/google_containers/busybox@sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee
    Port:

kubectl create secret docker-registry harbor-auth –docker-server=https://repo.k8s.local –docker-username=k8s_pull –docker-password=k8s_Pul1 –[email protected] -n default

k8s节点配置

containerd 配置私有仓库
Containerd 目前没有直接配置镜像加速的功能，但 containerd 中可以修改 docker.io 对应的 endpoint，所以可以通过修改 endpoint 来实现镜像加速下载。因为 endpoint 是轮询访问，所以可以给 docker.io 配置多个仓库地址来实现 加速地址+默认仓库地址

注意：这个配置文件是给crictl和kubelet使用，ctr是不可以用这个配置文件的，ctr 不使用 CRI，因此它不读取 plugins."io.containerd.grpc.v1.cri"配置。
直接修改：/etc/containerd/config.toml配置文件，这种方式在较新版本的contaienrd中已经被废弃，将来肯定会被移除，
两者取一
https://github.com/containerd/cri/blob/master/docs/registry.md
[plugins."io.containerd.grpc.v1.cri".registry.configs."repo.k8s.local".tls]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
和 config_path 会冲突

方式一 registry.mirrors
vi /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."repo.k8s.local".tls]
        insecure_skip_verify = true
      [plugins."io.containerd.grpc.v1.cri".registry.configs."repo.k8s.local".auth]
        username = "k8s_pull"
        password = "k8s_Pul1"

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."repo.k8s.local"]
        endpoint = ["http://repo.k8s.local:5100"]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry.cn-hangzhou.aliyuncs.com"]

保存并关闭配置文件。

方式二 config_path
vi /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"
      [plugins."io.containerd.grpc.v1.cri".registry.configs."repo.k8s.local".auth]
        username = "k8s_pull"
        password = "k8s_Pul1"

#注意创建的目录即为configs后面填写的内容
mkdir -p /etc/containerd/certs.d/docker.io/

cat > /etc/containerd/certs.d/docker.io/hosts.toml <
http/https两者取一配制

mkdir -p /etc/containerd/certs.d/repo.k8s.local

#harbor私仓使用http方式
cat > /etc/containerd/certs.d/repo.k8s.local/hosts.toml <
#harbor私仓使用http 非常规端口方式
cat > /etc/containerd/certs.d/repo.k8s.local:5100/hosts.toml <

从harbor复制证书到节点
scp [email protected]:/etc/docker/certs.d/repo.k8s.local/* /etc/containerd/certs.d/repo.k8s.local/

#harbor私仓使用https方式 带证书
cat > /etc/containerd/certs.d/repo.k8s.local/hosts.toml <
#harbor私仓使用https方式
cat > /etc/containerd/certs.d/repo.k8s.local/hosts.toml <

重新启动containerd服务以使更改生效。可以使用以下命令之一：
使用systemd：

systemctl daemon-reload
systemctl restart containerd
systemctl status containerd

使用Docker Compose（如果您是使用Docker Compose运行的）：
docker-compose restart

现在，containerd将使用您配置的私有Harbor源作为容器镜像的默认来源。您可以验证配置是否生效，通过拉取和运行一个位于Harbor Registry上的镜像来测试。

在harbor推送证书到节点
从harbor复制证书到节点
scp [email protected]:/etc/docker/certs.d/repo.k8s.local/* /etc/containerd/certs.d/repo.k8s.local/

节点上测试
ctr -n k8s.io i pull --user k8s_user1:k8s_Uus1 --tlscacert /etc/containerd/certs.d/repo.k8s.local/ca.crt repo.k8s.local/google_containers/busybox:9.9
ctr -n k8s.io images ls

ctr -n k8s.io i pull --user k8s_user1:k8s_Uus1 -k repo.k8s.local/google_containers/busybox:9.9

https 私库

ctr -n k8s.io i push --user k8s_user1:k8s_Uus1 -k repo.k8s.local/google_containers/busybox:9.9

推送到http

ctr -n k8s.io i push --user k8s_user1:k8s_Uus1 --plain-http repo.k8s.local/google_containers/busybox:9.9

ctr image pull --platform linux/amd64 docker.io/library/nginx:1.18.0

jq安装

jq安装
https://pkgs.org/centos-7/epel-x86_64/jq-1.5-1.el7.x86_64.rpm.html

wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum install jq
  Installing : oniguruma-6.8.2-2.el7.x86_64                                                                                    1/2 
  Installing : jq-1.6-2.el7.x86_64                                                                                             2/2

卸载

卸载harbor步骤：（针对脚本一键安装）

docker-compose down   #停止 docker-compose 
rm -rf /data_harbor *    #删除挂载的harbor持久化数据的目录内容
rm -rf /opt/harbor   #删除harbor目录
docker-compose up -d   #重启

配置Harbor的高可用（双组复制）

新建用户
admin_sync/sync_9OK

新建目标（仓库管理-新建）：
目标名：s2
url：https://10.100.5.6
用户名：admin
密码：Harbor12345
远程验证：x
连接测试：√

新建规则：
登陆S1的Harbor管理页面，进入kubernetes项目
点击“复制”，新建规则
名称：cp_to_s2
目标：https://10.100.5.6
触发模式：即刻

在S2的Harbor做相同配置

harbor-连接其他仓库报错
pkg/reg/adapter/native/adapter.go:126]: failed to ping registry 

docker exec -it -u root docker ps |awk '/core/{print $1}' /bin/bash
echo "192.168.244.6    repo.k8s.local">>/etc/hosts
curl http://repo.k8s.local:5100

cat > /etc/hosts <

两个harbor配制相同主机会同步失败，先将备机配成ip
vi /usr/local/harbor/harbor.yml

hostname: 10.100.5.6

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 5100
systemctl stop harbor
cd /usr/local/harbor/
./prepare 
systemctl start harbor

harbor从机需要重新分配用户及项目的权限

各项目添加 k8s_pull 为访客
各项目添加 k8s_user1 为开发者

Posted in 安装k8s/kubernetes.

Tagged with , , .

By C1G – 2024/05/17

No Responses (yet)

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.



Some HTML is OK

or, reply to this post via trackback.

« »