您好,欢迎来到站长目录(28sn.com)!


Harbor-hlem镜像库重新部署后PV数据恢复

来源:网络整理 浏览:170次 时间:2021-03-03

起因

开发反馈habor镜像库登陆不了,初步查看是证书过期了。

解决方案
之前Harbor-helm部署镜像库文档可以回顾链接https://minminmsn.com/middleware/698/
1.首先新建新证书的secret

[root@elasticsearch01 harbor-helm]#  kubectl  create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt 

2.然后修改harbor-helm的value.yaml,把secretName替换下

[root@elasticsearch01 harbor-helm]# head -n 20 values.yamlexpose:  # Set the way how to expose the service. Set the type as "ingress",   # "clusterIP" or "nodePort" and fill the information in the corresponding   # section  type: ingress  tls:    # Enable the tls or not. Note: if the type is "ingress" and the tls     # is disabled, the port must be included in the command when pull/push    # images. Refer to https://github.com/goharbor/harbor/issues/5291     # for the detail.    enabled: true    # Fill the name of secret if you want to use your own TLS certificate    # and private key. The secret must contain keys named tls.crt and     # tls.key that contain the certificate and private key to use for TLS    # The certificate and private key will be generated automatically if     # it is not set    secretName: "ingress-secret2021"    # By default, the Notary service will use the same cert and key as    # described above. Fill the name of secret if you want to use a     # separated one. Only needed when the type is "ingress".

3.最后使用helm upgrade更新版本

[root@elasticsearch01 harbor-helm]#  helm upgrade  minminmsn . -f values.yaml

到这个时候应该能解决需求,可是事与愿违,不知道哪儿除了问题,这时登陆Harbor证书问题是解决了,但是项目及库访问不了提示内部错误,看Pod的运行状态也都是Running。
最后打算使用helm先delete掉再install,但是这样创建的harbor看起来一切正常,实际上是个初始化环境,是自动生成的新PV并没有原来的数据。此时发现原来的PV还在,下面就开始找PV恢复的方案。

调整PV状态

1.查询此时PV与PVC状态

[root@elasticsearch01 harbor-helm]# kubectl get pvNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                            STORAGECLASS   REASON   AGE                    9hpvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Released   default/minminmsn-harbor-chartmuseum                rbd                     417dpvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/minminmsn-harbor-jobservice                 rbd                     417dpvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Released   default/minminmsn-harbor-registry                   rbd                     417dpvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/database-data-minminmsn-harbor-database-0   rbd                     417dpvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Released   default/data-minminmsn-harbor-redis-0               rbd                     417d[root@elasticsearch01 harbor-helm]# kubectl get pvcNAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGEdata-minminmsn-harbor-redis-0               Bound    pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9hdatabase-data-minminmsn-harbor-database-0   Bound    pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9hminminmsn-harbor-chartmuseum                Bound    pvc-6c903857-c5f0-11ea-9386-52540089b2b6   50Gi       RWO            rbd            9hminminmsn-harbor-jobservice                 Bound    pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6   20Gi       RWO            rbd            9hminminmsn-harbor-registry                   Bound    pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6   500Gi      RWO            rbd            9h

2.修改PV状态
先把PV的状态由Released改变成
备注:默认创建的PV的回收策略是Delete就是用完就删除,之前特意把RECLAIM POLICY改为了Retain,在线修改PV回收策略可以参考文档https://minminmsn.com/cloud/1091/。否则这里Helm Delete后就会自动删除PV,就没有后来这篇PV数据恢复操作了。
在线编辑PV,需要把其中claimRef这段删除,这样状态就可以变成Available了。

  claimRef:    apiVersion: v1    kind: PersistentVolumeClaim    name: minminmsn-harbor-chartmuseum    namespace: default    resourceVersion: "91736092"    uid: b31ec8ca-c649-11ea-9386-52540089b2b6  persistentVolumeReclaimPolicy: Retain

具体如下修改

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 # Please edit the object below. Lines beginning with a '#' will be ignored,# and an empty file will abort the edit. If an error occurs while saving this file will be# reopened with the relevant failures.#apiVersion: v1kind: PersistentVolumemetadata:  annotations:    pv.kubernetes.io/bound-by-controller: "yes"    pv.kubernetes.io/provisioned-by: ceph.com/rbd    rbdProvisionerIdentity: ceph.com/rbd  creationTimestamp: "2019-05-24T06:33:55Z"  finalizers:  - kubernetes.io/pv-protection  name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6  resourceVersion: "91736100"  selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6  uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6spec:  accessModes:  - ReadWriteOnce  capacity:    storage: 50Gi  claimRef:    apiVersion: v1    kind: PersistentVolumeClaim    name: minminmsn-harbor-chartmuseum    namespace: default    resourceVersion: "91736092"    uid: b31ec8ca-c649-11ea-9386-52540089b2b6  persistentVolumeReclaimPolicy: Retain  rbd:    image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905    keyring: /etc/ceph/keyring    monitors:    - 10.0.4.8:6789    pool: rbd-k8s    secretRef:      name: ceph-secret      namespace: default    user: admin  storageClassName: rbd  volumeMode: Filesystemstatus:  phase: Released

3.其他四个PV同样操作

[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6

4.查看效果
现在看PV的STATUS已经变成了Available,然后CLAIM也变空了,这样就可以在后面绑定使用了

[root@elasticsearch01 harbor-helm]# kubectl get pvNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGEpvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Available                               rbd                     417dpvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417dpvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Available                               rbd                     417dpvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417dpvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Available                               rbd                     417d

创建PVC

1.先设置好PVC及PV对应关系

[root@elasticsearch01 yaml]# cat minminmsn.pvc apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: minminmsn-harbor-registryspec:  accessModes:    - ReadWriteOnce  storageClassName: "rbd"  resources:    requests:      storage: 2000Gi  volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6"---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: minminmsn-harbor-jobservicespec:  accessModes:    - ReadWriteOnce  storageClassName: "rbd"  resources:    requests:      storage: 20Gi  volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6"---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: minminmsn-harbor-chartmuseumspec:  accessModes:    - ReadWriteOnce  storageClassName: "rbd"  resources:    requests:      storage: 50Gi  volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6"---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: database-data-minminmsn-harbor-database-0spec:  accessModes:    - ReadWriteOnce  storageClassName: "rbd"  resources:    requests:      storage: 20Gi  volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6"---apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: data-minminmsn-harbor-redis-0spec:  accessModes:    - ReadWriteOnce  storageClassName: "rbd"  resources:    requests:      storage: 20Gi  volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"

2.创建PVC

[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc persistentvolumeclaim/minminmsn-harbor-registry createdpersistentvolumeclaim/minminmsn-harbor-jobservice createdpersistentvolumeclaim/minminmsn-harbor-chartmuseum createdpersistentvolumeclaim/database-data-minminmsn-harbor-database-0 createdpersistentvolumeclaim/data-minminmsn-harbor-redis-0 created

3.检查PV与PVC

[root@elasticsearch01 yaml]# kubectl get pvNAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                            STORAGECLASS   REASON   AGEpvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   50Gi       RWO            Retain           Bound    default/minminmsn-harbor-chartmuseum                rbd                     417dpvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/minminmsn-harbor-jobservice                 rbd                     417dpvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            Retain           Bound    default/minminmsn-harbor-registry                   rbd                     417dpvc-e7d38097-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/database-data-minminmsn-harbor-database-0   rbd                     417dpvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   20Gi       RWO            Retain           Bound    default/data-minminmsn-harbor-redis-0               rbd                     417d[root@elasticsearch01 yaml]# kubectl get pvcNAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGEceph-rbd-pv-claim                        Bound     ceph-rbd-pv                                20Gi       RWO                           540ddata-minminmsn-harbor-redis-0               Pending   pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12sdatabase-data-minminmsn-harbor-database-0   Pending   pvc-e7d38097-7ded-11e9-a09d-52540089b2b6   0                         rbd            12sminminmsn-harbor-chartmuseum                Pending   pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6   0                         rbd            12sminminmsn-harbor-jobservice                 Pending   pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6   0                         rbd            12sminminmsn-harbor-registry                   Bound     pvc-e7985b55-7ded-11e9-a09d-52540089b2b6   2000Gi     RWO            rbd            12s[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registryName:          minminmsn-harbor-registryNamespace:     defaultStorageClass:  rbdStatus:        BoundVolume:        pvc-e7985b55-7ded-11e9-a09d-52540089b2b6Labels:        <none>Annotations:   kubectl.kubernetes.io/last-applied-configuration:                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...               pv.kubernetes.io/bind-completed: yesFinalizers:    [kubernetes.io/pvc-protection]Capacity:      2000GiAccess Modes:  RWOVolumeMode:    FilesystemEvents:        <none>Mounted By:    <none>

使用Hlem重新部署Harbor镜像库

1.部署前先删除版本

[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsnhelm delete --purge minminmsnrelease "minminmsn" deleted

2.修改Harbor-helm的values.yaml中PVC相关值
注意existingClaim: ""由空值改成上面生成的PVC名字,注意对应关系,其他不变,具体变更如下

persistence:  enabled: true  # Setting it to "keep" to avoid removing PVCs during a helm delete   # operation. Leaving it empty will delete PVCs after the chart deleted  resourcePolicy: "keep"  persistentVolumeClaim:    registry:      # Use the existing PVC which must be created manually before bound      existingClaim: "minminmsn-harbor-registry"      # Specify the "storageClass" used to provision the volume. Or the default      # StorageClass will be used(the default).      # Set it to "-" to disable dynamic provisioning      storageClass: "rbd"      subPath: ""      accessMode: ReadWriteOnce      size: 2000Gi    chartmuseum:      existingClaim: "minminmsn-harbor-chartmuseum"      storageClass: "rbd"      subPath: ""      accessMode: ReadWriteOnce      size: 50Gi    jobservice:      existingClaim: "minminmsn-harbor-jobservice"      storageClass: "rbd"      subPath: ""      accessMode: ReadWriteOnce      size: 20Gi    # If external database is used, the following settings for database will     # be ignored    database:      existingClaim: "database-data-minminmsn-harbor-database-0"      storageClass: "rbd"      subPath: ""      accessMode: ReadWriteOnce      size: 20Gi    # If external Redis is used, the following settings for Redis will     # be ignored    redis:      existingClaim: "data-minminmsn-harbor-redis-0"      storageClass: "rbd"      subPath: ""      accessMode: ReadWriteOnce      size: 20Gi

3.重新部署

[root@elasticsearch01 harbor-helm]# helm  install . --name minminmsnNAME:   minminmsnLAST DEPLOYED: Wed Jul 15 11:18:13 2020NAMESPACE: defaultSTATUS: DEPLOYEDRESOURCES:==> v1/ServiceNAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)            AGEminminmsn-harbor-adminserver    ClusterIP  10.254.58.23    <none>       80/TCP             1sminminmsn-harbor-chartmuseum    ClusterIP  10.254.154.44   <none>       80/TCP             1sminminmsn-harbor-clair          ClusterIP  10.254.25.107   <none>       6060/TCP           1sminminmsn-harbor-core           ClusterIP  10.254.56.153   <none>       80/TCP             1sminminmsn-harbor-database       ClusterIP  10.254.65.18    <none>       5432/TCP           1sminminmsn-harbor-jobservice     ClusterIP  10.254.81.97    <none>       80/TCP             1sminminmsn-harbor-notary-server  ClusterIP  10.254.99.90    <none>       4443/TCP           1sminminmsn-harbor-notary-signer  ClusterIP  10.254.175.105  <none>       7899/TCP           1sminminmsn-harbor-portal         ClusterIP  10.254.242.113  <none>       80/TCP             1sminminmsn-harbor-redis          ClusterIP  10.254.127.40   <none>       6379/TCP           1sminminmsn-harbor-registry       ClusterIP  10.254.158.222  <none>       5000/TCP,8080/TCP  1s==> v1/DeploymentNAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGEminminmsn-harbor-adminserver    1        1        1           0          1sminminmsn-harbor-chartmuseum    1        1        1           0          1sminminmsn-harbor-clair          1        0        0           0          1sminminmsn-harbor-core           1        0        0           0          1sminminmsn-harbor-jobservice     1        0        0           0          1sminminmsn-harbor-notary-server  1        0        0           0          1sminminmsn-harbor-notary-signer  1        0        0           0          1sminminmsn-harbor-portal         1        0        0           0          1sminminmsn-harbor-registry       1        0        0           0          1s==> v1/StatefulSetNAME                    DESIRED  CURRENT  AGEminminmsn-harbor-database  1        1        1sminminmsn-harbor-redis     1        1        1s==> v1beta1/IngressNAME                   HOSTS                                                    ADDRESS  PORTS  AGEminminmsn-harbor-ingress  core-harbor.minminmsn.com,notary-harbor.minminmsn.com  80, 443  1s==> v1/Pod(related)NAME                                          READY  STATUS             RESTARTS  AGEminminmsn-harbor-adminserver-b5d58db8c-wmrbd     0/1    ContainerCreating  0         1sminminmsn-harbor-chartmuseum-7c6b9d4977-94rhb    0/1    Pending            0         1sminminmsn-harbor-clair-54465ff7dd-d7bxx          0/1    Pending            0         1sminminmsn-harbor-core-587cc5d9b5-2xxl9           0/1    Pending            0         1sminminmsn-harbor-jobservice-764bb697d-wsxqx      0/1    Pending            0         1sminminmsn-harbor-notary-server-77fbb84fcc-2bw7c  0/1    Pending            0         1sminminmsn-harbor-notary-signer-8466d68f5b-klv76  0/1    Pending            0         1sminminmsn-harbor-database-0                      0/1    Pending            0         1sminminmsn-harbor-redis-0                         0/1    Pending            0         1s==> v1/SecretNAME                       TYPE    DATA  AGEminminmsn-harbor-adminserver  Opaque  4     1sminminmsn-harbor-chartmuseum  Opaque  1     1sminminmsn-harbor-core         Opaque  4     1sminminmsn-harbor-database     Opaque  1     1sminminmsn-harbor-jobservice   Opaque  1     1sminminmsn-harbor-registry     Opaque  1     1s==> v1/ConfigMapNAME                         DATA  AGEminminmsn-harbor-adminserver    39    1sminminmsn-harbor-chartmuseum    24    1sminminmsn-harbor-clair          1     1sminminmsn-harbor-core           1     1sminminmsn-harbor-jobservice     1     1sminminmsn-harbor-notary-server  5     1sminminmsn-harbor-registry       2     1sNOTES:Please wait for several minutes for Harbor deployment to complete.Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. For more details, please visit https://github.com/goharbor/harbor.3.查看新生成Pods的信息[root@elasticsearch01 harbor-helm]# kubectl get podsNAME                                           READY   STATUS              RESTARTS   AGEminminmsn-harbor-adminserver-b5d58db8c-wmrbd      0/1     ContainerCreating   0          9sminminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     0/1     ContainerCreating   0          9sminminmsn-harbor-clair-54465ff7dd-d7bxx           0/1     Running             0          9sminminmsn-harbor-core-587cc5d9b5-2xxl9            0/1     Running             0          9sminminmsn-harbor-database-0                       0/1     Init:0/1            0          9sminminmsn-harbor-jobservice-764bb697d-wsxqx       0/1     ContainerCreating   0          9sminminmsn-harbor-notary-server-77fbb84fcc-2bw7c   0/1     ContainerCreating   0          9sminminmsn-harbor-notary-signer-8466d68f5b-klv76   0/1     ContainerCreating   0          9sminminmsn-harbor-portal-64cf8b9b69-xm8nl          0/1     ContainerCreating   0          8sminminmsn-harbor-redis-0                          0/1     ContainerCreating   0          9sminminmsn-harbor-registry-755746c5bb-q8m55        0/2     ContainerCreating   0          8s

再等2分钟查看就上恢复了

[root@elasticsearch01 harbor-helm]# kubectl get podsNAME                                           READY   STATUS    RESTARTS   AGEjenkins-0                                      1/1     Running   0          62drbd-provisioner-67b4857bcd-rjwlg               1/1     Running   0          61dminminmsn-harbor-adminserver-b5d58db8c-wmrbd      1/1     Running   1          2m33sminminmsn-harbor-chartmuseum-7c6b9d4977-94rhb     1/1     Running   0          2m33sminminmsn-harbor-clair-54465ff7dd-d7bxx           1/1     Running   1          2m33sminminmsn-harbor-core-587cc5d9b5-2xxl9            1/1     Running   1          2m33sminminmsn-harbor-database-0                       1/1     Running   0          2m33sminminmsn-harbor-jobservice-764bb697d-wsxqx       1/1     Running   0          2m33sminminmsn-harbor-notary-server-77fbb84fcc-2bw7c   1/1     Running   0          2m33sminminmsn-harbor-notary-signer-8466d68f5b-klv76   1/1     Running   0          2m33sminminmsn-harbor-portal-64cf8b9b69-xm8nl          1/1     Running   0          2m32sminminmsn-harbor-redis-0                          1/1     Running   0          2m33sminminmsn-harbor-registry-755746c5bb-q8m55        2/2     Running   0          2m32s

4.Harbor控制验证
证书更新了项目也恢复了
https://core-harbor.minminmsn.com/harbor/projects

Harbor-hlem镜像库重新部署后PV数据恢复

推荐站点

  • 我爱发烧音乐我爱发烧音乐

    我爱发烧音乐囊括了从流行音乐到古典音乐多个类型的音乐作品,专栏推荐最新的音乐,提供音乐排名榜单!可供免费线上收听音乐,歌曲流畅,音效极佳! 网站提供的钢琴以及二胡专栏,可供收听者,陶冶情操,改善心情,是难得的轻音乐典藏!

    www.520fs.com
  • 世纪音乐网世纪音乐网

    世纪音乐网是专业的在线音乐试听MP3下载网站。歌曲总计30余万首,收录了网上最新歌曲和流行音乐,DJ舞曲,非主流音乐,经典老歌,劲舞团歌曲,搞笑歌曲,儿童歌曲,英文歌曲等。是您上网听歌的最佳网站。

    www.ssjj.com
  • 杭州网杭州网

      杭州网是杭州地区唯一的新闻门户网站,由中共杭州市委宣传部、杭州日报报业集团和杭州广播电视集团共同组建的杭州网络传媒有限公司运营。

    www.hangzhou.com.cn
  • 深圳在线深圳在线

      深圳在线 www.szol.net是深圳本地最大、最早的地方生活资讯网站之一,网站名“深圳在线www.szol.net”由南方报业传媒集团编辑委员会总编辑、南方日报社总编辑、南方都市报总编辑、南方书画院名誉院长王春芙亲笔题名,深圳在线www.szol.net团队与深圳热线www.szonline.net、奥一网www.oeeee.com都源于全国最早成立于1996年的知名网络公司——深圳万用网。

    www.szol.net
  • 今题网今题网

     今题网- 中国领先的社区服务网,提供社区服务, 在线交友和商家推广服务,于2004年创建上线,公司现有员工超过百名。今题网自成立以来,凭借其独特的定位和丰富的社区交友功能, 凭借其团队超强的搜索引擎优化技术吸引超过千万的用户成为今题网的注册会员。

    www.jinti.com

鄂公网安备 42062502000001号