ROOK 06：使用对象存储

/ Rook-Ceph

文章大纲

对象存储通过暴露 S3 API 以便应用放置和获取存储集群中的数据。

配置对象存储

Rook 能够在 Kubernetes 中部署对象存储或者连接到外部的 RGW 服务。
最常见的是在本地部署一个 RADOS 网关，默认部署 Rook 时不会创建 rgw 服务。

创建本地对象网关

部署本地 rgw 可以借助 CephObjectStore CRD，示例资源定义：

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  preservePoolsOnDelete: true
  gateway:
    sslCertificateRef:
    port: 80
    # securePort: 443
    instances: 1

应用并检查：

[vagrant@master01 rook-ceph]$ kubectl apply -f object.yaml 
cephobjectstore.ceph.rook.io/my-store created
[vagrant@master01 rook-ceph]$ kubectl get pod -n rook-ceph -l app=rook-ceph-rgw
NAME                                        READY   STATUS    RESTARTS   AGE
rook-ceph-rgw-my-store-a-79ffd75dc8-fp95c   2/2     Running   0          87s

在 toolbox 中，验证 rgw 已部署：

[vagrant@master01 rook-ceph]$ kubectl exec -it rook-ceph-tools-66b77b8df5-x97q4 -n rook-ceph -- /bin/ceph -s
  cluster:
    id:     f8bdb7b9-12c8-4814-b4a2-6122366ddd1a
    health: HEALTH_WARN
            too many PGs per OSD (254 > max 250)

  services:
    mon: 3 daemons, quorum a,b,c (age 13m)
    mgr: a(active, since 12m), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 13m), 3 in (since 4w)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   15 pools, 265 pgs
    objects: 279 objects, 565 KiB
    usage:   112 MiB used, 60 GiB / 60 GiB avail
    pgs:     265 active+clean

  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

验证相关的 Pool 已创建：

[vagrant@master01 rook-ceph]$ kubectl exec -it rook-ceph-tools-66b77b8df5-x97q4 -n rook-ceph -- /bin/ceph osd pool ls
.mgr
replicapool
ecpool
replicated-metadata-pool
ec-data-pool
myfs-metadata
myfs-replicated
my-store.rgw.control
my-store.rgw.meta
my-store.rgw.log
my-store.rgw.buckets.index
my-store.rgw.buckets.non-ec
my-store.rgw.otp
.rgw.root
my-store.rgw.buckets.data

创建 Bucket

Object Store 已经配置好了，下一步就是创建 bucket 让客户端能够读写对象。
bucket 的创建可以定义 storage class，定义了 storage class 后，就能够允许客户端创建 bucket。

storageclass-bucket.yaml 的示例定义：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-bucket
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: my-store
  objectStoreNamespace: rook-ceph

应用并检查：

vagrant@master01 rook-ceph]$ kubectl apply -f storageclass-bucket.yaml 
storageclass.storage.k8s.io/rook-ceph-bucket created
[vagrant@master01 rook-ceph]$ kubectl get sc
NAME                 PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block      rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   20h
rook-ceph-block-ec   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   20h
rook-ceph-bucket     rook-ceph.ceph.rook.io/bucket   Delete          Immediate           false                  3s
rook-cephfs          rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           false                  13h

基于创建出来的 storageclass，客户端可以请求创建 Object Bucket Claim(OBC)，这种资源类似于 PVC。

创建 OBC 的示例定义：

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-bucket
  namespace: obc-demo
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-bucket

应用并检查：

[vagrant@master01 rook-ceph]$ kubectl apply -f object-bucket-claim.yaml 
objectbucketclaim.objectbucket.io/ceph-bucket created
[vagrant@master01 rook-ceph]$ kubectl get obc -n obc-demo
NAME          AGE
ceph-bucket   9s

[vagrant@master01 rook-ceph]$ kubectl describe obc/ceph-bucket -n obc-demo
Name:         ceph-bucket
Namespace:    obc-demo
Labels:       bucket-provisioner=rook-ceph.ceph.rook.io-bucket
Annotations:  <none>
API Version:  objectbucket.io/v1alpha1
Kind:         ObjectBucketClaim
Metadata:
  Creation Timestamp:  2024-03-04T01:48:17Z
  Finalizers:
    objectbucket.io/finalizer
  Generation:        4
  Resource Version:  181929
  UID:               c47194f4-8ab1-48c1-b436-2ac6c8313eb4
Spec:
  Bucket Name:           ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9
  Generate Bucket Name:  ceph-bkt
  Object Bucket Name:    obc-obc-demo-ceph-bucket
  Storage Class Name:    rook-ceph-bucket
Status:
  Phase:  Bound
Events:   <none>

使用对象存储

应用需要使用对象存储需要以下信息：

  • RGW 的端点
  • RGW 的端口
  • ACCESS_KEY
  • SECRET_KEY
  • Bucket 名称

其中 Bucket 名称，可以查看创建的出来的 OBC 资源，前面创建的 ceph-bucket obc 所对应的 Bucket 是 ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9 。

随着 OBC 的创建，还会生成相应的 configmap 和 secret：

[vagrant@master01 rook-ceph]$ kubectl get secret -n obc-demo
NAME          TYPE     DATA   AGE
ceph-bucket   Opaque   2      2m26s
[vagrant@master01 rook-ceph]$ kubectl get cm -n obc-demo
NAME               DATA   AGE
ceph-bucket        5      2m31s
kube-root-ca.crt   1      2m35s

其中 secret 里面存储是 ACCESS_KEY 和 SECRET_KEY：

[vagrant@master01 rook-ceph]$ kubectl get secret -o yaml ceph-bucket -n obc-demo
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: UURVUlRDTEQxU0YxTE1MVzFWNUs=
  AWS_SECRET_ACCESS_KEY: MngwNGtQODI4ZUs2bXk3djFnSUJRelA5b0ZadHRSVG5mOWduazBRcQ==
kind: Secret
metadata:
  creationTimestamp: "2024-03-04T01:48:19Z"
  finalizers:
  - objectbucket.io/finalizer
  labels:
    bucket-provisioner: rook-ceph.ceph.rook.io-bucket
  name: ceph-bucket
  namespace: obc-demo
  ownerReferences:
  - apiVersion: objectbucket.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ObjectBucketClaim
    name: ceph-bucket
    uid: c47194f4-8ab1-48c1-b436-2ac6c8313eb4
  resourceVersion: "181924"
  uid: 295cd046-9ab3-4126-8112-1b2d09f4beef
type: Opaque

configmap 中存储的端点、端口等信息：

[vagrant@master01 rook-ceph]$ kubectl get cm -o yaml ceph-bucket -n obc-demo
apiVersion: v1
data:
  BUCKET_HOST: rook-ceph-rgw-my-store.rook-ceph.svc
  BUCKET_NAME: ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9
  BUCKET_PORT: "80"
  BUCKET_REGION: ""
  BUCKET_SUBREGION: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2024-03-04T01:48:19Z"
  finalizers:
  - objectbucket.io/finalizer
  labels:
    bucket-provisioner: rook-ceph.ceph.rook.io-bucket
  name: ceph-bucket
  namespace: obc-demo
  ownerReferences:
  - apiVersion: objectbucket.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ObjectBucketClaim
    name: ceph-bucket
    uid: c47194f4-8ab1-48c1-b436-2ac6c8313eb4
  resourceVersion: "181925"
  uid: e6c1cb11-de23-4381-9b7d-0ffbc64d0518

所以应用需要访问对象存储，只需将 OBC 对应的 secret 和 configmap 注入到 Pod 中即可。

使用 s5cmd 访问对象存储

toolbox Pod 不提供 s5cmd，可以部署 toolbox-operator-image ，里面包含了该客户端工具。
资源定义文件在项目中的 deploy/examples/toolbox-operator-image.yaml 中：

[vagrant@master01 rook]$ kubectl apply -f deploy/examples/toolbox-operator-image.yaml 
deployment.apps/rook-ceph-tools-operator-image created

[vagrant@master01 rook]$ kubectl get pod -n rook-ceph | grep tools-operator
rook-ceph-tools-operator-image-5df88d9f87-tb7zt                  1/1     Running     0              38s

配置 s5cmd

在 toolbox 中配置对象存储，需要注意的是 Secret 中的信息是 base64 加密的，如果直接通过 get -o yaml 方式，需要解密：

[vagrant@master01 rook]$ kubectl exec -it -n rook-ceph rook-ceph-tools-operator-image-5df88d9f87-tb7zt -- /bin/bash
[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ mkdir ~/.aws
[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ cat > ~/.aws/credentials <<EOF
> [default]
> aws_access_key_id = QDURTCLD1SF1LMLW1V5K
> aws_secret_access_key = 2x04kP828eK6my7v1gIBQzP9oFZttRTnf9gnk0Qq
> EOF

验证凭据是否正确：

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc ls 
2024/03/04 01:48:19  s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9

PUT 和 GET 对象

上传一个新对象到对象存储中：

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt /]$ echo "Hello Rook!" > /tmp/rookObj

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc cp /tmp/rookObj s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9
cp /tmp/rookObj s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc ls s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9
2024/03/04 02:15:36                12  rookObj

从对象存储中下载对象：

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ s5cmd --endpoint-url http://rook-ceph-rgw-my-store.rook-ceph.svc cp s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj /tmp/download_obj
cp s3://ceph-bkt-09b00322-7504-4e94-9c1b-15abb1a4a6f9/rookObj /tmp/download_obj

[rook@rook-ceph-tools-operator-image-5df88d9f87-tb7zt ~]$ diff /tmp/rookObj /tmp/download_obj

相关文章

发表评论

您的邮箱地址不会被公开。 必填项已用 * 标注

此站点使用Akismet来减少垃圾评论。了解我们如何处理您的评论数据。

滚动至顶部