Question

0

160

Visualizações

ceph rbd with docker volumes

I just initialized a ceph instances within two differents servers

cluster 241b5d19-15f5-48be-b98c-285239d70038
 health HEALTH_WARN
        64 pgs degraded
        64 pgs stuck degraded
        64 pgs stuck unclean
        64 pgs stuck undersized
        64 pgs undersized
 monmap e3: 2 mons at {serv1=10.231.69.9:6789/0,serv2=10.231.69.34:6789/0}
        election epoch 6, quorum 0,1 serv1,serv2
    mgr no daemons active
 osdmap e10: 2 osds: 2 up, 2 in
        flags sortbitwise,require_jewel_osds,require_kraken_osds
  pgmap v22: 64 pgs, 1 pools, 0 bytes data, 0 objects
        68292 kB used, 1861 GB / 1861 GB avail
              64 active+undersized+degraded

with only mon and osd (I do not setup mds, rgw or CephFS).

I would use rbd to create a persisted shared storage for containers volumes, but I'm really confused on how to plug my osd inside docker.

I saw some rbd docker plugins exists:

But none seems to be compatible with latest docker version or at least >= 1.13.

Thus I'm asking myself how can I achieve what I want, some solutions come on my mind but I'm really not sure which is best (or even if it's possible).

Use CephFS + standard docker filesystem mounting volume
Use rexray (flocker is no more maintained)
Install Ceph Object Gateway S3 and use existing docker S3 plugins
Other?

But 1. solution seems to be inelegant and will be harder to manage in larger environment (more than 2 servers).

Whereas 2. solution seems to be a great starting point but anyone have feedback?

over 4 years ago · Santiago Trujillo

2 Respostas

Responde à pergunta

0

Responde à pergunta

Encontrar trabalhos remotos

Answer 1 · 2022-06-13T19:26:02.821Z

You can extend Docker Engine with a v2 plugin for RBD volumes.

Docker volume plugin for RBD works with Docker Engine v2 plugin system. (we use it in with Swarm clusters)

Here an excerpt from the Readme:

install the plugin:

docker plugin install wetopi/rbd \
  --alias=wetopi/rbd \
  LOG_LEVEL=1 \
  RBD_CONF_POOL="ssd" \
  RBD_CONF_CLUSTER=ceph \
  RBD_CONF_KEYRING_USER=client.admin

create a volume:

docker volume create -d wetopi/rbd -o size=206 my_rbd_volume

docker volume ls
DRIVER              VOLUME NAME
local               069d59c79366294d07b9102dde97807aeaae49dc26bb9b79dd5b983f7041d069
local               11db1fa5ba70752101be90a80ee48f0282a22a3c8020c1042219ed1ed5cb0557
local               2d1f2a8fac147b7e7a6b95ca227eba2ff859325210c7280ccb73fd5beda6e67a
wetopi/rbd          my_rbd_volume

run a container with rbd volume:

docker run -it -v my_rbd_volume:/data --volume-driver=wetopi/rbd busybox sh

Answer 2 · 2022-06-13T19:26:02.828Z

From the output of ceph -s you have provided it seems that you created a ceph cluster with 2 nodes with one osd each. By default ceph is using a size (nr of replications) of 3. This is the recommended value. You could change the size to 2 and the minimum size to 1, but it is considered insane to use this setup for anything else but as a playground for ceph unless the data is of no value to you.

Ceph should at a bare minimum be run on at least 3 nodes in identical configuration. Identical includes: CPU, RAM, OSD count (one OSD per CPU Core). For medium traffic it is ok to put the mons on those three nodes as well. 10GBit node interconnection is highly recommended.

0

160