Cephadm-ansible이란?

Overview

Cephadm-ansible에 대해서 알아보려고 한다.

출처 : https://github.com/ceph/cephadm-ansible/releases

 

 

모든 리소스는 해당 Github 디렉토리에 업로드 되어있다.

https://github.com/somaz94/install-cephadm-script

GitHub - somaz94/install-cephadm-script: install-cephadm-script

 

Rook-Ceph 내용은 아래와 같다.

2024.02.20 - [Open Source Software] - Rook-Ceph란?

rook-ceph란?

 

 

---

 

Cephadm 이란?

Cephadm은 Ceph의 최신 배포 및 관리 도구로, Ceph Octopus 릴리즈부터 도입되었다.

Ceph 클러스터를 배포, 구성, 관리하고 확장하는 작업을 단순화하기 위해 설계되었다. 단일 명령으로 클러스터를 부트스트랩하고, 컨테이너 기술을 사용하여 Ceph 서비스를 배포한다.

Cephadm은 Ansible, Rook 또는 Salt와 같은 외부 구성 도구에 의존하지 않는다. 그러나 이러한 외부 구성 도구를 사용하면 cephadm 자체에서 수행되지 않는 작업을 자동화할 수 있다.

​

• https://github.com/ceph/cephadm-ansible
• https://rook.io/docs/rook/v1.10/Getting-Started/intro/
• https://github.com/ceph/ceph-salt

 

---

Cephadm-ansible 이란?

cephadm-ansible은 cephadm 에서 다루지 않는 워크플로를 단순화하기 위한 Ansible 플레이북 모음이다 .

다루는 워크플로는 다음과 같다.

• Preflight: 클러스터를 부트스트래핑하기 전 호스트의 초기 설정
• Client: 클라이언트 호스트 설정
• Purge: Ceph 클러스터 제거

 

---

 

K8S 설치

아래의 링크를 참고바란다. 주의할점은 Storage Node가 Memory 32G 이상은 되어야 한다.

2024.02.02 - [Container Orchestration/Kubernetes] - Kubernetes 클러스터 구축하기(kubespray 2024v.)

Kubernetes 클러스터 구축하기(kubespray 2024v.)

 

Master Node(Control Plane)	IP	CPU	Memory
test-server	10.77.101.47	16	32G

 

Worker Node	IP	CPU	Memory
test-server-agent	10.77.101.43	16	32G

 

Worker Node	IP	CPU	Memory
test-server-storage	10.77.101.48	16	32G

 

 

아래의 내용만 추가하면 된다.

## test_server_storage ##
resource "google_compute_address" "test_server_storage_ip" {
  name = var.test_server_storage_ip
}

resource "google_compute_instance" "test_server_storage" {
  name                      = var.test_server_storage
  machine_type              = "n2-standard-8"
  labels                    = local.default_labels
  zone                      = "${var.region}-a"
  allow_stopping_for_update = true

  tags = [var.kubernetes_client]

  boot_disk {
    initialize_params {
      image = "ubuntu-os-cloud/ubuntu-2004-lts"
      size  = 50
    }
  }

  attached_disk {
    source      = google_compute_disk.additional_test_storage_1_pd_balanced.self_link
    device_name = google_compute_disk.additional_test_storage_1_pd_balanced.name
  }

  attached_disk {
    source      = google_compute_disk.additional_test_storage_2_pd_balanced.self_link
    device_name = google_compute_disk.additional_test_storage_2_pd_balanced.name
  }

  attached_disk {
    source      = google_compute_disk.additional_test_storage_3_pd_balanced.self_link
    device_name = google_compute_disk.additional_test_storage_3_pd_balanced.name
  }

  metadata = {
    ssh-keys = "somaz:${file("~/.ssh/id_rsa_somaz94.pub")}"
  }

  network_interface {
    network    = var.shared_vpc
    subnetwork = "${var.subnet_share}-mgmt-a"

    access_config {
      ## Include this section to give the VM an external ip ##
      nat_ip = google_compute_address.test_server_storage_ip.address
    }
  }

  depends_on = [
    google_compute_address.test_server_storage_ip,
    google_compute_disk.additional_test_storage_1_pd_balanced,
    google_compute_disk.additional_test_storage_2_pd_balanced,
    google_compute_disk.additional_test_storage_3_pd_balanced
  ]
}

resource "google_compute_disk" "additional_test_storage_1_pd_balanced" {
  name = "test-storage-disk-1"
  type = "pd-balanced"
  zone = "${var.region}-a"
  size = 50
}

resource "google_compute_disk" "additional_test_storage_2_pd_balanced" {
  name = "test-storage-disk-2"
  type = "pd-balanced"
  zone = "${var.region}-a"
  size = 50
}

resource "google_compute_disk" "additional_test_storage_3_pd_balanced" {
  name = "test-storage-disk-3"
  type = "pd-balanced"
  zone = "${var.region}-a"
  size = 50
}

 

설치는 블로그 참고하여 진행하면 된다.

k get nodes
NAME                STATUS   ROLES           AGE     VERSION
test-server         Ready    control-plane   6m27s   v1.29.1
test-server-agent   Ready    <none>          5m55s   v1.29.1

 

 

---

 

Ceph-ansible 설치

 

git clone을 한다.

git clone <https://github.com/ceph/cephadm-ansible>

VENVDIR=cephadm-venv
CEPAHADMDIR=cephadm-ansible
python3.10 -m venv $VENVDIR
source $VENVDIR/bin/activate
cd $CEPAHADMDIR

pip install -U -r requirements.txt

 

`test-server-storage` 디스크를 확인한다.

lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0  63.9M  1 loop /snap/core20/2105
loop1     7:1    0 368.2M  1 loop /snap/google-cloud-cli/207
loop2     7:2    0  40.4M  1 loop /snap/snapd/20671
loop3     7:3    0  91.9M  1 loop /snap/lxd/24061
sda       8:0    0    50G  0 disk
├─sda1    8:1    0  49.9G  0 part /
├─sda14   8:14   0     4M  0 part
└─sda15   8:15   0   106M  0 part /boot/efi
sdb       8:16   0    50G  0 disk
sdc       8:32   0    50G  0 disk
sdd       8:48   0    50G  0 disk

 

`inventory.ini` 생성

[all]
test-server ansible_host=10.77.101.47
test-server-agent ansible_host=10.77.101.43
test-server-storage ansible_host=10.77.101.48

# Ceph Client Nodes (Kubernetes nodes that require access to Ceph storage)
[clients]
test-server
test-server-agent
test-server-storage

# Admin Node (Usually the first monitor node)
[admin]
test-server

 

 

`cephadm-distribute-ssh-key.yml` 수정

 

`or not cephadm_pubkey_path_stat.stat.isfile | bool → or not cephadm_pubkey_path_stat.stat.isre | bool`

---
# Copyright Red Hat
# SPDX-License-Identifier: Apache-2.0
# Author: Guillaume Abrioux <gabrioux@redhat.com>
#
# This playbook copies an SSH public key to a specified user on remote hosts.
#
# Required run-time variables
# ------------------
# admin_node : The name of a node with enough privileges to call `cephadm get-pub-key` command.
#              (usually the bootstrap node).
#
# Optional run-time variables
# ------------------
# fsid : The fsid of the Ceph cluster.
# cephadm_ssh_user : ssh username on remote hosts.
# cephadm_pubkey_path : Full path name of the ssh public key file *on the ansible controller host*.
#                       If not passed, the playbook will assume it has to get the key from `cephadm get-pub-key` command.
#
# Example
# -------
# ansible-playbook -i hosts cephadm-distribute-ssh-key.yml -e cephadm_ssh_user=foo -e cephadm_pubkey_path=/home/cephadm/ceph.key -e admin_node=ceph-node0
#
# ansible-playbook -i hosts cephadm-distribute-ssh-key.yml -e cephadm_ssh_user=foo -e admin_node=ceph-node0

- hosts: all
  become: true
  gather_facts: false
  tasks:
    - name: fail if admin_node is not defined
      fail:
        msg: "You must set the variable admin_node"
      run_once: true
      delegate_to: localhost
      when: admin_node is undefined

    - name: get ssh public key from a file on the Ansible controller host
      when: cephadm_pubkey_path is defined
      block:
        - name: get details about {{ cephadm_pubkey_path }}
          stat:
            path: "{{ cephadm_pubkey_path }}"
          register: cephadm_pubkey_path_stat
          delegate_to: localhost
          run_once: true

        - name: fail if {{ cephadm_pubkey_path }} doesn't exist
          fail:
            msg: "{{ cephadm_pubkey_path }} doesn't exist or is invalid."
          run_once: true
          delegate_to: localhost
          when:
            - not cephadm_pubkey_path_stat.stat.exists | bool
              or not cephadm_pubkey_path_stat.stat.isreg | bool 

    - name: get the cephadm ssh pub key
      command: "cephadm shell {{ '--fsid ' + fsid if fsid is defined else '' }} ceph cephadm get-pub-key"
      changed_when: false
      run_once: true
      register: cephadm_get_pub_key
      delegate_to: "{{ admin_node }}"
      when: cephadm_pubkey_path is undefined

    - name: allow ssh public key for {{ cephadm_ssh_user | default('root') }} account
      authorized_key:
        user: "{{ cephadm_ssh_user | default('root') }}"
        key: "{{ lookup('file', cephadm_pubkey_path) if cephadm_pubkey_path is defined else cephadm_get_pub_key.stdout }}"

    - name: set cephadm ssh user to {{ cephadm_ssh_user }}
      command: "cephadm shell {{ '--fsid ' + fsid if fsid is defined else '' }} ceph cephadm set-user {{ cephadm_ssh_user | default('root') }}"
      changed_when: false
      run_once: true
      delegate_to: "{{ admin_node }}"

 

`cephadm-preflight.yml` 수정 

    - name: redhat family of OS related tasks
      when: ansible_facts['os_family'] == 'RedHat'
      block:
...
        - name: install jq package
          package:
            name: jq
            state: present
          register: result
          until: result is succeeded
...
    - name: Ubuntu related tasks
      when: ansible_facts['distribution'] == 'Ubuntu'
      block:
        - name: Update apt cache
          apt:
            update_cache: yes
            cache_valid_time: 3600 # Optional: cache validity in seconds
          changed_when: false

...
        - name: install container engine
          block:
...
           - name: install docker
                - name: ensure Docker is running
                  service:
                    name: docker
                    state: started
                    enabled: true

        - name: install jq package
          apt:
           name: jq
           state: present
           update_cache: yes
          register: result
          until: result is succeeded

 

 

스크립트 작성

 

`ceph_vars.sh`

#!/bin/bash

# Define variables(Modify)
SSH_KEY="/home/somaz/.ssh/id_rsa_ansible" # SSH KEY Path
INVENTORY_FILE="inventory.ini" # Inventory Path
CEPHADM_PREFLIGHT_PLAYBOOK="cephadm-preflight.yml"
CEPHADM_CLIENTS_PLAYBOOK="cephadm-clients.yml"
CEPHADM_DISTRIBUTE_SSHKEY_PLAYBOOK="cephadm-distribute-ssh-key.yml"
HOST_GROUP=(test-server test-server-agent test-server-storage) # All host group
ADMIN_HOST="test-server" # Admin host name
OSD_HOST="test-server-storage" # Osd host name
HOST_IPS=("10.77.101.47" "10.77.101.43" "10.77.101.48") # Corresponding IPs and Select the first IP address for MON_IP
OSD_DEVICES=("sdb" "sdc" "sde") # OSD devices, without /dev/ prefix
CLUSTER_NETWORK="10.77.101.0/24" # Cluster network CIDR
SSH_USER="somaz" # SSH user
CLEANUP_CEPH="false" # Ensure this is reset based on user input

 

`ceph_functions.sh`

#!/bin/bash

stop_all_ceph_services() {
    echo "Stopping all Ceph services..."
    local services=$(sudo ceph orch ls --format=json | jq -r '.[].service_name')
    for service in $services; do
        # Skip attempting to stop 'mgr', 'mon', and 'osd' services
        if [[ "$service" != "mgr" && "$service" != "mon" && "$service" != "osd" ]]; then
            echo "Stopping $service..."
            sudo ceph orch stop $service
        else
            echo "Skipping $service (cannot be stopped directly)."
        fi
    done
}

remove_osds_and_cleanup_lvm() {
    echo "Checking for existing OSDs and cleaning them up..."
    osd_ids=$(sudo ceph osd ls)
    if [ -z "$osd_ids" ]; then
        echo "No OSDs to remove."
    else
        for osd_id in $osd_ids; do
            echo "Removing OSD.$osd_id..."
            # Stop the OSD
            sudo ceph orch daemon stop osd.$osd_id
            sleep 10
            # Mark the OSD out
            sudo ceph osd out $osd_id
            sleep 10
            # Remove the OSD
            sudo ceph osd rm $osd_id
            # Wait a bit to ensure the OSD is fully purged
            sleep 10
        done
    fi

    echo "Cleaning up LVM volumes on $OSD_HOST..."
    ssh $OSD_HOST <<'EOF'
        sudo lvscan | awk '/ceph/ {print $2}' | xargs -I{} sudo lvremove -y {}
        sleep 5
        sudo vgscan | awk '/ceph/ {print $4}' | xargs -I{} sudo vgremove -y {}
        sleep 5
        sudo pvscan | grep '/dev/sd' | awk '{print $2}' | xargs -I{} sudo pvremove -y {}
EOF
}

# Function to cleanup existing Ceph cluster
cleanup_ceph_cluster() {
    if [ "$CLEANUP_CEPH" == "true" ]; then
        echo "Initiating Ceph cluster cleanup..."

        # Stop all Ceph services managed by cephadm
        stop_all_ceph_services

        # Wait a bit to ensure all services are stopped
        sleep 10

        # Remove OSDs and clean up LVM before removing the cluster
        remove_osds_and_cleanup_lvm

        # Proceed with cluster cleanup
        if sudo test -d /var/lib/ceph; then
            echo "Cleaning up existing Ceph cluster..."
            sudo cephadm rm-cluster --fsid=$(sudo ceph fsid) --force
            echo "Ceph cluster removed successfully."
        else
            echo "No existing Ceph cluster found to clean up."
        fi

        # Dynamically determine whether to use Docker or Podman for container cleanup
        echo "Removing any leftover Ceph containers..."
        for host in "${HOST_GROUP[@]}"; do
            echo "Cleaning up containers on $host..."
            ssh "$SSH_USER@$host" '
            if command -v docker &> /dev/null; then
                container_runtime="docker"
            elif command -v podman &> /dev/null; then
                container_runtime="podman"
            else
                echo "No container runtime (Docker or Podman) found on '"$host"'. Skipping container cleanup."
                exit 1 # Exit the SSH command with an error status
            fi

            # Remove Ceph containers using the detected container runtime
            sudo $container_runtime ps -a | grep ceph | awk '"'"'{print $1}'"'"' | xargs -I {} sudo $container_runtime rm -f {}
            '
            if [ $? -ne 0 ]; then
                echo "Error cleaning up containers on $host"
            else
                echo "Leftover Ceph containers removed successfully on $host."
            fi
        done
    else
        echo "Skipping Ceph cluster cleanup as per user's choice."
    fi
}

# Function to execute ansible playbook
run_ansible_playbook() {
    playbook=$1
    extra_vars=$2
    ansible-playbook -i $INVENTORY_FILE $playbook $extra_vars --become
    if [ $? -ne 0 ]; then
        echo "Execution of playbook $playbook failed. Exiting..."
        exit 1
    fi
}

# Function to add SSH key to known hosts
add_to_known_hosts() {
    host_ip=$1
    ssh-keyscan -H $host_ip >> ~/.ssh/known_hosts
}

# Function to add OSDs and wait for them to be ready
add_osds_and_wait() {
    for device in "${OSD_DEVICES[@]}"; do
        echo "Attempting to add OSD on /dev/$device..."
        # Attempt to add an OSD and capture the output
        output=$(sudo /usr/bin/ceph orch daemon add osd $OSD_HOST:/dev/$device 2>&1)
        retval=$?

        if [ $retval -ne 0 ]; then
            echo "Command to add OSD on /dev/$device failed. Please check logs for errors. Output: $output"
            continue # If the command failed, move to the next device.
        fi

        # Since OSD ID might not be immediately available, wait a few seconds before checking
        echo "Waiting a moment for OSD to be registered..."
        sleep 10

        # Attempt to find the OSD ID using the ceph osd tree command, assuming it will be listed there if creation was successful
        osd_id=$(sudo /usr/bin/ceph osd tree | grep -oP "/dev/$device.*osd.\\K[0-9]+")

        if [ -z "$osd_id" ]; then
            echo "Unable to find OSD ID for /dev/$device. It might take a moment for the OSD to be visible in the cluster."
        else
            echo "OSD with ID $osd_id has been added on /dev/$device."
        fi

        echo "Monitoring the readiness of OSD.$osd_id on /dev/$device..."

        # Initialize success flag
        success=false
        for attempt in {1..12}; do
            # Directly check if the OSD is "up" and "in"
            if sudo /usr/bin/ceph osd tree | grep "osd.$osd_id" | grep -q "up" && sudo /usr/bin/ceph osd tree | grep "osd.$osd_id"; then
                echo "OSD.$osd_id on /dev/$device is now ready."
                success=true
                break
            else
                echo "Waiting for OSD.$osd_id on /dev/$device to become ready..."
                sleep 10
            fi
        done

        if ! $success; then
            echo "Timeout waiting for OSD.$osd_id on /dev/$device to become ready. Please check Ceph cluster status."
        fi
    done
}

# Function to check OSD creation and cluster status
check_osd_creation() {
    echo "Checking Ceph cluster status and OSD creation..."
    sudo ceph -s
    sudo ceph osd tree
}

# Add and label hosts in the Ceph cluster
add_host_and_label() {
  echo "Adding and labeling hosts in the cluster..."
  for i in "${!HOST_GROUP[@]}"; do
      host="${HOST_GROUP[$i]}"
      ip="${HOST_IPS[$i]}"

      # Add the public key of the Ceph cluster to each host
      ssh-copy-id -f -i /etc/ceph/ceph.pub $host

      # Add host to Ceph cluster
      sudo ceph orch host add $host $ip

      # Apply 'mon' and 'mgr' labels to the admin host
      if [[ "$host" == "$ADMIN_HOST" ]]; then
          sudo ceph orch host label add $host mon && \\
          sudo ceph orch host label add $host mgr && \\
          echo "Labels 'mon' and 'mgr' added to $host."
      fi

      # Apply 'osd' label to the OSD host
      if [[ "$host" == "$OSD_HOST" ]]; then
          sudo ceph orch host label add $host osd && \\
          echo "Label 'osd' added to $host."
      fi

      # Apply '_no_schedule' label only to hosts that are neither ADMIN_HOST nor OSD_HOST
      if [[ "$host" != "$ADMIN_HOST" ]] && [[ "$host" != "$OSD_HOST" ]]; then
          sudo ceph orch host label add $host _no_schedule && \\
          echo "Label '_no_schedule' added to $host."
      fi

      # Verify the labels have been applied
      labels=$(sudo ceph orch host ls --format=json | jq -r '.[] | select(.hostname == "'$host'") | .labels[]')
      echo "Current labels for $host: $labels"
  done
}

# Function to label all OSD hosts with '_no_schedule'
label_osd_hosts_no_schedule() {
    echo "Applying '_no_schedule' label to all OSD hosts..."
    # Assuming OSD_HOST could be a single host or an array of hosts
    # Convert OSD_HOST to an array if it's not already one
    if [[ ! "${OSD_HOST[@]}" ]]; then
        osd_hosts=($OSD_HOST) # Convert single host to an array
    else
        osd_hosts=("${OSD_HOST[@]}") # Use the array as is
    fi

    for osd_host in "${osd_hosts[@]}"; do
        sudo ceph orch host label add $osd_host _no_schedule && \\
        echo "Label '_no_schedule' added to $osd_host."
    done
}

 

`setup_ceph_cluster.sh`

#!/bin/bash

# Load functions from ceph_functions.sh and ceph_vars.sh
source ceph_vars.sh
source ceph_functions.sh

read -p "Do you want to cleanup existing Ceph cluster? (yes/no): " user_confirmation
if [[ "$user_confirmation" == "yes" ]]; then
    CLEANUP_CEPH="true"
else
    CLEANUP_CEPH="false"
fi

# Start setup
echo "Starting Ceph cluster setup..."

# Check for existing SSH key and generate if it does not exist
if [ ! -f "$SSH_KEY" ]; then
    echo "Generating SSH key..."
    ssh-keygen -f "$SSH_KEY" -N '' # No passphrase
    echo "SSH key generated successfully."
else
    echo "SSH key already exists. Skipping generation."
fi

# Copy SSH key to each host in the group
for host in "${HOST_GROUP[@]}"; do
    echo "Copying SSH key to $host..."
    ssh-copy-id -i "${SSH_KEY}.pub" -o StrictHostKeyChecking=no "$host" && \
    echo "SSH key copied successfully to $host." || \
    echo "Failed to copy SSH key to $host."
done

# Cleanup existing Ceph setup if confirmed
cleanup_ceph_cluster

# Wipe OSD devices
echo "Wiping OSD devices on $OSD_HOST..."
for device in ${OSD_DEVICES[@]}; do
    if ssh $OSD_HOST "sudo wipefs --all /dev/$device"; then
        echo "Wiped $device successfully."
    else
        echo "Failed to wipe $device."
    fi
done

# Run cephadm-ansible preflight playbook
echo "Running cephadm-ansible preflight setup..."
run_ansible_playbook $CEPHADM_PREFLIGHT_PLAYBOOK ""

# Create a temporary Ceph configuration file for initial settings
TEMP_CONFIG_FILE=$(mktemp)
echo "[global]
osd crush chooseleaf type = 0
osd_pool_default_size = 1" > $TEMP_CONFIG_FILE

# Bootstrap the Ceph cluster
MON_IP="${HOST_IPS[0]}"  # Select the first IP address for MON_IP
echo "Bootstrapping Ceph cluster with MON_IP: $MON_IP"
add_to_known_hosts $MON_IP
sudo cephadm bootstrap --mon-ip $MON_IP --cluster-network $CLUSTER_NETWORK --ssh-user $SSH_USER -c $TEMP_CONFIG_FILE --allow-overwrite --log-to-file
rm -f $TEMP_CONFIG_FILE

# Distribute Cephadm SSH keys to all hosts
echo "Distributing Cephadm SSH keys to all hosts..."
run_ansible_playbook $CEPHADM_DISTRIBUTE_SSHKEY_PLAYBOOK \\
                "-e cephadm_ssh_user=$SSH_USER -e admin_node=$ADMIN_HOST -e cephadm_pubkey_path=$SSH_KEY.pub"

# Fetch FSID of the Ceph cluster
FSID=$(sudo ceph fsid)
echo "Ceph FSID: $FSID"

# Add and label hosts in the Ceph cluster
add_host_and_label

# Prepare and add OSDs
sleep 60
add_osds_and_wait

# all OSD hosts with '_no_schedule'
sleep 60
label_osd_hosts_no_schedule

# Check Ceph cluster status and OSD creation
check_osd_creation

echo "Ceph cluster setup and client configuration completed successfully."

 

 

스크립트 실행

# 스크립트 실행
chmod +x setup_ceph_cluster.sh
./setup_ceph_cluster.sh

 

...
Ceph Dashboard is now available at:

             URL: <https://test-server:8443/>
            User: admin
        Password: 9m16nzu1h7

 

 

명령어

# ceph host 확인
sudo ceph orch host ls
test-server          10.77.101.47  _admin
test-server-agent    10.77.101.43
test-server-storage  10.77.101.48  osd
3 hosts in cluster

# ceph 상태확인
sudo ceph -s
  cluster:
    id:     43d4ca77-cf91-11ee-8e5d-831aa89df15f
    health: HEALTH_WARN
            1 pool(s) have no replicas configured

  services:
    mon: 1 daemons, quorum test-server (age 7m)
    mgr: test-server.nckhts(active, since 6m)
    osd: 3 osds: 3 up (since 3m), 3 in (since 3m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 577 KiB
    usage:   872 MiB used, 149 GiB / 150 GiB avail
    pgs:     1 active+clean

# osd 확인
sudo ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME                     STATUS  REWEIGHT  PRI-AFF
-1         0.14639  root default
-3         0.14639      host test-server-storage
 0    ssd  0.04880          osd.0                     up   1.00000  1.00000
 1    ssd  0.04880          osd.1                     up   1.00000  1.00000
 2    ssd  0.04880          osd.2                     up   1.00000  1.00000

sudo ceph orch ls --service-type mon
NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
mon              3/3  10m ago    5m   count:5

sudo ceph orch ls --service-type mgr
NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
mgr              2/2  2m ago     5m   count:2

sudo ceph orch ls --service-type osd
NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
osd                3  10m ago    -    <unmanaged>

# error 해결 replica 1 -> 2
sudo ceph osd pool get .mgr size
size: 1

sudo ceph osd pool set .mgr size 2
set pool 1 size to 2

sudo ceph -s
  cluster:
    id:     fcc8af47-cfa2-11ee-8e5d-831aa89df15f
    health: HEALTH_WARN
            Reduced data availability: 1 pg peering

  services:
    mon: 1 daemons, quorum test-server (age 99m)
    mgr: test-server.jdncfq(active, since 97m)
    osd: 3 osds: 3 up (since 94m), 3 in (since 95m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 577 KiB
    usage:   872 MiB used, 149 GiB / 150 GiB avail
    pgs:     100.000% pgs not active
             1 peering

# 완료
sudo ceph -s
  cluster:
    id:     fcc8af47-cfa2-11ee-8e5d-831aa89df15f
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum test-server (age 99m)
    mgr: test-server.jdncfq(active, since 97m)
    osd: 3 osds: 3 up (since 94m), 3 in (since 95m)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 577 KiB
    usage:   873 MiB used, 149 GiB / 150 GiB avail
    pgs:     1 active+clean

  io:
    recovery: 96 KiB/s, 0 objects/s

 

pool을 생성한다.

# pool 확인

ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    150 GiB  149 GiB  872 MiB   872 MiB       0.57
TOTAL  150 GiB  149 GiB  872 MiB   872 MiB       0.57

--- POOLS ---
POOL  ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr   1    1  1.1 MiB        2  1.1 MiB      0    142 GiB

# pool 생성
ex. ceph osd pool create {pool-name} {pg-num} [{pgp-num}] [replicated] [crush-rule-name] [expected-num-objects]

ceph osd pool create kube 128

# pool replica 확인
ex. ceph osd pool get kube size
size: 1

# pool replica 변경
ex. ceph osd pool set kube size 3

# pool 삭제
ex. ceph osd pool delete {pool-name} {pool-name} --yes-i-really-really-mean-it

# pg 변경
ex. ceph osd pool set {pool-name} pg_num {new-pg-num}

ceph osd pool set kube pg_num 256

# 확인
ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    150 GiB  149 GiB  872 MiB   872 MiB       0.57
TOTAL  150 GiB  149 GiB  872 MiB   872 MiB       0.57

--- POOLS ---
POOL  ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr   1    1  1.1 MiB        2  1.1 MiB      0    142 GiB
kube   2  128      0 B        0      0 B      0    142 GiB

• {pool-name}: 생성할 풀의 이름이다.
• {pg-num}: 풀에 대한 Placement Group의 수이다. Ceph 클러스터의 성능과 확장성을 결정하는 중요한 값으로, 적절한 값 설정이 필요하다.
• {pgp-num}: (선택 사항) Placement Group의 수이다. 일반적으로 {pg-num}과 같은 값을 사용한다.
• replicated 또는 erasure: 풀의 타입을 지정한다. replicated는 복제 풀을, erasure는 오류 정정(Erasure Coding) 풀을 의미한다.
• [crush-rule-name]: (선택 사항) CRUSH 맵에서 사용할 규칙의 이름이다.
• [expected-num-objects]: (선택 사항) 풀에서 예상하는 오브젝트의 수이다.

 

 

Placement Groups(PG란?)

Ceph에서 PGs는 "Placement Groups"의 약자이다. Placement Group은 Ceph 클러스터 내의 데이터 분산 및 관리를 위한 논리적인 단위이다. Ceph는 데이터를 오브젝트로 저장하고, 이 오브젝트들을 PGs에 할당한다. 그런 다음, PGs는 클러스터 내의 다양한 OSDs(Object Storage Daemons)에 분산된다. PGs의 주요 목적은 클러스터의 확장성, 성능, 복구력을 최적화한다.

​

 

---

 

Ceph-csi 설치

Ceph를 Kubernetes Pod용 스토리지 솔루션으로 사용하려면 Ceph-CSI(Container Storage Interface)를 설치해야 한다. Ceph-CSI를 사용하면 Ceph를 Kubernetes용 영구 스토리지로 사용할 수 있으며 블록 및 파일 스토리지를 모두 지원한다.

 

Kubernetes 클러스터에 Ceph-CSI 설치

helm chart 등록 및 설치 방법

# helm chart 등록
helm repo add ceph-csi <https://ceph.github.io/csi-charts>
helm repo update

# search
helm search repo ceph-csi
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
ceph-csi/ceph-csi-cephfs        3.10.2          3.10.2          Container Storage Interface (CSI) driver, provi...
ceph-csi/ceph-csi-rbd           3.10.2          3.10.2          Container Storage Interface (CSI) driver, provi...

# rbd(사용)
helm install ceph-csi-rbd ceph-csi/ceph-csi-rbd --namespace ceph-csi --create-namespace --version 

# cephfs
helm install ceph-csi-cephfs ceph-csi/ceph-csi-cephfs --namespace ceph-csi --create-namespace --version 

 

 

`ceph-csi-values.yaml` 작성

Worker Node가 한대이기 때문에 replicaCount를 1로 작성한다.

# ceph id 확인
sudo ceph fsid
afdfd487-cef1-11ee-8e5d-831aa89df15f

ss -nlpt | grep 6789
LISTEN  0        512         10.77.101.47:6789           0.0.0.0:*

csiConfig:
- clusterID: "afdfd487-cef1-11ee-8e5d-831aa89df15f" # ceph id
  monitors:
  - "10.77.101.47:6789"
provisioner:
  replicaCount: 1

 

ceph-csi driver 설치

# namespace 생성
kubectl create namespace ceph-csi

# 설치
helm install -n ceph-csi ceph-csi ceph-csi/ceph-csi-rbd -f ceph-csi-values.yaml

# 확인
k get all -n ceph-csi
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/ceph-csi-ceph-csi-rbd-nodeplugin-76k5s              3/3     Running   0          3s
pod/ceph-csi-ceph-csi-rbd-provisioner-5d5dc6cc4-62dzb   7/7     Running   0          3s

NAME                                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/ceph-csi-ceph-csi-rbd-nodeplugin-http-metrics    ClusterIP   10.233.37.117   <none>        8080/TCP   3s
service/ceph-csi-ceph-csi-rbd-provisioner-http-metrics   ClusterIP   10.233.41.120   <none>        8080/TCP   3s

NAME                                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/ceph-csi-ceph-csi-rbd-nodeplugin   1         1         1       1            1           <none>          3s

NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ceph-csi-ceph-csi-rbd-provisioner   1/1     1            1           3s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/ceph-csi-ceph-csi-rbd-provisioner-5d5dc6cc4   1         1         1       3s

 

StorageClass 생성

ceph auth 확인한다.

sudo ceph auth list
osd.0
        key: AQBH+NJldCD+LxAAJEYjXTU8Ut/X+HIhY3+B2A==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.1
        key: AQBq+NJlM32PCBAAiqRNFT51lZySBQf6OBx4CA==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
osd.2
        key: AQCM+NJlRVzuFRAAD9WmLwaux2o9IVekaj+EPA==
        caps: [mgr] allow profile osd
        caps: [mon] allow profile osd
        caps: [osd] allow *
client.admin
        key: AQC899JlcL6CKBAAQsBOJqWw/CVTQKUD+2FbyQ==
        caps: [mds] allow *
        caps: [mgr] allow *
        caps: [mon] allow *
        caps: [osd] allow *
client.bootstrap-mds
        key: AQC/99Jlnn4YKxAAHWu/37QhHVoBcIVLE+MESA==
        caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
        key: AQC/99JlhIcYKxAAVGoYajMGN/HG+sFiNUR48w==
        caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
        key: AQC/99JlZI8YKxAA0anFdZuya3lAX6m6udZKfw==
        caps: [mon] allow profile bootstrap-osd
client.bootstrap-rbd
        key: AQC/99JlGJgYKxAAKFkn9FURVTfPJeZXkJuYNg==
        caps: [mon] allow profile bootstrap-rbd
client.bootstrap-rbd-mirror
        key: AQC/99Jllp8YKxAAr08flZFO67AeVveinX5DbA==
        caps: [mon] allow profile bootstrap-rbd-mirror
client.bootstrap-rgw
        key: AQC/99JluacYKxAAuBaFQKoCH7Wj3043SDdaaQ==
        caps: [mon] allow profile bootstrap-rgw
client.crash.test-server
        key: AQD+99JlG6eAKRAAHBF0nKwc8ydGbUG+5/YrJg==
        caps: [mgr] profile crash
        caps: [mon] profile crash
client.crash.test-server-agent
        key: AQAp+NJlJEhRLRAAvsezeXOx4ARsGPlxW/DC9A==
        caps: [mgr] profile crash
        caps: [mon] profile crash
client.crash.test-server-storage
        key: AQBA+NJlyYUxIxAAmsCECURifC9HRPOmdQvpXg==
        caps: [mgr] profile crash
        caps: [mon] profile crash
mgr.test-server-agent.loabgm
        key: AQAr+NJlAqCMIRAAoCUieg9dQryf8qyI1zpo0Q==
        caps: [mds] allow *
        caps: [mon] profile mgr
        caps: [osd] allow *
mgr.test-server.nkvwfr
        key: AQC999JlEXy3ABAAVwCd+muVuoT5V+r66Groig==
        caps: [mds] allow *
        caps: [mon] profile mgr
        caps: [osd] allow *

 

Secret과 StorageClass를 생성한다. client.admin 키를 userKey로 사용했다.

cat <<EOF > ceph-csi-storageclass.yaml
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: kube-system
stringData:
  userID: admin # ceph user id(client.admin) admin이 user
  userKey: "AQC899JlcL6CKBAAQsBOJqWw/CVTQKUD+2FbyQ==" # client.admin key
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rbd
   annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true"
     storageclass.kubesphere.io/supported-access-modes: '["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]'
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: "afdfd487-cef1-11ee-8e5d-831aa89df15f"
   pool: "kube"
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: kube-system
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: kube-system
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: kube-system
   csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard
EOF

# apply
k apply -f ceph-csi-storageclass.yaml 

# 확인
k get sc
NAME            PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rbd (default)   rbd.csi.ceph.com   Delete          Immediate           true                   21s

 

간단하게 Pod를 배포한다.

cat <<EOF > test-pod.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-using-ceph-rbd
spec:
  containers:
  - name: my-container
    image: nginx
    volumeMounts:
    - mountPath: "/var/lib/www/html"
      name: mypd
  volumes:
  - name: mypd
    persistentVolumeClaim:
      claimName: ceph-rbd-pvc
EOF

# apply
k apply -f test-pod.yaml

 

잘 생성되었다.

k get po,pv,pvc
NAME                     READY   STATUS    RESTARTS   AGE
pod/pod-using-ceph-rbd   1/1     Running   0          16s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-83fd673e-077c-4d24-b9c9-290118586bd3   1Gi        RWO            Delete           Bound    default/ceph-rbd-pvc   rbd            <unset>                          16s

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/ceph-rbd-pvc   Bound    pvc-83fd673e-077c-4d24-b9c9-290118586bd3   1Gi        RWO            rbd            <unset>                 16s

 

 

 

---

Reference

https://github.com/ceph/cephadm-ansible

https://docs.ceph.com/en/latest/cephadm/

https://idchowto.com/ceph-ansible-을-이용하여-ceph-구성-ubuntu22-04/

https://devocean.sk.com/blog/techBoardDetail.do?ID=163924

https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/values.yaml

https://docs.ceph.com/en/latest/cephadm/host-management/#special-host-labels