Replacing OSDs

There is not one perfect way to replace an OSD, it depends on the circumstance and your hardware.

Maintaining Redundancy

We always want to maintain as many copies as possible. If the OSD daemon is still running and not causing performance issues, it is prudent to drain it first:

ceph osd out $ID
ceph osd crush reweight osd.$ID 0 # Optional: Will cause data to backfill across other hosts and improve balancing
# Then you have to wait for this command to clear before proceeding
ceph osd safe-to-destroy osd.$ID

If the OSD is not running/starting anymore and your cluster is otherwise healthy, you can proceed. If the OSD has been down long enough, chances are Ceph already drained the OSD for you.

Recovery Speed

We might want to replace an OSD as quickly as possible. If we have enough copies and are not concerned about maintaining as many copies as possible, we might be able proceed right away. This can be checked with:

ceph osd ok-to-stop osd.$ID

If it shows "ok_to_stop":true and no PGs would become inactive, we can continue.

Warning Proceeding without draining the OSD first means you will be running degraded!

On HDDs, recovering from a failure can take weeks. Running degraded for extended periods of time increases the risk of outages tremendously. It's therefore always recommended to drain HDDs first before replacing them if possible.

Disaster Case

If an OSD died and you don't have enough copies to recover, please contact us. We might be able to extract PGs from dead disks. Don't remove them from the cluster!

Replacing an OSD

Once you feel confident to proceed:

1. Stop the old OSD.
2. Wipe it (this will cleanup any dedicated DB/WAL partitions as well).
3. Use the LED button to locate the disk in the chassis and replace it.
4. Create a new OSD on the replacement disk.
5. Navigate to CRUSH -> Map and delete the old OSD.

If you have space in the chassis, there is no need to wait for the old OSD to drain. Just create a new OSD and remove the old OSD once it has drained.

We are currently working on making both draining and replacing of OSDs easier.