On this Page:

Hardware Suggestions

Choosing the Right Hardware for Your Ceph Cluster

When building a Ceph cluster, selecting the appropriate hardware can be challenging. This guide aims to simplify the process by providing example Bill of Materials (BOMs) that serve as blueprints tailored to specific scenarios and manufacturers.

Key Considerations

Hardware selection for a Ceph cluster is highly use-case specific. Factors such as workload type, performance expectations, scalability needs, and data center conditions significantly influence the ideal configuration. Ceph's flexibility allows for system designs that can adapt to diverse requirements. While the example BOMs provided in this document are designed to get you started, they might not deliver optimal results for every situation.

Note: We assume no liability for the accuracy, availability, or performance of the collection.

Example BOMs

The following BOMs serve as starting points for commonly encountered use cases:

CROIT Mgmt

Runs the croit software, PXE boot stack, centralized logging, and statistics.

Hardware Specs

  • 1U cost-effective chassis
  • Small and efficient 8-core CPU
  • 32 GB memory
  • 2x NVMe 1.92TB PCIe Gen4+ (redundancy for your OS)
  • Dual-port 10 GbE NIC

Optional

  • Use 64 GB memory for larger clusters (30+ servers)

CEPH-NVMe-10BAY

Throughput Optimized Workloads: Suitable for VMs, Kubernetes storage, databases, and high-performance S3 object storage. Optimal for high-performance services.

Hardware Specs

  • 1U 10-bay chassis Gen5 with 40 PCIe lanes to the bays
  • Minimum 24 CPU cores or more with 48 CPU threads, each ≥ 2.5 GHz
  • 3.84/7.68/15.36TB Gen5+ write-intensive NVMe drives
  • 128 GB memory
  • Dual-port 25/50/100+ GbE NIC
  • 1x 1 TB M.2 or 2.5" NVMe for Croit use (with PLP preferably)

Important

  • No expander backplane!
  • Avoid any PCIe switches, retimers, etc., where possible
  • No RAID cards!

Optional

  • Upgrade to 192/256 GB RAM for better performance
  • Upgrade to more CPU cores for higher overall IOPS
  • Upgrade to higher CPU frequency for better single-client performance
  • Double the memory/CPU cores per additional NVMe namespace

CEPH-HDD-12BAY

Capacity Optimized Workloads: Suitable for archive, backup, price-optimized S3 object storage, data analytics, and AI training data. Optimal for clusters no larger than 10 PB.

Hardware Specs

  • 1U or 2U 12-bay SAS chassis
  • Minimum 8 CPU cores with 16 CPU threads, each ≥ 2.2 GHz
  • 22/24/26 TB CMR SAS HDD
  • 256 GB memory
  • Dual-port 10 or 25 GbE NIC
  • 2x 3.84 TB Gen5+ write-intensive NVMe for DB/WAL for much lower latency and higher performance
  • 1x 1 TB M.2 or 2.5" NVMe for Croit use (with PLP preferably)

Important

  • Do not add any RAID controller; use a simple HBA
  • Ensure that the NVMe drives are directly connected to the CPU

Optional

  • Upgrade to 384/512 GB RAM for slightly better performance

CEPH-HDD-24BAY

Capacity Optimized Workloads: Suitable for archive, backup, price-optimized S3 object storage, data analytics, and AI training data. Optimal for clusters no larger than 20 PB.

Hardware Specs

  • 24-bay SAS chassis
  • Minimum 16 CPU cores with 32 CPU threads, each ≥ 2.2 GHz
  • 22/24/26 TB CMR SAS HDD
  • 256 GB memory
  • Dual-port 10 or 25 GbE NIC
  • 2x 3.84 TB Gen5+ write-intensive NVMe for DB/WAL for much lower latency and higher performance
  • 1x 1 TB M.2 or 2.5" NVMe for Croit use (with PLP preferably)

Important

  • Do not add any RAID controller; use a simple HBA
  • Ensure that the NVMe drives are directly connected to the CPU

CEPH-HDD-60BAY

Very High Density: Suitable for large-scale archive, backup, price-optimized S3 object storage, data analytics, and AI training data. Only suitable for clusters larger than 15 PB.

Hardware Specs

  • 4U 60-bay SAS chassis
  • Minimum 32 CPU cores with 64 CPU threads, each ≥ 2.2 GHz
  • 22/24/26 TB CMR SAS HDD
  • 768 GB memory
  • Dual-port 25/50/100 GbE NIC
  • 4x 7.68 TB Gen5+ write-intensive NVMe for DB/WAL for much lower latency and higher performance
  • 1x 1 TB M.2 or 2.5" NVMe for Croit use (with PLP preferably)

Important

  • Do not add any RAID controller; use a simple HBA
  • Ensure that the NVMe drives are directly connected to the CPU

CephFS MDS Service

Increase CPU cores by 4 or more and increase RAM by 64 GB or more depending on the number of files, clients, and other factors.

For performance reasons, if you expect more than 50k RGW objects or CephFS files, you should equip a separate NVMe 1TB drive per node for RGW/CephFS metadata. Rear slot or M.2 is sufficient. Can use namespace.

Note: For large or complex deployments, we offer consulting services to tailor your hardware choices to your needs.

Note: Choose dedicated MDS servers with high CPU clock frequency to get the best performance.

Seagate MACH.2 HDDs

If you want to go with Western Digital or Seagate MACH.2 dual actuator drives^1, please double the RAM, CPU, and DB/WAL. These drives come with higher IO and bandwidth performance and can benefit your workloads on HDDs.

Note: Please be reminded, an HDD is always much slower than an NVMe.

SAS SSDs

The cost difference between SAS SSDs vs NVMe is so minimal that we would not recommend SAS SSDs as they are constrained to the bandwidth of the controller, if you find prices being dramatically different contact us to work with our partners for competitive pricing. If you already have SAS SSDs and see performance issues you may see benefits by equipping NVMe’s for DB/WALs as this will shift metadata IOPs from the SAS controller to the NVMe drives.

Customized Hardware Proposals

Since hardware needs vary greatly, these example BOMs are not a one-size-fits-all solution. As part of our consulting services, we offer tailored hardware recommendations based on your specific use case and operational constraints. Contact us to discuss your requirements and receive a customized proposal.