Building croit Ceph on Used Hardware: An Affordable Path to Enterprise Storage

Server component prices are climbing fast. DRAM is up over 250% since October 25, and flash storage costs have more than doubled in recent months it’s becoming increasingly important to maximize the value of existing infrastructure investment.
For many organizations, this means delaying critical storage infrastructure upgrades or expansions simply because budget will not match the need. But there's an alternative which has a cost advantage over other storage systems. Building a Ceph storage cluster on existing, used or cascaded hardware, managed by croit Ceph.

Why Ceph on commodity hardware works

Ceph was designed from the ground up to run on standard, off-the-shelf, commodity hardware. Enabling you to avoid costly controllers and specialist hardware. The Ceph system is designed with the assumption that hardware will fail due to wear or unforeseen evens and builds redundancy into the system. Data is protected automatically using replicated or variable size erasure-coded copies; think raid 1 or raid 6 but across networked nodes. When data is lost, like a failed drive or host, Ceph self-heals and rebalances for performance without intervention.

This design philosophy has an important practical implication: you don't need the latest generation hardware to run production storage. Ceph clusters happily runs on servers that are five, eight, even ten years old. Servers in the cluster don’t even need to be the same vendor or processor type to operate.

Best practices distilled into hardware guidance

If you browse YouTube, you'll find hobbyists running Ceph on old laptops and USB sticks. That's fun for home labs, but obviously not how you'd want to run a critical storage environment.

At croit, we've taken the community best practices and in-house lessons learned from supporting over 200 customers, and combined them to create hardware guidance that covers the majority of real-world use cases not just theoretical requirements from the developer labs.

This includes appropriate sizing for CPUs, memory, drive bays, storage media, and the correct ratios for things like metadata pools and DB/WAL drives for OSDs.

You can find this guidance at: https://www.croit.io/docs/getting-started/hardware-suggestions

Use these specifications as a starting point. Whether you're repurposing existing hardware you already own or shopping the used server market for a new cluster build.

Paths to affordable storage

Extend what you have. Review your current hardware against our reference BOM. Often, modest upgrades, adding RAM, swapping in NVMe for DB/WAL, or adding network cards can bring older servers up to spec. This approach maximizes the value of existing investments.

Adaptor boards. If your servers have been designed around SAS/SATA only there are a number of options for retrofitting older servers with the ability to host NVMe drives. By using 4+x m.2 port adaptors that have a PCIe switch onboard, which take enterprise m.2 boards, you can leverage the systems PCI-e expansion bays.
Additionally, you have the option of upgrading JBOD enclosures to have SAS controller or NVMe bays. By using OCuLink it’s possible to get PCI express switches that present multiple 8x way PCIe.

Buy used. The refurbished enterprise server market is mature and well-established with reputable vendors that test hardware, update firmware, and provide warranties. Enterprise-grade servers from major manufacturers are available at sensible prices. Our hardware suggestions give you the specifications to match when shopping.

Bonus Tips:

In some cases when using used enterprise SSDs it is possible to alter the overprovisioning which would add more life to a drive at the expense of usable capacity, meaning it’s possible to take a read intensive drive and alter it for mixed or write intensive workloads in the same way the OEM vendor does.
Enterprise Hard disks can be reused from proprietary storage arrays by changing the block sizes back to 512 or 4kn using low level formatting tools from the drives OEM vendor. This enables a path way to reusing the disks from a decommissioned EOL SAN environment for your new ceph cluster.

Addressing the used drive lifespan question

When buying used storage media, you can't know its full history or remaining lifespan. That's a legitimate concern, but context matters:

Ceph's redundancy changes the risk calculation. A single drive failure in a properly configured Ceph cluster is a non-event. The system continues operating while automatically rebuilding lost data onto surviving drives, allowing you to fulfil your desired redundancy target seamlessly. If your cluster is architected with appropriate failure domains you can lose multiple drives, servers, entire racks, or even datacenters without data loss; and even maintaining storage availability.

Read-heavy workloads dramatically reduce wear. SSDs wear out from writes, not reads. Archive storage, media streaming, backup targets, and many production workloads are overwhelmingly read-biased. Used drives with high DWPD ratings often have substantial life remaining.

Higher DWPD means more headroom. Enterprise drives with higher Drive Writes Per Day ratings have more internal NAND capacity for wear leveling. When buying used, these drives offer better odds of long service life and they're often available at commodity prices.

Monitoring tells you what you need to know. croit Ceph has SMART monitoring built into the GUI and you can integrate your monitoring software with our flexible management platform. Many used enterprise drives arrive with 80-95% life remaining despite years of datacenter service - your milage may vary.

What you get with croit Ceph

A croit-managed Ceph cluster delivers:

24x7 global support from a team with deep Ceph expertise (including Ceph developers)
High performance and redundancy you can design for your requirements, and we can help.
Multi-protocol access CephFS, RBD, iSCSI, NVMe-oF, NFS, SMB, and S3/Swift-compatible object storage.
No vendor lock-in standard hardware and open-source software defined storage. You can even leave croit, just install and configure the Ceph of your choice - we’ll even show you how
Zero Touch Provisioning with our network booted OS building a cluster is fast. So is expanding or rebuilding (should older hardware break down).
Long track record established 2017 and have been growing year over year.
An active contributor we are founding members of the Ceph foundation and have dedicated Ceph developers on staff contributing to upstream Ceph codebase.
A total solution appropriately sized hardware is half the battle when building an enterprise grade Ceph cluster, the other half is Ceph. We do all the QA on Ceph editions and only build on stable versions of ceph which has met our quality standards. Also, deployment, day 2 operations, and upgrades are made repeatable and reliable via a simple to use GUI. You don’t need to be a Ceph expert to get all the benefits!
See croit Ceph in action: https://youtube.com/playlist?list=PL1g9zo59diHDPyHBqq7ka9QppZklkvPN9&si=suEwgRk3LC4vf6Mp