Modernizing Software-defined Storage Solutions

Challenge: Data Storage in Research

Whether day or night, in the office or in the lab, on a high mountain or in a remote valley – researchers work in a variety of environments and need to access their data independently of location and without limitations. However, this is only possible with secure, reliable, and fast data storage. Eurac Research relies on a Ceph Software-Defined Storage solution, which, after several years in use, needed to be modernized. They found the right partner for this in croit, a specialist in software-defined storage solutions.

Whether it’s satellite data from the Institute of Earth Observation, research data from the Center for Climate and Change Transformation, or the gene pools of the Institute of Biomedicine – the data base of the private research center Eurac Research in Bolzano is immense. In its eleven institutes and five centers, more than 650 scientists from various disciplines are currently dedicated to one topic: shaping the future. People need health, a functioning environment, sustainable energy, and political and social systems that work. To this end, the researchers develop concrete solutions to regional problems that can be applied globally.

The foundation of all this research is a functioning IT infrastructure. The ability to collect, store, and provide large amounts of data to all employees without downtime is a basic requirement of the South Tyrolean research company. Even though its IT strategy is cloud-oriented, Eurac Research opted for the open-source software-defined storage solution Ceph as an on-premise data storage system some time ago. “Our focus is less on the security aspects of cloud-based IT solutions and more on ensuring very low latency and an extremely large data base,” explains Norbert Andreatta, IT Coordinator for Eurac Research.

Challenge: Upgrading and Supporting the Ceph Cluster

However, managing and professionally operating the established Ceph storage solution required so many personnel that the research company eventually reached its limits. Keeping the Ceph cluster, with around two petabytes of storage and 18 nodes, up to date on both the software and hardware sides was no longer feasible. Therefore, the company sought a partner who could bring the cluster up to date and provide ongoing support. These two tasks are often costly, especially in the long term, as new expenses arise with each update.

Since Eurac Research operates in the academic environment, they learned about the simple support and intuitive management tools from the software provider croit, which specializes in software-defined storage solutions. “When we inquired, croit immediately understood the technical necessity of modernization, the core requirements of our researchers, and our financial constraints. That’s why they offered to license the entire open-source setup,” says Andreatta. This was aimed at minimizing future costs. Additionally, their solution made it possible to implement the software installation and upgrade functions with little involvement from Eurac Research – a crucial advantage given the shortage of IT staff.

In-place Upgrade Instead of Parallel Setup

Together with croit, the Eurac Research team set out to modernize the Ceph cluster. While the hardware replacement proved to be less complicated than initially feared, the software update required careful consideration. A particular challenge was ensuring 24/7 data availability. Researchers do not have predefined working hours like shift workers in industry or office employees. Instead, they need access to their data at all times, making IT outages something to avoid at all costs. Upgrade periods, during which production or work areas are shut down, are also unusual in research. Therefore, an approach had to be found for the cluster upgrade that avoided downtime while consuming minimal resources from Eurac Research. The option of building a second parallel cluster, allowing researchers to handle configuration tasks on their servers, was ruled out due to the effort involved. Additionally, Eurac Research was so far behind on Ceph releases that the option of only upgrading parts of the cluster was also off the table.

croit’s upgrade path provided an alternative. It enabled the in-place upgrade of the cluster to the current Ceph releases. Individual nodes of the cluster were taken out and replaced with new croit nodes until the old Ceph cluster was fully replaced by the modern croit Ceph cluster. Afterward, croit’s mechanism allowed a simple upgrade of the entire cluster, with individual nodes being rebooted with the current version until the entire cluster was up to date. After several releases – professionally supported by the croit team in case of difficulties – the cluster was successfully modernized.

Impact on the Entire Company

The transition is now complete, and Eurac Research is fully operating with the croit Ceph storage solution – primarily using CephFS, to which over 100 servers are directly connected. Since the researchers work with various applications, diverse task areas, and different algorithms in their research projects, the company highly values a shared storage approach. This distributed architecture eliminates performance issues. Furthermore, the research company’s HPC cluster now also operates in the backend with Ceph storage.

Conclusion

The modernization of the Ceph storage solution by croit has significantly reduced the management workload for the software-defined storage architecture at Eurac Research. A crucial benefit, as Norbert Andreatta is convinced: “We are a very small, highly skilled team – constantly on the lookout for new IT specialists. Less system management means we can focus on other tasks while still performing necessary upgrades. If questions or problems arise, we know we can always rely on our partner croit.” Besides this core benefit, the system’s unlimited scalability and hardware-agnostic nature are particularly noteworthy. Stability and performance issues are now a thing of the past.