Lenovo built a clustered AI data lake system with AMD servers running Cloudian’s HyperStore object storage.
The hardware used is a Lenovo SR635 V3 all-flash server with a single-socket AMD Gen 4 EPYC 9454P processor (48 cores). A six-node test system equipped with 8 x 7.68 TB NVMe SSDs for data and 2 x 3.84 TB metadata SSDs per node delivered 28.7 Gbps read and 18.4 Gbps write. According to Cloudian’s tests, this system was 74% more energy efficient than an equivalent disk-based system.
Cloudy Michael Tso, CEO and co-founder, said: “Lenovo’s market-leading servers powered by AMD EPYC processors are a perfect complement to Cloudian’s high-performance data software platform. Together, they provide the scalable, high-performance, and efficient foundation that AI and data analytics workloads require.” He suggested that the combined Lenovo-Cloudian system is well-suited for AI, machine learning, and high-performance computing workloads.
Stuart McRae, executive director and general manager of storage at Lenovo, said: “This partnership enables us to offer our customers a leading-edge, scalable and secure platform that will help them accelerate their AI initiatives and drive innovation.”
Cloudian’s HyperStore software is S3-compatible, scalable to exabytes, and features Object Lock immutability to protect against ransomware. There are over 800 enterprise-scale deployments of Cloudian’s HyperStore software.
MinIO recently announced a DataPOD reference architecture for feeding data from its object storage software to Nvidia GPU servers. It cited 46.54 GB/s read and 34.4 GB/s write bandwidth from an eight-node storage server system, with 24 SSDs per node. On a per-node basis, that works out to 5.82 GB/s read and 4.3 GB/s write; faster than the Lenovo-Cloudian system’s 4.78 GB/s read and 3.1 GB/s write.
However, on a per-disk basis, the Lenovo-Cloudian system delivered 0.478 Gbps read and 0.31 Gbps write, while the MinIO DataPOD RA system delivered 0.243 Gbps read and 0.179 Gbps write. This comparison is simplistic and lacks detailed specifications on the systems’ processors, core counts, memory, PCIe fabrics, and network ports. Costs are also not considered. Detailed research is needed to draw concrete conclusions.
Both Cloudian and Lenovo say energy efficiency will become increasingly important, citing a Morgan Stanley Report Generative AI energy consumption is predicted to increase by an average of 70% per year through 2027, meaning that “by 2027, generative AI could use as much energy as Spain needed to power itself in 2022.”
Morgan Stanley analysts believe that the energy needs of generative AI can be met by sustainable sources, and “massive energy demand can also drive advances in sustainable energy technology across sectors.”
The Lenovo/AMD/Cloudian AI Data Lake combination system is now available from Lenovo and authorized resellers.
