SAN MATEO, Calif., July 09, 2024 (GLOBE NEWSWIRE) — AlluxionOpen Source Data Platform, developer of the open source data platform, today announced the immediate availability of the latest enhancements to Alluxio Enterprise AI. Version 3.2 highlights the platform’s ability to universally utilize GPU resources, I/O performance improvements, and competitive end-to-end performance with HPC storage. It also introduces a new Python interface and sophisticated cache management capabilities. These advancements enable organizations to fully leverage their AI infrastructure, ensuring optimal performance, cost-effectiveness, flexibility, and ease of management.
AI workloads face several challenges, including the mismatch between data access speed and GPU compute, leading to GPU underutilization due to slow data loading in frameworks like Ray, PyTorch, and TensorFlow. Alluxio Enterprise AI 3.2 addresses this issue by improving I/O performance and achieving GPU utilization above 97%. Additionally, while HPC storage offers good performance, it requires significant infrastructure investments. Alluxio Enterprise AI 3.2 delivers comparable performance using existing data lakes, eliminating the need for additional HPC storage. Finally, managing complex integrations between compute and storage is challenging, but the new release simplifies this with a Pythonic file system interface, supporting POSIX, S3, and Python, making it easily adoptable by different teams.
“At Alluxio, our vision is to provide data to all data-driven applications, including the most advanced AI applications,” said Haoyuan Li, Founder and CEO of Alluxio. “With our latest enterprise AI product, we are taking a significant leap forward in enabling organizations to unlock the full potential of their data and AI investments. We are committed to delivering cutting-edge solutions that address the evolving challenges of the AI landscape, ensuring our customers stay ahead of the curve and unlock the true value of their data.”
Alluxio Enterprise AI includes the following key features:
- Leverage GPUs anywhere for more speed and agility – Alluxio Enterprise AI 3.2 enables enterprises to run AI workloads anywhere GPUs are available, ideal for hybrid and multi-cloud environments. Its intelligent caching and data management bring data closer to GPUs, ensuring efficient use even with remote data. The unified namespace simplifies access across storage systems, enabling seamless AI execution across diverse and distributed environments, enabling scalable AI platforms without data locality constraints.
- Performance comparable to HPC storage – MLPerf benchmarks show that Alluxio Enterprise AI 3.2 matches HPC storage performance, using existing data lake resources. In benchmarks like BERT and 3D U-Net, Alluxio delivers comparable model training performance on various A100 GPU configurations, proving its scalability and efficiency in real-world production environments without requiring additional HPC storage infrastructure.
- Higher I/O performance and GPU utilization over 97% – Alluxio Enterprise AI 3.2 improves I/O performance, achieving up to 10GB/s throughput and 200,000 IOPS with a single client, scalable to hundreds of clients. This performance fully saturates 8 A100 GPUs on a single node, showing over 97% GPU utilization in large language model training tests. New support for read/write checkpointing optimizes training recommendation engines and large language models, avoiding GPU idle time.
- New Filesystem API for Python Applications – Version 3.2 introduces the Alluxio Python FileSystem API, an FSSpec implementation, allowing seamless integration with Python applications. This extends Alluxio’s interoperability within the Python ecosystem, allowing frameworks like Ray to easily access local and remote storage systems.
- Advanced cache management for greater efficiency and control – Version 3.2 delivers advanced cache management capabilities, giving administrators fine-grained control over data. A new RESTful API facilitates transparent cache management, while an intelligent cache filter optimizes disk usage by selectively caching important data. The cache free command provides granular control, improving cache efficiency, reducing costs, and increasing data management flexibility.
“The latest release of Alluxio Enterprise AI is a game changer for our customers, delivering unmatched performance, flexibility, and ease of use,” said Adit Madan, Chief Product Officer at Alluxio. “By achieving performance comparable to HPC storage and enabling GPU usage anywhere, we are not only solving today’s challenges, we are also preparing AI workloads for the next generation of innovation. With the introduction of our Python FileSystem API, Alluxio is enabling data scientists and AI engineers to focus on building breakthrough models without worrying about data access bottlenecks or resource constraints.”
“We have successfully deployed a secure and efficient data lake architecture based on Alluxio. This strategic initiative has significantly improved the performance of our compute engines and simplified data engineering workflows, making data processing and analysis smoother and more efficient,” said Hu Zhicheng, Data Architect at Geely (Volvo’s parent company). “We are honored to collaborate with Alluxio to build a cutting-edge data and AI platform that powers the future of data-driven intelligent development.”
Availability
Alluxio Enterprise AI version 3.2 is immediately available for download here: https://www.alluxio.io/download/.
Support Resources
About Alluxio
AlluxionAlluxio, a leading provider of high-performance data platforms for analytics and AI, accelerates time to value for data and AI initiatives and maximizes ROI on infrastructure. Uniquely positioned at the intersection of compute and storage, Alluxio provides a universal view of workloads on the data platform across stages of a data pipeline. This enables Alluxio to provide high-performance data access regardless of where the data resides, simplify data engineering, optimize GPU utilization, and reduce cloud and storage costs. With Alluxio, organizations can achieve significantly faster model training and serving without the need for specialized storage and build AI infrastructure on existing data lakes. Backed by leading investors, Alluxio powers technology, internet, financial services, and telecommunications companies, including 9 of the world’s top 10 internet companies. To learn more, visit the website www.alluxio.io.
Media contact:
Beth Winkowski
Winkowski Public Relations, LLC for Alluxio
978-649-7189
beth@alluxio.com