Many customers associate a performance cost with data compression, but that’s not the case with Amazon FSx for Lustre. With FSx for Lustre, data compression reduces storage costs and increases aggregate file system throughput. As organizations continue to build applications faster than ever, the amount of data that organizations must store grows rapidly. Organizations need tools to help reduce the amount they spend storing this data. Data compression is an easy way to do just that. With the new data compression feature of Amazon FSx for Lustre, you can now improve performance when reading and writing to your file system while reducing your cost by using a smaller file system. This is a win-win across the board for most workloads. In this post, I’ll show how you can increase both read and write throughput to your Amazon FSx for Lustre file system while using a smaller-sized file system.
Amazon FSx for Lustre is a fully managed service that provides cost-effective, high-performance, scalable storage for compute workloads. It is powered by Lustre, the world’s most popular high-performance file system, and delivers sub-millisecond latencies, up to hundreds of gigabytes per second of throughput and millions of IOPS. It is designed to serve many types of workloads, including machine learning, high performance computing (HPC), video rendering, and financial simulations; in essence, any Linux-based, compute-heavy workload that needs high-performance shared storage.
As of May 2021, you can enable data compression on your Amazon FSx for Lustre file systems to reduce the storage consumption of both your file system storage and your file system backups. This feature is designed to deliver high levels of compression, and based on my testing I’ve found that it does this while delivering higher levels of throughput for read and write operations. This is achieved by automatically compressing the data before writing to disk and automatically uncompressing data after reading from disk. Due to this design, we’re able to read and write more data to disk in the same amount of time, thus increasing throughput and IOPS.
In this blog post, I walk you through the new Amazon FSx for Lustre data compression feature, share the results of a storage consumption and throughput test comparing compressed and uncompressed file systems, and list compression ratios for some common data types.
Read the full blog to learn how Amazon FSx for Lustre’s new data compression feature can help accelerate performance and lower costs.
Reminder: You can learn a lot from AWS HPC engineers by subscribing to the HPC Tech Short YouTube channel, and following the AWS HPC Blog channel.