HPC compute and storage components will continue to converge during the coming decade. On-prem and cloud storages require greater speed and flexibility to move data between storage tiers rapidly, securely and economically. Vastly increased compute performances are pushing storage loads to the limit. Simply moving data to middle and archive tier storages has become a genuine technological and economical challenge.

This paper looks at the current HPC data lifecycle landscape and illustrates how customers manage high data volumes rapidly and securely on the edge and in the cloud. It examines ways to move and archive data within the various storage tiers and how rapid file tree scans for Lustre, GPFS and other file systems enable incremental forever backup and data versioning capability even at petabyte scale.