Aug. 6, 2024 — AMD today announced that the latest release of AMD ROCm 6.2 open stack software is now available. In this blog post, AMD’s Ronak Shah highlights five key enhancements that make this release transformative for AI and HPC development.

---

Whether you’re working on cutting-edge AI models, developing next-gen AI applications, or optimizing complex simulations, this new release brings amazing performance, efficiency, and scalability enhancements. In this blog, we’ll dive into the top 5 key enhancements that make this release transformative, solidifying position of AMD ROCm as one of the leading platforms for AI & HPC development.

1. Extending vLLM Support in ROCm 6.2 – Advancing AI Inference Capabilities of AMD Instinct Accelerators

AMD is expanding vLLM support to enhance the efficiency and scalability of AI models on AMD Instinct Accelerators. Designed for Large Language Models (LLMs), vLLM addresses key inferencing challenges such as efficient multi-GPU computation, reduced memory usage and minimized computational bottlenecks. Customers can enable various upstream vLLM features like multi-GPU execution and FP8 KV cache to tackle these challenges by following the steps provided in the ROCm documentation here. To access cutting-edge performance features, the ROCm/vLLM branch offers advanced experimental capabilities such as FP8 GEMMs and custom decode paged attention. To utilize these features, follow the steps provided here and select the rocm/vllm branch when cloning the git repository. Alternatively, these features are also available through a dedicated Docker file.

With the ROCm 6.2 release, existing and new AMD Instinct customers can confidently integrate vLLM into their AI pipelines, benefiting from the latest features for improved performance and efficiency.

2. Bitsandbytes Quantization support in ROCm – Enhancing AI Training and Inference on AMD Instinct by Boosting Memory Efficiency and Performance

The Bitsandbytes quantization library support via AMD ROCm revolutionizes AI development by significantly boosting memory efficiency and performance on AMD Instinct GPU accelerators. Utilizing 8-bit optimizers, it can reduce memory usage during AI training, enabling developers to work with larger models on limited hardware. LLM.Int8() quantization optimizes AI allowing effective deployment of LLMs on systems with less memory. Lower-bit quantization can speed up both AI training and inference, enhancing overall efficiency and productivity.

By reducing memory and computational demands, Bitsandbytes makes advanced AI capabilities accessible to a broader range of users, offers cost savings, democratizing AI development and expanding innovation opportunities. It supports scalability by enabling efficient management of larger models within existing hardware constraints while maintaining accuracy close to 32-bit precision versions.

Developers can easily integrate Bitsandbytes with ROCm for efficient AI model training and inference with reduced memory and hardware requirements on AMD Instinct GPU accelerators by following the instructions on this link.

3. New Offline Installer Creator – For Simplified ROCm Installation Experience

The ROCm Offline Installer Creator simplifies the installation process by providing a complete solution for systems without internet access or local repository mirrors. It creates a single installer file that includes all necessary dependencies, making deployment straightforward with a user-friendly GUI that allows easy selection of ROCm components and versions.

This tool reduces the complexity of managing multiple installation tools by integrating functionalities into a unified interface, enhancing efficiency and consistency. Additionally, it automates post-installation tasks such as user group management and driver handling, helping ensure correct and consistent installations.

By downloading and packaging all relevant files from AMD repository and the OS package manager, the ROCm Offline Installer Creator helps ensure installations are performed correctly and consistently, reducing the risk of errors and improving overall system stability. Ideal for systems without internet access, it also provides a simplified and efficient installation process for IT administrators, making the deployment of ROCm across various environments easier than ever before.

4. New Omnitrace and Omniperf Profiler Tools (Beta) – Revolutionizing AI & HPC Development in AMD ROCm

The new Omnitrace and Omniperf Profiler Tools (Beta version) are set to revolutionize AI and HPC development in ROCm by providing comprehensive performance analysis and a streamlined development workflow.

Omnitrace offers a holistic view of system performance across CPUs, GPUs, NICs, and network fabrics, helping developers identify and address bottlenecks, while Omniperf delivers detailed GPU kernel analysis for fine-tuning. Together, these tools optimize both application wide and compute kernel specific performance, supporting real-time performance monitoring and enabling developers to make informed decisions and adjustments throughout the development process.

By resolving performance bottlenecks, they help ensure efficient resource utilization, leading to fast AI training, inference, and HPC simulations.

5. Broader FP8 Support – Enhancing AI Inferencing with ROCm 6.2

Broad FP8 Support in ROCm can significantly improve the process of running AI models, particularly in inferencing. It helps address key challenges such as memory bottlenecks and high latency associated with higher precision formats. , allowing for larger models or batches to be handled within the same hardware constraints, thus enabling more efficient training and inference processes. Additionally, reduced precision calculations in FP8 can decrease latency involved in data transfers and computations.

ROCm 6.2 has expanded FP8 support across its ecosystem, from frameworks to libraries and more, enhancing performance and efficiency

• Transformer Engine: Adding FP8 GEMM support in PyTorch and JAX via HipBLASLt, maximizing throughput and reducing latency compared to FP16/BF16
• XLA FP8: JAX and Flax now support FP8 GEMM through XLA to improve performance.
• vLLM Integration: Further optimizes vLLM with FP8 capabilities
• FP8 RCCL: RCCL now handles FP8-specific collective operations expanding its versatility
• MIOPEN: Supports FP8-based Fused Flash attention, boosting efficiency
• Unified FP8 Header: Standardizes FP8 headers across libraries, simplifying development and integration

With ROCm 6.2, AMD continues to demonstrate its commitment to providing robust, competitive, and innovative solutions for the AI and HPC community. This release means that developers have the tools and support needed to push the boundaries of what’s possible, fostering confidence in ROCm as the open platform of choice for next-generation computational tasks. Join us in embracing these advancements and elevate your projects to unprecedented levels of performance and efficiency.

Discover the range of new features introduced in ROCm 6.2 by reviewing the release notes.

---

Source: Ronak Shah, AMD