Review of Mixed-Precision Optimization Methods for Tensor Computations on GPUs

Khushboo Kumari Yadav

Citation: Khushboo Kumari Yadav, "Review of Mixed-Precision Optimization Methods for Tensor Computations on GPUs", Universal Library of Engineering Technology, Volume 02, Issue 04.

Copyright: This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

The article presents a comprehensive analysis of existing methods for optimizing tensor computations on graphics processing units (GPU) when using mixed-precision modes. The relevance of the study is determined by the computational complexity of modern neural network architectures and the need to ensure higher energy efficiency of computations while maintaining the required level of numerical accuracy. The scientific novelty of the study lies in the consistent systematization of approaches to the use of FP8 and BF16 formats on NVIDIA Ampere and Hopper architectures, as well as in the formulation of an adaptive strategy for selecting data representation precision depending on the nature of the computational workload. Within the framework of the study, architectural and hardware features of tensor cores are considered, and algorithms for dynamic loss scaling and stochastic rounding are analyzed, which determine the behavior of numerical errors under reduced precision. Particular emphasis in the work is placed on ensuring numerical stability during quantization of transformer models, where the combination of deep architecture and long chains of matrix operations makes the system sensitive to error accumulation. The aim of the study is to identify preferable combinations of data representation formats for various classes of tensor operations. To achieve this aim, methods of comparative analysis of the existing literature and theoretical modeling are used. The final part of the article presents a hybrid-precision scheme oriented toward practical application in high-performance computing systems and intended for specialists engaged in the development and study of deep learning methods.


Keywords: FP8 Quantization, GPU Optimization, High-Performance Computing? Mixed-Precision, Tensor Cores.

Download doi https://doi.org/10.70315/uloap.ulete.2025.0204012