Mixed Precision

How Mixed Precision Works

The core idea behind mixed precision is to leverage the speed and memory benefits of lower-precision formats while mitigating potential numerical stability issues. A common approach involves these steps:

1. Maintain Master Weights in FP32: A primary copy of the model's weights is kept in the standard FP32 format to ensure high precision for weight updates.
2. Use FP16/BF16 for Computations: During the training loop, the FP32 weights are cast to FP16 or BF16 for the forward and backward passes. Computations using these lower-precision formats are significantly faster on modern hardware like NVIDIA GPUs equipped with Tensor Cores, which are specifically designed to accelerate matrix multiplications at lower precisions.
3. Loss Scaling: When using FP16, the range of representable numbers is much smaller than FP32. This can cause small gradient values calculated during backpropagation to become zero (underflow), hindering learning. To prevent this, the loss value is scaled up before backpropagation, effectively scaling up the gradients into a range representable by FP16. Before the weight update, these gradients are scaled back down. BF16, with its wider dynamic range similar to FP32 but lower precision, often avoids the need for loss scaling.
4. Update Master Weights: The computed gradients (scaled back down if loss scaling was used) are used to update the master copy of the weights, which remain in FP32.

This careful balance allows models to train faster and use less GPU memory.

Benefits of Mixed Precision

• Faster Training: Lower-precision computations (FP16/BF16) execute much faster on compatible hardware, significantly reducing the time required for each training epoch. This allows for quicker iteration and experimentation.
• Reduced Memory Consumption: FP16/BF16 values require half the memory of FP32 values. This reduction applies to activations stored during the forward pass and gradients computed during the backward pass. Lower memory usage enables training larger models or using larger batch sizes, which can improve model performance and training stability.
• Improved Efficiency: The combination of faster computation and lower memory bandwidth requirements leads to more efficient use of hardware resources, potentially lowering training costs for cloud computing or on-premise clusters.

Mixed Precision vs. Related Concepts

• Full Precision (FP32): Traditional training uses FP32 for all storage and computation. It's generally more stable numerically but slower and more memory-intensive than mixed precision.
• Half Precision (FP16/BF16): Using only FP16 or BF16 throughout training can lead to significant numerical instability (especially FP16 without techniques like loss scaling) and potential loss of accuracy. Mixed precision is a more robust approach that combines FP32 and FP16/BF16.
• Model Quantization: This typically refers to converting model weights and/or activations to even lower precision formats, such as 8-bit integers (INT8), primarily for optimizing inference speed and efficiency, especially on edge devices. While sometimes used during training (Quantization-Aware Training), it's distinct from the typical FP32/FP16 mixed precision used during standard training phases.

Applications and Examples

Mixed precision is widely adopted in training large-scale machine learning (ML) models.

1. Training Large Language Models (LLMs): Models like GPT-3, BERT, and T5 have billions of parameters. Training them using only FP32 would require prohibitive amounts of GPU memory and time. Mixed precision makes training such foundation models feasible by significantly reducing memory needs and accelerating computations. Frameworks like PyTorch and TensorFlow provide built-in support for mixed precision training.
2. Accelerating Computer Vision Models: In computer vision (CV), mixed precision speeds up the training of complex models like Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) used for tasks like object detection, image segmentation, and image classification. For instance, Ultralytics YOLO models, including the latest Ultralytics YOLO11, can leverage mixed precision during training for faster convergence and efficient resource utilization, as noted in our model training tips and model comparisons. This allows users to train high-performance models more quickly on datasets like COCO. The faster training cycles facilitate quicker hyperparameter tuning and model development within platforms like Ultralytics HUB. Mixed precision can also be used during inference to speed up deployment, particularly when exporting models to formats like TensorRT which heavily optimize for lower precisions.