Learn how the Adam optimizer powers efficient neural network training with adaptive learning rates, momentum, and real-world applications in AI.
Adam (Adaptive Moment Estimation) is a widely adopted optimization algorithm used extensively in deep learning (DL) and machine learning (ML). It's designed to efficiently update network weights during the training process by adapting the learning rate for each parameter individually. Introduced in the paper "Adam: A Method for Stochastic Optimization" by Diederik P. Kingma and Jimmy Ba, Adam combines the advantages of two other popular optimization techniques: AdaGrad (Adaptive Gradient Algorithm) and RMSprop (Root Mean Square Propagation). This combination makes it particularly effective for training large neural networks with numerous parameters and complex datasets.
Adam calculates adaptive learning rates for each parameter based on estimates of the first and second moments of the gradients. Essentially, it keeps track of an exponentially decaying average of past gradients (similar to momentum) and an exponentially decaying average of past squared gradients (similar to AdaGrad/RMSprop).
Compared to simpler algorithms like Stochastic Gradient Descent (SGD), which uses a single, fixed learning rate (or one that decays according to a schedule), Adam's per-parameter adaptation often allows for quicker progress in finding a good solution, especially with complex loss landscapes.
Adam is popular for several reasons:
Adam is a go-to optimizer for many state-of-the-art models:
In computer vision, Adam is frequently used to train deep Convolutional Neural Networks (CNNs) for tasks like image classification, object detection, and image segmentation. For instance, training an Ultralytics YOLO model for detecting objects in images (like those in the COCO dataset) or performing instance segmentation can leverage Adam for efficient convergence during the training phase. It's also applied in medical image analysis for tasks like tumor detection.
Adam is a standard optimizer for training large language models (LLMs) like BERT and GPT variants. When training models for tasks such as machine translation, text summarization, or sentiment analysis, Adam helps efficiently navigate the complex loss function landscape associated with these large (transformer-based) models.
Within the Ultralytics ecosystem, Adam and its variant AdamW (Adam with decoupled weight decay) are available optimizers for training Ultralytics YOLO models. Leveraging Adam's adaptive learning rates can accelerate convergence during the training of object detection, instance segmentation, or pose estimation models like YOLO11 or YOLOv10. While SGD is often the default and recommended optimizer for some YOLO models due to potentially better final generalization (avoiding overfitting), Adam provides a robust alternative, particularly useful in certain scenarios or during initial experimentation and model evaluation. You can easily configure the optimizer and other training settings. Tools like Ultralytics HUB streamline the process, allowing users to train models using various optimizers, including Adam, either locally or via cloud training. Frameworks like PyTorch and TensorFlow provide standard implementations of Adam, which are utilized within the Ultralytics framework. For further performance improvements, consider techniques like knowledge distillation or exploring different model architectures.
EN
ZH
KO
JA
RU
DE
FR
ES
PT
IT
AR
TR
VI
