Covariance Matrix Adaptation Evolution Strategy

CMA-ES is a stochastic, derivative-free optimization algorithm for single-objective continuous optimization. It belongs to the family of Evolution Strategies (ES) and is particularly effective when:

variables are correlated (the objective is non-separable), or
dimensions have very different scales (ill-conditioned landscape)

Unlike genetic algorithms such as NSGA-II, CMA-ES does not use crossover. Instead, it samples new candidates from a multivariate normal distribution and adapts that distribution based on the history of successful search steps.

How CMA-ES Works

Optimization by CMA-ES is performed by repeatedly executing STEP2 to STEP4.

STEP1: Initialization

Three quantities are initialized:

Mean m: the starting point (typically the center of the search range)
Step size σ: controls the overall scale of the search
Covariance matrix C = I: starts as the identity matrix (isotropic, no correlation between variables)

STEP2: Sampling

A population of λ candidate solutions is drawn from a multivariate normal distribution:

x_i ~ m + σ · N(0, C),  i = 1, ..., λ

Each candidate represents one set of variable values to evaluate.

STEP3: Selection and Mean Update

Candidates are ranked by their objective value. The best μ candidates (μ < λ) are selected, and the mean is updated as a weighted centroid of those candidates:

m ← Σ w_i · x_i   (weighted average of the top μ)

STEP4: Covariance Matrix and Step Size Adaptation

This is the defining step of CMA-ES:

Covariance matrix update: C is adapted to reflect the direction and magnitude of successful steps. Over iterations, C learns the correlations between variables and reshapes the sampling ellipsoid to match the problem's geometry.
Step size update (σ): increased when progress is steady (the search moves consistently in one direction) and decreased when oscillating. This keeps exploration efficient throughout the run.

STEP2 through STEP4 are repeated until the number of trials is reached.

Image of behavior

The animation below shows how the sampling distribution (ellipse) gradually rotates and scales to align with the objective function's contours.

CMA-ES behavior

Key Properties

Property	Detail
Objective type	Single-objective only
Variable type	Continuous
Invariance	Rotation- and scale-invariant
Recommended number of variables	Up to ~100

Rotation invariance means CMA-ES performs equally well regardless of how the coordinate axes are oriented relative to the problem. This is a significant advantage over methods that assume axis-aligned search directions (such as independent random or QMC sampling).

When to Use CMA-ES

Single-objective, continuous optimization
Variables are correlated or the landscape is not axis-aligned
The number of variables is moderate (roughly 2–100)
Bayesian methods (TPE, GP) are not converging well

CMA-ES