Allows detailed optimization settings to be performed in the UI.
See below for the meaning of each setting.

Useful resources for understanding each algorithm settings

TPE (Tree-structured Parzen Estimator)

TPE fits one Gaussian Mixture Model (GMM) l(x)to the set of parameter values associated with the best objective values, and another GMM g(x)to the remaining parameter values. It chooses the parameter value xthat maximizes the ratio l(x)/g(x).

API Reference

• optuna.samplers.TPESampler

Referenced Papers

• Algorithms for Hyper-Parameter Optimization
• Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures
• Multiobjective tree-structured parzen estimator for computationally expensive optimization problems
• Multiobjective Tree-Structured Parzen Estimator

About constraints

• A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible.
• https://github.com/optuna/optuna/pull/3506

How many should the Number of startup trials be?

• Bayesian optimization first performs random sampling to create a surrogate model. This is the setting for how many trials random sampling is performed.
• The original paper recommends “number of variables” * 11-1.

BoTorch (Gaussian Process)

API Reference

• optuna.integration.BoTorchSampler

About constrains

• A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible.
• more detail Constraints · BoTorch

CMA-ES (Covariance Matrix Adaptation Evolution Strategy)

API Reference

• optuna.samplers.CmaEsSampler

• Optuna uses the following libraries

  • https://github.com/CyberAgentAILab/cmaes

NSGA-II (Non-dominated Sorting Genetic Algorithm)

Multi-objective sampler using the NSGA-II algorithm.

API Reference

• optuna.samplers.NSGAIISampler

Referenced Paper

• A fast and elitist multiobjective genetic algorithm: NSGA-II

What difference of corossover algorithm

• You can select several variants of the crossover option from uniform (default), blxalpha, sbx, vsbx, undx, and spx.
• detail
  • https://github.com/optuna/optuna/pull/2903

About constraints

• A value strictly larger than 0 means that a constraints is violated. A value equal to or smaller than 0 is considered feasible.
• https://github.com/optuna/optuna/pull/3506

QMC (Quasi-Monte Carlo)

A Quasi Monte Carlo Sampler that generates low-discrepancy sequences.

API Reference

• optuna.samplers.QMCSampler

This process uses scipy inside optuna. If you want to know the detailed process, please refer to the following scipy documentation.

• scipy.stats.qmc

Related to PullRequest

• https://github.com/optuna/optuna/pull/2423

When should QMC be selected instead of random?

Random sampling, as the name implies, samples randomly. This means that the spacing between the sampled points is not guaranteed to be far apart, and it is possible to sample a point right next to a point that has already been sampled.
On the other hand, quasi-Monte Carlo allows for random sampling with some spacing. For example, if you want to check the entire solution space with a small sample, you may be able to do so faster than with random.