Optimizers

The optimizers module contains the core optimization algorithms for hyperparameter tuning.

Overview

FCVOpt provides two main optimizer classes:

• FCVOpt: The main fractional cross-validation optimizer using hierarchical Gaussian processes

• BayesOpt: Base Bayesian optimization framework for standard (full) cross-validation

API Reference

BayesOpt Base Class

class fcvopt.optimizers.BayesOpt(obj, config, minimize=True, acq_function='EI', acq_function_options=None, batch_acquisition=False, acquisition_q=1, verbose=1, n_jobs=1, seed=None, tracking_uri=None, tracking_dir=None, experiment=None, run_name=None, model_checkpoint_freq=1)

Bases: object

Bayesian Optimization for optimizing a given objective function.

This class implements a Bayesian optimization loop that iteratively fits a Gaussian Process model to the objective function and proposes new configurations to evaluate via an acquisition function. The optimizer supports MLflow tracking for experiment management and can be used with various acquisition functions and batch optimization.

Parameters:

• obj (Callable) – The objective function mapping a configuration dict to a scalar value.

• config (ConfigurationSpace) – The search space configuration.

• minimize (bool) – If True, minimizes the objective; otherwise maximizes it. Defaults to True.

• acq_function (str) – Acquisition function to use. One of {‘EI’, ‘LCB’, ‘KG’}. Defaults to ‘EI’.

• acq_function_options (Optional[Dict]) – Additional keyword arguments passed to the acquisition function constructor. Defaults to None.

• batch_acquisition (bool) – If True, a batch of configurations (the number specified by acquisition_q) is selected for each iteration. Defaults to False.

• acquisition_q (int) – Number of points in each proposed batch when batch_acquisition is True. Defaults to 1.

• verbose (int) – Verbosity level to print to console; 0=no output, 1=summary at end, 2=detailed per-iteration log. Defaults to 1.

• n_jobs (int) – Number of parallel jobs for objective evaluation and model fitting. Use -1 to utilize all available CPU cores. Defaults to 1.

• seed (Optional[int]) – Random seed for reproducibility. Defaults to None.

• tracking_uri (Optional[str]) – MLflow tracking URI (e.g., “file:/abs/path” or “http://host:5000”). Defaults to None.

• tracking_dir (Optional[str]) – Directory for MLflow tracking (e.g., “./results”). Gets converted to absolute file URI. Defaults to None.

• experiment (Optional[str]) – MLflow experiment name. Defaults to the class name (e.g., "BayesOpt" or "FCVOpt" for subclasses).

• run_name (Optional[str]) – MLflow run name. Defaults to a timestamp string of the form "run_YYYYMMDD_HHMMSS".

• model_checkpoint_freq (int) – Model checkpointing frequency. Save checkpoint every N iterations. 1 = every iteration, 5 = every 5 iterations. Always saves final iteration. Defaults to 1.

Examples

Basic usage:

>>> from fcvopt.optimizers import BayesOpt
>>> from fcvopt.configspace import ConfigurationSpace
>>>
>>> # Define objective function
>>> def objective(config):
...     return config['x']**2 + config['y']**2
>>>
>>> # Define search space
>>> import ConfigSpace as CS
>>> cs = ConfigurationSpace()
>>> cs.add(CS.Float('x', bounds=(-5.0, 5.0)))
>>> cs.add(CS.Float('y', bounds=(-5.0, 5.0)))
>>>
>>> # Run optimization
>>> bo = BayesOpt(obj=objective, config=cs)
>>> best_config = bo.run(n_iter=10)
>>> print(f"Best config: {best_config}")
>>> print(f"Best value: {bo.curr_f_inc_obs}")

Continuing optimization:

>>> # Continue with more iterations
>>> best_config = bo.run(n_iter=5)  # MLflow run will be reactivated

end_run()

End the MLflow run and log final metrics and best configuration.

Logs final_f_inc_obs, final_f_inc_est, and final_total_evals as metrics on the parent run, writes the best configuration to a best_config.json artifact, and terminates the run with status FINISHED.

Note

This method is automatically called when using the optimizer as a context manager. If not using a context manager, call this explicitly when done to avoid leaving the MLflow run in an active state.

Examples

>>> bo = BayesOpt(obj=objective, config=cs)
>>> bo.run(n_iter=10)
>>> bo.run(n_iter=5)  # continuation
>>> bo.end_run()  # properly closes the MLflow run

get_optimization_results()

Retrieve detailed optimization results for all iterations.

Returns a comprehensive summary of the optimization process, including incumbent configurations, observed values, and model predictions with uncertainty estimates for each iteration.

Returns:

List of dictionaries, one per iteration, with keys:

• ’iteration’ (int): Iteration number (0-based)

• ’incumbent_config’ (dict): Best configuration found so far

• ’observed_value’ (float): Actual observed objective value at incumbent

• ’predicted_value’ (float): Model’s predicted value at incumbent

• ’predicted_std’ (float): Model’s prediction uncertainty (standard deviation)

Return type:

List[Dict[str, Any]]

Raises:

RuntimeError – If no optimization has been performed yet or model is unavailable.

Examples

>>> bo = BayesOpt(obj=objective, config=config_space)
>>> bo.run(n_iter=5)
>>> results = bo.get_optimization_results()
>>>
>>> # Access results for each iteration
>>> for result in results:
...     print(f"Iteration {result['iteration']}:")
...     print(f"  Best config: {result['incumbent_config']}")
...     print(f"  Observed: {result['observed_value']:.4f}")
...     print(f"  Predicted: {result['predicted_value']:.4f} ± {result['predicted_std']:.4f}")
...
>>> # Get final result
>>> final_result = results[-1]
>>> best_config = final_result['incumbent_config']
>>> best_value = final_result['observed_value']

property observed_values: ndarray

All objective values observed during optimization, in evaluation order.

Returns:

1-D array of observed objective values, shape (n_evals,).

Return type:

np.ndarray

Raises:

RuntimeError – If no optimization has been performed yet.

optimize(n_trials, n_init=None)

Run Bayesian optimization for a specified number of trials in this call.

This method treats n_trials as the number of evaluations to perform in this specific call, making it suitable for both initial runs and continuation runs from restored optimizers.

For initial runs, it evaluates n_trials total configurations: n_init random initializations followed by n_trials - n_init acquisition-guided evaluations.

For continuation runs (when optimization has already been started), it performs exactly n_trials additional evaluations via acquisition, ignoring the n_init parameter.

Parameters:

• n_trials (int) – Number of objective evaluations to perform in this call. For initial runs, includes both random and acquisition-guided trials. For continuation runs, specifies additional acquisition-guided trials. Must be positive.

• n_init (int, optional) – Number of initial random configurations for the first call only. If None on initial run, defaults to len(self.config) + 1. Ignored and warned about for continuation runs.

Returns:

Best configuration found so far.

Return type:

Configuration

Raises:

ValueError – If n_trials is not positive or invalid n_init for initial runs.

Examples

>>> # Initial run: 15 total evaluations (3 random + 12 acquisitions)
>>> best_config = bo.optimize(n_trials=15, n_init=3)
>>> len(bo.train_confs)  # Should be 15

>>> # Continuation: 10 more evaluations (all acquisitions)
>>> best_config = bo.optimize(n_trials=10)  # n_init ignored
>>> len(bo.train_confs)  # Should be 25 (15 + 10)

>>> # Another continuation: 5 more evaluations
>>> best_config = bo.optimize(n_trials=5)

classmethod restore_from_mlflow(obj, run_id=None, experiment_name=None, run_name=None, tracking_uri=None, tracking_dir=None, model_checkpoint='latest')

Restore a BayesOpt instance from MLflow run artifacts to continue optimization.

This method reconstructs the complete state of a previous optimization run from MLflow logs, allowing you to seamlessly continue optimization where you left off. The restored instance will use the same experiment and run configuration to maintain continuity.

Parameters:

• obj (Callable) – Objective function for the restored optimizer.

• run_id (Optional[str]) – Specific MLflow run ID to restore from. If provided, takes precedence over experiment_name/run_name.

• experiment_name (Optional[str]) – Name of the MLflow experiment to search for the run. Used with run_name to identify the run to restore.

• run_name (Optional[str]) – Name of the specific run to restore within the experiment. Used with experiment_name to identify the run to restore.

• tracking_uri (Optional[str]) – MLflow tracking URI (e.g., “file:/path” or “http://host:5000”). Cannot be used with tracking_dir.

• tracking_dir (Optional[str]) – Directory for MLflow tracking (e.g., “./results”). Gets converted to absolute file URI. Cannot be used with tracking_uri.

• model_checkpoint (str) – Model checkpoint to load. Either “latest” for the most recent checkpoint, or a specific filename.

Returns:

Restored optimizer instance ready to continue optimization.

Return type:

BayesOpt

Raises:

• RuntimeError – If run cannot be found or required artifacts are missing.

• ValueError – If neither run_id nor (experiment_name, run_name) is provided.

Examples

>>> # Restore by run ID with tracking_uri
>>> bo = BayesOpt.restore_from_mlflow(
...     obj=objective_function,
...     run_id="abc123",
...     tracking_uri="file:/path/to/mlruns"
... )
>>> results = bo.run(n_iter=5)  # Continue optimization

>>> # Restore by experiment and run name with tracking_dir
>>> bo = BayesOpt.restore_from_mlflow(
...     obj=objective_function,
...     experiment_name="BayesOpt",
...     run_name="optimization_run_1",
...     tracking_dir="./my_results"
... )
>>> results = bo.run(n_iter=10)  # Continue optimization

Note

The restored instance will: - Reconstruct all training data and state from MLflow artifacts - Reuse the same experiment and run configuration - Continue with the same MLflow run for seamless tracking - Load the model checkpoint for warm-starting

run(n_iter, n_init=None)

Run BO for exactly n_iter acquisition steps.

If this is the first call, initializes the optimizer with n_init random points. If called again on the same instance, continues optimization from where it left off.

The MLflow run remains active after this method returns, allowing for continuation runs. Call end_run() when finished to properly close the MLflow run, or use the optimizer as a context manager for automatic cleanup.

Parameters:

• n_iter (int) – Number of acquisition iterations to run.

• n_init (int, optional) – Number of initial random points. Only used on first call. If None on first call, defaults to len(self.config) + 1. Ignored on subsequent calls.

Returns:

Best configuration found so far.

Return type:

Configuration

Examples

Using context manager (recommended):

>>> with BayesOpt(obj=objective, config=cs) as bo:
...     bo.run(n_iter=10)
...     bo.run(n_iter=5)  # continuation
... # MLflow run automatically closed

Manual run management:

>>> bo = BayesOpt(obj=objective, config=cs)
>>> bo.run(n_iter=10)
>>> bo.run(n_iter=5)  # continuation
>>> bo.end_run()  # must call to close MLflow run

FCVOpt Class

class fcvopt.optimizers.FCVOpt(obj, config, n_folds, n_repeats=1, fold_selection_criterion='variance_reduction', fold_initialization='random', minimize=True, acq_function='LCB', **kwargs)

Bases: BayesOpt

Fractional cross-validation for hyperparameter optimization.

Implements the fractional CV approach from “Fractional cross-validation for optimizing hyperparameters of supervised learning algorithms.” This method uses a hierarchical Gaussian process (HGP) model to exploit the correlation of single-fold out-of-sample errors across hyperparameter configurations, enabling efficient Bayesian optimization with only a fraction of the K folds evaluated per configuration.

Rather than performing full K-fold CV at every candidate point, FCVOpt:

• Employs a hierarchical GP that models both fold-wise and hyperparameter-wise covariance structures.

• Evaluates only one fold (or a small subset of folds) for most configurations, drastically reducing computation.

• Selects folds adaptively based on variance reduction or random sampling.

Parameters:

• obj (Callable) – Objective function that accepts a hyperparameter configuration dict and a fold_idxs keyword argument (list of int), and returns a scalar cross-validation error for the requested fold(s).

• config (ConfigurationSpace) – Hyperparameter search space.

• n_folds (int) – Number of folds in standard K-fold cross-validation.

• n_repeats (int) – Number of independent repeats of K-fold CV; used to expand the fold index set. For example, n_folds=5, n_repeats=2 gives valid fold indices 0–9. Defaults to 1.

• fold_selection_criterion (str) –

  Strategy for selecting the next fold to evaluate:

  • 'variance_reduction': choose the fold that minimizes predictive variance via fcvopt.models.HGP._fold_selection_metric().

  • 'random': choose folds uniformly at random.

  Defaults to 'variance_reduction'.

• fold_initialization (str) –

  Strategy for assigning folds in the initial random sample of configurations:

  • 'random': sample folds uniformly at random.

  • 'stratified': use stratified sampling across folds via fcvopt.util.samplers.stratified_sample().

  • 'two_folds': randomly pick two distinct folds and split the initial samples evenly between them.

  Defaults to 'random'.

• minimize (bool) – If True, minimizes the cross-validation error; otherwise maximizes. Defaults to True.

• acq_function (str) – Acquisition function to use. One of {'LCB', 'KG'}. Note that 'EI' is not supported for FCVOpt and will raise a RuntimeError. Defaults to 'LCB'.

• acq_function_options – Additional keyword arguments passed to the acquisition function constructor. Defaults to None.

• batch_acquisition – If True, a batch of acquisition_q configurations is proposed at each iteration (using qLCB or qKG). Each candidate in the batch is assigned its own fold index via fold_selection_criterion. Defaults to False.

• acquisition_q – Number of points in each proposed batch when batch_acquisition is True. Defaults to 1.

• verbose – Verbosity level; 0=no output, 1=summary at end, 2=detailed per-iteration log. Defaults to 1.

• n_jobs – Number of parallel jobs for objective evaluation and GP hyperparameter fitting. Use -1 to utilise all available CPU cores. Defaults to 1.

• seed – Random seed for reproducibility. Defaults to None.

• tracking_uri – MLflow tracking URI (e.g., "file:/abs/path" or "http://host:5000"). Mutually exclusive with tracking_dir. Defaults to None (uses ./mlruns).

• tracking_dir – Directory for MLflow tracking (e.g., "./results"). Gets converted to an absolute file: URI. Mutually exclusive with tracking_uri. Defaults to None.

• experiment – MLflow experiment name. Defaults to "FCVOpt".

• run_name – MLflow run name. Defaults to a timestamp string of the form "run_YYYYMMDD_HHMMSS".

• model_checkpoint_freq – Save a GP model checkpoint every N iterations. 1 saves every iteration; the final iteration is always saved. Defaults to 1.

Examples

Basic usage with a scikit-learn cross-validation objective:

>>> from fcvopt.optimizers import FCVOpt
>>> from fcvopt.crossvalidation import SklearnCVObj
>>> from fcvopt.configspace import ConfigurationSpace
>>> from ConfigSpace import Float
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.metrics import log_loss
>>>
>>> config = ConfigurationSpace(seed=42)
>>> config.add(Float('max_features', bounds=(0.1, 1.0)))
>>> config.add(Float('min_samples_leaf', bounds=(0.01, 0.1)))
>>>
>>> cv_obj = SklearnCVObj(
...     estimator=RandomForestClassifier(n_estimators=100),
...     X=X_train, y=y_train,
...     task='classification',
...     loss_metric=log_loss,
...     n_splits=5
... )
>>>
>>> optimizer = FCVOpt(
...     obj=cv_obj, config=config, n_folds=5,
...     tracking_dir='./results'
... )
>>> best_config = optimizer.run(n_iter=20, n_init=5)
>>> optimizer.end_run()

References

• fcvopt.models.HGP

classmethod restore_from_mlflow(obj, run_id=None, experiment_name=None, run_name=None, tracking_uri=None, tracking_dir=None, model_checkpoint='latest', n_folds=5, n_repeats=1, fold_selection_criterion='variance_reduction', fold_initialization='random', **kwargs)

Restore an FCVOpt instance from MLflow with fold information.

This extends the base class restoration to also rebuild the training fold information from logged evaluations.

Parameters:

• obj (Callable) – Objective function to use for the restored optimizer.

• run_id (Optional[str]) – MLflow run ID to restore from.

• experiment_name (Optional[str]) – MLflow experiment name (used with run_name).

• run_name (Optional[str]) – MLflow run name (used with experiment_name).

• tracking_uri (Optional[str]) – MLflow tracking URI.

• tracking_dir (Optional[str]) – MLflow tracking directory (alternative to tracking_uri).

• model_checkpoint (str) – Model checkpoint to load.

• n_folds (int) – Number of folds in cross-validation.

• n_repeats (int) – Number of independent repeats of K-fold CV.

• fold_selection_criterion (str) – Strategy for selecting the next fold to evaluate.

• fold_initialization (str) – Strategy for assigning folds in the initial random sample.

• **kwargs – Additional keyword arguments for BayesOpt.

Returns:

Restored optimizer instance with fold information.

Return type:

FCVOpt