Knowledge base

[bouthilx]

How should we do it? :)

[Delaunay]

Goals

• Knowledge base: dataset of past hyperparameter optimization
• Leverage the knowledge base to implement warm start algorithm
  • Given previous experience, it will sample better hyperparameter sets

Challenges

• We need to extract/save all relevant metadata about the experiment.

  • Space config (initial dimension priors)
  • Model specifics (depth, activation, layers, arch name)
  • Optimizers
  • LRScheduler
  • Dataset
  • Task (Resnet can be used for both image classification and object detection)
  • All the trials

• Training the warm start algo

• warm start algo queries to the knowledge base

Warm Starting

The knowledge base goal is mainly to feed the warm start algorithms we might implement.
To make sure the knowledge base answers all of the requirements of warm start algorithms we need to identify the
requirements of warm starting.

Resemblance

The base case of warm starting would be to optimize an experiment Y that is "close" another experiment X.

1. From Y identify the closest experiment X
2. From X, we can build a new search space with dimensions that are centered on the best parameters found during the previous optimization process, all new dimensions could be ~normal(x_best, d) with d a measure of distance between X and Y. Maybe d could be tweaked during the optimization process as confidence grows/shrink
3. Sample from the new search space

This algo is going to need:

1. Ways to compare to experiment |X_config - Y_config| and |X_meta - Y_meta|
   this is O(n) with n the number of experiment)
   We could reduce the number of experiment n by filtering against a set of 'dealbreaker' config/metadata values
   i.e only check RL experiments/classification/image based
2. Retrieve all the trials of X to build the new dimension priors
   This is already implemented

We can do 1. with something like MongoDB aggregation pipeline.
it would be fairly efficient, the computation would be done entirely in the db itself and would not use much bandwidth.

Model

Train a model that takes the config & metadata as input what would be the optimal priors.
This would require us to dump all the data in a efficient format for training.