Orion as a Service

[Delaunay]

Goals

• Simplify user setup
• Allow users to use algorithms without installing them locally
• High performance database deployment for free (instead of pickeldb)
• Gather data for warm starting
• Reduce bandwidth usage
  • The database will be accessed through a rest API, which we control
  • Because the algo and the database is inside our infra the communication between them is "free"
  • The algo is instantiated once which will simplify syncing and reduce contention

Self Setup Optimization

Here is an overview of the current process when optimizing with orion without the Orin as a service

Orion as a Service

• No change required from the user side
• Same Client API
• Simple configuration change
  • Service URL
  • User Token

Overview

Implementation

Orion Runner Rest API

experiment

Creates a new experiment, returns the experiment_id

• Request

url: /experiment
{
     'token': <usertoken>,
     **kwargs: experiment config
}

• Response

{
     'experiment_id': <experiment>,
     'error': <error>
}

suggest

Returns new trials for a given experiment

• Request

url: /suggest
{
     'token': <usertoken>,
     'experiment_id': <experiment>
     'count': <number of trials requested>
}

• Response

{
    'trials': [
         {
              'trial_id': <id>,
             'space_sample': { ... }
         }, ...
    ]
     'error': <error>
}

observe

Update a given trial's result

• Request

url: /observe
{
     'token': <usertoken>,
     'trial_id': <trial>,
     'results': [
         {'type': "", 'value': "", ....}
    ],
}

• Response

{
     'error': <error>
}

is_done

returns true if the experiment is done

• Request

url: /is_done
{
     'token': <usertoken>,
     'experiment_id': <experiment>
}

• Response

{
     'done': true/false
     'error': <error>
}

Orion Experiment Client

If the Orion Service URL is specified the clients will need to make rest API requests instead of the regular function calls.
The storage is not necessary anymore.

[bouthilx]

I like how the runner gets decoupled from the running algorithm process. I think that will be a great improvement.

In first section (experiment):

Creates a new trial, returns the experiment_id
I guess you meant creates a new experiment? The part about experiment creation is missing in the schema but I understand that you plan adding it in the REST API anyhow.

There would also need to have commands to delete experiments/trials.

In the Oríon server, we may need to have a broker of some sort to redirect calls for algorithms that have different requirements. For example we may not be able to support RoBO and Ax in the same server instance if they require incompatible torch versions.

How would you plan to make the Runner still compatible with a collocated (inside the same process) algorithm? Would the storage configuration be optional as either a URL to a service or a storage configuration for co-located algorithms? To some degree the storage and the remote service will share a large common interface, but the service will have additional functionalities like suggest or plot (this one is already implemented in our REST API).

[Delaunay]

I guess you meant creates a new experiment?

yes I corrected it.

In the Oríon server, we may need to have a broker of some sort to redirect calls for algorithms that have different requirements. For example we may not be able to support RoBO and Ax in the same server instance if they require incompatible torch versions.

Yes, I think we can ignore for it for now to keep the initial story shorter.
But we might be able to implement it as a RemoteAlgo adapter with would be a server like process that waits for function calls from the orion service. The orion service could start a RemoteAlgo using a virtualenv which would solve our issues.
Maybe the RemoteAlgo could even be on a different machine

How would you plan to make the Runner still compatible with a collocated (inside the same process) algorithm?

I am thinking we could turn the Runner into an interface and implement a RemoteRunner and rename the current Runner LocalRunner.

Would the storage configuration be optional as either a URL to a service or a storage configuration for co-located algorithms

I think I prefer having the service as its own configuration option, because the service is not going to provide all the functionalities of the storage. The idea is that if we are to provide Orion as a service we want to protect the storage from outside party;
so their access to it will be kept at a minimum.

I think we should keep the REST API for the RemoteRunner and the REST API for the dashboard separate, so modification to one does not impact the other. If some functions are similar we can use the same underlying implementation but I would like the API to stay separate.
This also will keep the RemoteRunner Rest API fairly short.

RemoteRunner could also be running under a lower security regime (token as authentication) because its API would be limited to creating/adding observation, while the dashboard will be under a higher security regime (o-auth, 2FA) which will enable the user to maybe delete trials etc...

[bouthilx]

In the Oríon server, we may need to have a broker of some sort to redirect calls for algorithms that have different requirements. For example we may not be able to support RoBO and Ax in the same server instance if they require incompatible torch versions.

Yes, I think we can ignore for it for now to keep the initial story shorter.
But we might be able to implement it as a RemoteAlgo adapter with would be a server like process that waits for function calls from the orion service. The orion service could start a RemoteAlgo using a virtualenv which would solve our issues.
Maybe the RemoteAlgo could even be on a different machine

Oh yes, we could have 2 levels with the algo process being separate from the server process. I agree this is for later, but it may be good to keep in mind for the decisions we take now.

How would you plan to make the Runner still compatible with a collocated (inside the same process) algorithm?

I am thinking we could turn the Runner into an interface and implement a RemoteRunner and rename the current Runner LocalRunner.

Would the storage configuration be optional as either a URL to a service or a storage configuration for co-located algorithms

I think I prefer having the service as its own configuration option, because the service is not going to provide all the functionalities of the storage. The idea is that if we are to provide Orion as a service we want to protect the storage from outside party;
so their access to it will be kept at a minimum.

I think we should keep the REST API for the RemoteRunner and the REST API for the dashboard separate, so modification to one does not impact the other. If some functions are similar we can use the same underlying implementation but I would like the API to stay separate.
This also will keep the RemoteRunner Rest API fairly short.

RemoteRunner could also be running under a lower security regime (token as authentication) because its API would be limited to creating/adding observation, while the dashboard will be under a higher security regime (o-auth, 2FA) which will enable the user to maybe delete trials etc...

The users will likely need to make the same kind of access to the database so it seems likely that the REST API will eventually cover the same kind of queries/udates than the storage. But I think it is a good idea to have a RemoteRunner and LocalRunner, and also have separate APIs for separate needs (RemoteRunner vs Dashboard). I'm looking forward to see this taking form. 😄

[Delaunay]

I have started looking into it with more depth, I think we actually need a RemoteClient rather than a remote runner, as the runner is mostly using the client's function, by implementing a RemoteClient we will be able to reuse the same runner.

It can't really be implemented as a storage because it is not the kind of granularity I would want to expose to clients & that would mean the Experiment instantiate a bunch of things we actually are not going to use.

The remote client should provide a high enough level of abstraction and we will be able to shortcut the creation of algo & the search space which is not necessary when using the REST API.

[Delaunay]

We also need a way to filter the database per username/token.
Currently the users connects to the database using a username and a password,
in the service version it will be a the service own username and password and it will need to be able to insert things to the db for each users