This repository has been archived by the owner on Jul 31, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODO
45 lines (43 loc) · 2.44 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
besides the stuff mentioned in comments throughout the code,
a few things would be nice to have or somewhat important:
- dump <- input-hash association *signatures* (maybe use a Message Auth. Code)
- maybe more restricted containerization?
- build server cluster support (distcc-like)
- maybe parse ninja files, and the split compile-build-rules
into two to run preprocessing in a separate step
(this approach is similar to what distcc uses to decouple from input)
- proper cancellation of builds
- proper dump verification and repair (start repair as soon as detecting corruption)
- S3 support
- reduce the amount of async channels in the server
- regularly print progress for downloads
- store (on-disk data) garbage collection
- introduce a clonable struct which contains all reference-counted objects given to
async schedule(...) in yzix-server, to make the argument list of it smaller
- server: move log-coalesce of identical-inhash nodes into inhxcl_setup to
prevent race-conditions or loss-of-output-notification for
"last nodes" (with no dependees)
- in order to make distributed workflows both simple and reasonable reliable,
use etcd as a unordered message queue (for build requests/schedule ops)
and as a state hashmap.
structure:
- /yzix/active/{inhash}
contains a serialization of a build request, which,
with the prev/current design corresponds to a `build_graph::Node`.
it gains an additional field `state` (equiv. to current `rest.output`),
which contains one of the following values:
- `NotScheduled{ data, pushtime }` (unclaimed work)
- `Scheduled{ data, host }` (claimed work)
- `Success(outhash)` (done work)
- `Failure(errmsg)` (failed work)
Using an etcd transaction, the server updates the state from `NotScheduled`
to `Scheduled` (and nothing else, if the item is not in `NotScheduled` state,
no server can additionally claim it, as it is already claimed).
TODO: figure out how to check if the server doing the work hasn't crashed.
The `data` parameter contains all the `WorkItem` information, which is necessary
to execute the request.
TODO: add a `system` parameter to the `WorkItem` information.
The `pushtime` parameter contains the time at which the item was submitted,
and is used to sort requests when scheduling.
TODO: figure out how to make scheduling preference and such dependent on the
amount of data that the server needs to download in order to start working.