PackageRebuilder

Note: This software is in a very active development and is subject to major changes due to in progress work on Qubes, Debian and Fedora side and also dependent tools debrebuild and rpmreproduce. It may not work as expected until a proper stable release is made. For more information, don't hesitate to contact me.

Architecture

The current design of PackageRebuilder is based on individual tasks orchestrated with the help of celery engine (see https://docs.celeryproject.org/en/stable/). celery uses a broker to receive and distribute task in specific queues for which services getter, rebuilder, uploader and attester are connected to. All the task results are stored in backend.

                         .-----------.     .-----------.
                         | reporter  |    .| getter    |
                         '-----.-----'   / '-----------'
        .---------.            |        /
        | broker  |----.       |       '      .-----------.
        '---------'     '------'-------. .----| uploader  |
                        |              |'     '-----------'
                        | orchestrator |
                        |              |.     .-----------.
        .---------.     .--------------. '----| attester  |
        | backend |----'               '      '-----------'
        '---------'                     \
                                         \ .-----------.
                                          '| rebuilder |
                                           '-----------'

The chosen broker here for celery is redis and the backend is mongodb as database engine.

Each service is doing only tasks defined in separate queues defined as follows:

SERVICE	TASK/QUEUE
getter	get
rebuilder	rebuild
attester	attest
reporter	report
uploader	upload

                                                       .-----------.
                              .----------------------->| attester  |
                              |                        '-----------'
                              |                              |
                        build passed                  in-toto metadata
                              |                              |
                              |                              v
      .--------.        .-----------.                  .-----------.        .-----------.
      | getter |------->| rebuilder |---build failed-->| reporter  |------->| uploader  |
      '--------'        '-----------'                  '-----------'        '-----------'
          ^                                                                       |
          |                                                                     rsync
          |                                                                       |
          |                                                                       v
    *************                                                       *********************
    * buildinfo *                                                       * remote repository *
    *************                                                       *********************

The getter service is responsible to get the latest buildinfo on Qubes OS or Debian (soon Fedora) repositories and to add new rebuild tasks. Once a rebuilder has finished it adds a new attest task for attester if the build passed else it adds report task directly. Then, attester will collect rebuild artifacts and generate signed in-toto metadata. Once metadata are created, the reporter collects log and clean artifacts. Finally, upload task is triggered to upload using rsync, in-toto metadata, logs and statistics on a remote repository. The purpose of the remote repository is to serve in-toto metadata.

Either on success or failure, a task result containing useful information about the build is created in backend. Notably, it contains all the information about a build, its status and the number of retries. In practice, you will only scale rebuilder service.

There exist two side services for celery which are beat and flower. The former holds the periodic task scheduling and the latter is useful for monitoring celery (see flower). Notably, one can setup graphana integration. You can access flower interface at http://localhost:5556.

uploader service is also responsible to export all the rebuild task results and to generate some graphical stats (e.g. results).

PackageRebuilder: the machinery

This sections helps to configure PackageRebuilder on any environment (not Qubes OS specific) assuming it satisfies the following dependencies requirements. The current setup is done using Docker for which it can be replaced using a more classic or Qubes configuration with virtual machines for each service. This will be detailed in a near future. The use of celery allows extending and interacting with the broker easily. For example, one can add webhook trigger with or in place of periodic scheduling.

TODO: Add webhook setup to trigger get task.

Installation

We recommend installing PackageRebuilder in a Debian or CentOS based distribution which still supports Docker. In a future, we plan to test it with podman. Here we give installation steps for a Debian distribution.

A remote repository is needed to upload logs, statistics and in-toto metadata. If you don't plan to server in-toto repository and only rebuild packages to get statistics, you don't have to prepare a GPG key to be used for creating and signing metadata. In such case, attester service will simply skip the in-toto metadata creation.

Note: On the hosting machine, most of the commands as to be run as root.

Dependencies

Ensure to have docker and docker-compose installed:

$ apt install docker docker-compose

Enable and start docker:

$ systemctl enable docker
$ systemctl start docker

Clone package-rebuilder repository into /opt as rebuilder:

$ git clone https://github.com/fepitre/package-rebuilder /opt/rebuilder

Copy rebuilder systemd service and reload:

$ cp /opt/rebuilder/rebuilder.service /usr/lib/systemd/system
$ systemctl daemon-reload

Configuration

Create the following folders:

$ mkdir -p /var/lib/rebuilder/{rebuild,broker,backend,ssh,gnupg}

The previously created folders are mounted differently into containers to store or share persistent data (see /opt/rebuilder/docker-compose.yml).

In gnupg folder you need to provide a GPG keyring containing private keys used to sign in-toto metadata and unreproducible in-toto metadata if you plan to generate them. Two keys are necessary to certify a proper separation between reproducible and unreproducible cases. In practice, users would only trust the key used to sign reproducible metadata. Both keys will be used by the underlying project people to track the general status of packages.

If you want to create fresh keys directly, repeat the following operation two times by properly identify which one would be the reproducible key from the unreproducible one:

$ GNUPGHOME=/var/lib/rebuilder/gnupg gpg --full-generate-key

Then, don't forget to publish your keys as follows (replace by corresponding fingerprints):

$ GNUPGHOME=/var/lib/rebuidler/gnupg gpg --keyserver keyserver.ubuntu.com --send-keys REPRODUCIBLE_KEY_FINGERPRINT UNREPRODUCIBLE_KEY_FINGERPRINT

In ssh folder you need to add a private SSH key allowed to push on a remote host destination via rsync. Ensure to have proper permissions:

$ chmod 700 /var/lib/rebuilder/ssh
$ chmod 600 /var/lib/rebuilder/ssh/id_rsa

Then, you need to edit the configuration file /opt/rebuilder/rebuilder.conf which will be injected into docker images having all the needed configuration information.

Debian configuration

For example, here is a typical configuration which can be used to rebuild Debian unstable and bullseye (to be adapted with your configuration values):

#
# Specific 'celery' options. Don't modify unless you know what you are doing.
#
[celery]
broker = redis://broker:6379/0
backend = mongodb://backend:27017

[debian]
# Scheduled task period for fetching latest packages to rebuild
schedule_get = 1800

# Scheduled task period for generating results
schedule_generate_results = 300

# GPG key fingerprint
# local keyring: /var/lib/rebuilder/gnupg
# container keyring: /root/.gnupg
in-toto-sign-key-fpr = 0123456789ABCDEF
in-toto-sign-key-unreproducible-fpr = FEDCBA987654321

# SSH private key name to use for accessing remote host
# local path: /var/lib/rebuilder/ssh/id_rsa
# container path: /root/.ssh/id_rsa
repo-ssh-key = id_rsa

# Remote host (via SSH)
repo-remote-ssh-host = rebuilder@rebuilder.debian.net

# Local directory on the remote host
repo-remote-ssh-basedir = /rebuild

# Snapshot service to use for repositories and API queries
snapshot = http://snapshot.notset.fr

# Debian suite to rebuild in the format: {suite_name}+{package_set}.{arch}
dist = unstable+essential+required+build-essential+build-essential-depends+full.amd64
    unstable+essential+required+build-essential+build-essential-depends+full.all
    bullseye+essential+required+build-essential+build-essential-depends+full.amd64
    bullseye+essential+required+build-essential+build-essential-depends+full.all

@fepitre's configuration aka NOTSET rebuilder setup

For example, here is the one used by notset-rebuilder

#
# Specific 'celery' options. Don't modify unless you know what you are doing.
#
[celery]
broker = redis://broker:6379/0
backend = mongodb://backend:27017

#
# Each option in 'common' section can be set per project distribution section
#
[common]
# Scheduled task period for fetching latest packages to rebuild
schedule_get = 1800

# Scheduled task period for generating results
schedule_generate_results = 300

# GPG key fingerprint
# local keyring: /var/lib/rebuilder/gnupg
# container keyring: /root/.gnupg
in-toto-sign-key-fpr = 8DEB0BEF1D99FEB8B9A90FB192EF6D6141641E5C
in-toto-sign-key-unreproducible-fpr = C46AE96200D2F98FFDD8257073D2D5D1AEA68333

# SSH private key name to use for accessing remote host
# local path: /var/lib/rebuilder/ssh/id_rsa
# container path: /root/.ssh/id_rsa
repo-ssh-key = id_rsa

# Remote host (via SSH)
repo-remote-ssh-host = rebuilder@mirror.notset.fr
# Local directory on the remote host
repo-remote-ssh-basedir = /data/rebuilder

# Snapshot service to use for repositories and API queries
snapshot = http://snapshot.notset.fr

#
# Available sections are 'debian', 'qubesos' and 'fedora' (fixme: the latter is a WIP)
#
[debian]
dist = unstable+essential+required+build-essential+build-essential-depends+full.amd64
    unstable+essential+required+build-essential+build-essential-depends+full.all
    bullseye+essential+required+build-essential+build-essential-depends+full.amd64
    bullseye+essential+required+build-essential+build-essential-depends+full.all

[qubesos]
dist = qubes-4.1-vm-bullseye.amd64

In the current docker setup, you only need to adjust in-toto-sign-key-fpr, repo-ssh-key, repo-remote-ssh-host and repo-remote-ssh-basedir values. Here schedule value is the time in second for which get task will be run periodically. Default here is 30 minutes. Please note that values for broker and backend variables refers to docker containers broker and backend defined in docker-compose.yml.

You can now enable and start the service:

$ systemctl enable rebuilder
$ systemctl start rebuilder

Please note that the first start of rebuilder service can take few minutes. In background, it creates all needed docker images.

Once it's started, it will fetch for new buildinfo files in Debian and Qubes OS repositories periodically every 30 minutes (schedule value in /opt/rebuilder/rebuilder.conf). You can manually trigger the get.

For that, you need to install python3-celery, python3-pymongo, python3-packaging and rsync on the rebuilder machine then, in /opt/rebuilder, run:

$ CELERY_BROKER_URL="redis://localhost:6379/0" ./init_feed.py

TODO: Add initial feed if rebuild queue is empty.

Check rebuild proofs before installing packages: apt-transport-in-toto

In this section, we give configuration steps for bullseye based qube in order to use rebuild proofs before installing packages thank to apt-transport-in-toto. For more details on in-toto, its apt transport setup and configurations options, we refer to upstream projects https://in-toto.readthedocs.io and https://github.com/in-toto/apt-transport-in-toto.

Installation

For dependencies, you need to install:

$ apt install apt-transport-in-toto in-toto python3-securesystemslib python3-pathspec python3-iso8601 python3-cryptography

Configure apt-transport-in-toto

The APT configuration file /etc/apt/apt.conf.d/intoto:

APT::Intoto {
  LogLevel {"20"};
  Rebuilders {
    "https://debian.notset.fr/rebuild/";
    "https://qubes.notset.fr/rebuild/deb/r4.1/vm/";
  };
  GPGHomedir {"/var/lib/intoto/gnupg"};
  Layout {"/var/lib/intoto/root.layout"};
  Keyids {
    "9fa64b92f95e706bf28e2ca6484010b5cdc576e2";
  };
  NoFail {"true"}
};

This file uses GPGHomedir to refer to a GPG keyring containing all the public keys used for signing in-toto metadata, i.e. the rebuilders keys and the public key used to sign the root.layout file. The Keyids contains public keys ids used for signing root.layout only. The root.layout file contains information like expiration time for root.layout validity, rebuilders keys to use and also how the in-toto engine has to verify the metadata with respect to received metadata.

Create the following directory:

$ mkdir -p /var/lib/intoto/gnupg
$ chmod 700 /var/lib/intoto/gnupg

and add the root.layout file in /var/lib/intoto where we provide the one associated for notset-rebuilder:

{
 "signatures": [
  {
   "keyid": "9fa64b92f95e706bf28e2ca6484010b5cdc576e2",
   "other_headers": "04000108001d1621049fa64b92f95e706bf28e2ca6484010b5cdc576e205026008aba9",
   "signature": "ac067f95f977cf4c660040cc3493ef0e9a8173938f0860179706d6f19a1864fc3ac1840fbfe486b0bd794c5d210b3f22c2f09f27addc5d8e56d1b7cdda9ed6339d1a0f79c4ac505c675f02a553e4e695f7b6c88def64c079bd0ab8684d05505cd2ec5c3e701192f8021f6f1956875720ecfd8c1bb2a073e23a60e22d11b79e0463a9409bb46643dffd09d604ea5994543d279b94c592fe1d251f33908ee50db0d02b33eb1ea9f57d82a6d0b46df5e39f97fa8564b66f2e5d094ba56aaf94ffd5224bc3a764129c336c279a5dac5d87c6d190a7ded21dfb586c79d679c3996fd6f4a0c61bb4488ea040a6e2365f8e16c91b8f1deb3d2e03d5304e91b8251d4719938c50ca18dbf489bb0d9145184e4432b5f83eb10f571eb4c7f0dd68211f9300a6cbfb3a19ba4ff13a3cb49b4a2f03cc0fe024bffed4154fc091964cc514692cd641845af116ef2c3137e581b7f699b76a43ffc3818d434931bdf7f1f5841dda767b3ab073cc5cb6eef8d642c512e1e3fd28b6d5e58065f21f50145d00118c5f345a273921fc6040d3454ba5a3c1900e07ca5ef38316e84c4e87124a85a83e2e7988dc0c07f804333beddab74db4293bf7753b1fac06e7470d86687a0236ab7fc69b73f4c1d429f93b0a6a373b43d3f17bad375f210d285cef329cd61321af551f7ef939ae7bc96019dfb06f4c12cb6f163cc6826b9fa3ac2f7171aaf54fe866"
  }
 ],
 "signed": {
  "_type": "layout",
  "expires": "2023-01-20T00:00:00Z",
  "inspect": [
   {
    "_type": "inspection",
    "expected_materials": [
     [
      "MATCH",
      "*.deb",
      "WITH",
      "PRODUCTS",
      "FROM",
      "rebuild"
     ],
     [
      "DISALLOW",
      "*.deb"
     ]
    ],
    "expected_products": [],
    "name": "verify-reprobuilds",
    "run": [
     "/usr/bin/true"
    ]
   }
  ],
  "keys": {
   "8deb0bef1d99feb8b9a90fb192ef6d6141641e5c": {
    "creation_time": 1611149279,
    "hashes": [
     "pgp+SHA2"
    ],
    "keyid": "8deb0bef1d99feb8b9a90fb192ef6d6141641e5c",
    "keyval": {
     "private": "",
     "public": {
      "e": "010001",
      "n": "e3eed2ff783b0c5e23ed4910d22357bfbada1987ca160773865a7ef9e2fe3696cbde40b15e25f728ef03345902ce16318683987e167b5aacdc0d6eee559093148195192fffe0f37b5d7fa8ea2f1941ea04a761dee508b6c4223bb7af18a9fcae5ad7981fd3cd96f16d5397088d4569aac7a4124fc1361d6905af3ab4f8818508776c749aadb4658e080f6c7e572e3bfca4bf497e31106a28c53345468a50294f79ffb2e2d98ffcb3b5b4a9d9f366a0bc359806fd60664ac2a6a389fdcd6806c2de689669c34ad6a876a20ff80b89e6d536bbcab8eb385beaca07fb8aad8282493d9f597bea74f916602dffa0943ccbc736de530e905e0e1164c2e273253b38d4f1edaf4bc95100c865da0eb64f58ff33f21fc77e5495e8d4698c2dad7085a4f4f3ec06aa129ca3bdcc68340fedfc785b1fbb9ee4d4d9d89ce7ac866ba90f5263cc12760d264008b4ae545c9e7ed0086f0a4104d3c388d22a54dbeeace29174ceb175f4f12f7831583dfabf2f19c9f73f667b08986957d36434e6cc9c13407360c7e00769aeb027708687b663a01a30db799605ead9ce93ca34b397948f5f8c8cfb382e7bcdf4b81526bb136664ff538d859c84250314642b8ec6da11e9abb576ef6f274d4abe9a1d5b4e1cc1064ab59276ab8f161c4903e9f0b44747db3f243b023b93b79ab52873de4bcfd4d884ac3ce112a9c9576b7092a15c84233eef69e7"
     }
    },
    "method": "pgp+rsa-pkcsv1.5",
    "type": "rsa",
    "validity_period": 63072000
   },
   "9fa64b92f95e706bf28e2ca6484010b5cdc576e2": {
    "creation_time": 1545907057,
    "hashes": [
     "pgp+SHA2"
    ],
    "keyid": "9fa64b92f95e706bf28e2ca6484010b5cdc576e2",
    "keyval": {
     "private": "",
     "public": {
      "e": "010001",
      "n": "dc7f268e91eb9ffa0f7a4b589eab4e6d27cbd31ac42b35743944313b7df6a02f4f84a5f20fdf1ef7c794340f2dab51543ea0dd431dcb9e34bef446d36cf41167a16d0a667b246d3e27109f7040241adb695068893d90235575d8761c92a58ed77cbeda2f8bdeec5888edcf1c7943073699dd0363d00784d0b82aa2885b7b523c2d7b52c23c7c04a83401c8b58b729f66ceebba7ca4222f96da01ad89c4943d4fca9e48196883dbf603df6e623e5dbea875645125a66fdda1451535f5183e51db3c6d51ea873e1c3aa204ee3a57818cd337f92af2cc41321d8ead0fdb4b12da71c68d02921549955201ab1525f76c70d0cbf5f8e41c444afc08924e5787aec80b7f83cbc39928a8fb3fe89108548fadd29ca0f8f6bcbb01687984cfacb4a5c347fb6d74a69557eff20ecced15a69374f9408a5e1c0d23de511935b6115f3908f751c2b78d17d9b4dbda34ca82f07d8d08c46aba64e9ebaf1acd8883f538325532de91b912177611b9fc473df5257a5a03f18f689e438f3574abd9a46241832d823727f27eb36bb42abd15b302da20ddd6267ee42840f07dce2cda468061f6e513fa74cd58ab11506e9bbdb73051f80f1853acea1682d6b69f24473a88b4cf411585803e0fcea614f58a6fdae85afb6798596cb208069aa6c910cbed2db627952b77bf7da77d45e1d766903f053f2c2188ceae2814d56b369dda58f9d26a32cfa5"
     }
    },
    "method": "pgp+rsa-pkcsv1.5",
    "subkeys": {
     "77aff41dc7843d47d2a952956e8aaec1ab9505d4": {
      "creation_time": 1545907057,
      "hashes": [
       "pgp+SHA2"
      ],
      "keyid": "77aff41dc7843d47d2a952956e8aaec1ab9505d4",
      "keyval": {
       "private": "",
       "public": {
        "e": "010001",
        "n": "c03c9772172e0a798a6e1804d6c1608d1def17355a81a65574c95953e694a20b203750adf8265197f5824e8ff1653191a087ad6f6e22bea624639bf9726f5dbcc293239e83a6aa96d22dbadaf0d3fc32ee4564c72ff0ec6b488aff447ab685e160d63d7540e306c1eeaf3c2f75f2aa739a26988f6aa9298fd162c1bc57ef7e7692c453be7e9a91d4ec694409f3e61d86e7b22b262ffe3ec47fbc046e7aca0d1415b7eb358a272101529f993fc3a8bb8edeccfbe66659f115e6708fdaabf37980c600be816d654e8ee0e1acab0020a9e8982cb2f8c8dc54ff79e4f55d376f8e94f267834ee82ad42be5a34268da6ffe2e202ced4849c75f0f09ba81d7c1d9983fee34c9acac349163c597dac77cdedd2c0476f2edee4fc46189c8dd77b8606250d675b22684d2ff67b88bdde6b21fb4fda8054d539f3b3215af211132135823db672cd1ed66eaa305a5b1581b4ef4f4dd94c25a401fe3dddf96d38bdab6a35a47f75c41607e90befaf0b2e5581f0b9cdf15cb45545759763cc6f3454a56352c1f00e3a24eb55fd9a1d603ca1c5519b796a7cdb471bd2cd5febcf57097a85600cd37c28adf8479ba183913790868a85f34becf44bd2213db754bafaacef1bc2db8f1b33b9e6f3861d81533b1d6c9b3ada51a56a727301acef722f6f1c8b13d68f7bf670e0f4ca5af6c995c162a287cbc13404d57bd6a2614f0fd2cbac19793548d"
       }
      },
      "method": "pgp+rsa-pkcsv1.5",
      "type": "rsa"
     }
    },
    "type": "rsa"
   }
  },
  "readme": "",
  "steps": [
   {
    "_type": "step",
    "expected_command": [],
    "expected_materials": [],
    "expected_products": [
     [
      "CREATE",
      "*.deb"
     ],
     [
      "DISALLOW",
      "*.deb"
     ]
    ],
    "name": "rebuild",
    "pubkeys": [
     "8deb0bef1d99feb8b9a90fb192ef6d6141641e5c"
    ],
    "threshold": 1
   }
  ]
 }
}

In this file, GPG fingerprint 8deb0bef1d99feb8b9a90fb192ef6d6141641e5c corresponds to a rebuilder signing key (notset-rebuilder) and 9fa64b92f95e706bf28e2ca6484010b5cdc576e2 corresponds to the key used to sign the root.layout file. In this setup example, you can the keys by using links 9FA64B92F95E706BF28E2CA6484010B5CDC576E2.asc and 8DEB0BEF1D99FEB8B9A90FB192EF6D6141641E5C.asc. Once you have keys locally and verified them:

$ GNUPGHOME=/var/lib/intoto/gnupg gpg --import 9FA64B92F95E706BF28E2CA6484010B5CDC576E2.asc 8DEB0BEF1D99FEB8B9A90FB192EF6D6141641E5C.asc

Enable the transport

You need to enable in-toto transport for Qubes and Debian repositories. In /etc/apt/sources.list.d/qubes-r4.list you need to replace https by intoto. So up to commented entries, you should have:

# Main qubes updates repository
deb [arch=amd64] intoto://deb.qubes-os.org/r4.1/vm bullseye main

# Qubes updates candidates repository
deb [arch=amd64] intoto://deb.qubes-os.org/r4.1/vm bullseye-testing main

Doing the same for Debian, in /etc/apt/sources.list you obtain for a selected mirror:

deb intoto://ftp.fr.debian.org/debian bullseye main contrib non-free
deb intoto://ftp.fr.debian.org/debian-security bullseye-security main contrib non-free

Please note that there is currently an issue in replacing intoto for https mirrors only (see in-toto/apt-transport-in-toto#34). Ensure to have a supported http mirror selected.

DYO root.layout

If doing on your own the root.layout file, signatures section and keys are empty. Before signing the whole file, you need to provide keys sections. For that you can use the following command:

$ python3 <<EOL
import json

from securesystemslib.gpg.functions import *

keys = securesystemslib.gpg.functions.export_pubkeys(
  [
    "8deb0bef1d99feb8b9a90fb192ef6d6141641e5c", 
    "9fa64b92f95e706bf28e2ca6484010b5cdc576e2"
    ], homedir="/var/lib/intoto/gnupg")

print(json.dumps(keys, indent=1))
EOL

Then, the root.layout can be signed using in-toto tools:

$ GNUPG=qubes-gpg-client-wrapper in-toto-sign --verbose --gpg 9fa64b92f95e706bf28e2ca6484010b5cdc576e2 -f root.layout

If any syntax error has been made, the file won't be signed. Behind the scenes, in-toto tools use securesystemslib for GPG operations and now supports passing GNUPG gpg client to use. By default, it uses gpg2/gpg. Please note that if expiration is not given in original root.layout before signing process then, it uses one month as default expiration time. Also, pay attention that we have provided currently only one rebuilder in /etc/apt/apt.conf.d/intoto for which we can only set threshold to 1 in root.layout. See upstream documentation about that.