Rebuilderd
Rebuilderd is an independent verification system for binary packages. It allows verification of pre-compiled packages by repeating the build step in an identical environment and then verifies that the package is identical. Right now only Arch Linux packages can be verified.
If you run a rebuilder or consider running a rebuilder feel free to join the #archlinux-reproducible IRC channel.
Installation
Install the rebuilderd package.
Configuration
Single machine
On a single machine you just enable rebuilderd.service and the worker service(s): rebuilderd-worker@workername.service.
Afterwards you can verify rebuilderd is correctly running with a single worker by running:
# rebuildctl status
If you get an error that no authentication cookie could be found, you might need to add yourself to the rebuilderd user group.
Continue with the next section to sync packages that our worker can rebuild.
Multi-node rebuilder
To run a multi-node rebuilder network we need at least two servers:
- rebuilderd daemon
- This one only needs very few resources (a CX11 instance is fine), it keeps track of all packages, coordinates the workers and serves results to clients. This daemon should be setup with HTTPS.
- rebuilderd worker
- This server runs the actual builds. This server should have at least 16 GiB RAM, if you want to build all packages this should be closer to 32 GiB.
The worker needs to authenticate itself to the daemon with a secret signup key (worker.signup_secret). This key can be generated with pwgen -1s 32.
You can also generate a second secret to use rebuildctl remotely (auth.cookie). This is optional, you can also run rebuildctl on the server after adding yourself to the rebuilderd group.
/etc/rebuilderd.conf
## Configuration for http daemon [http] ## The address to bind to. This is 127.0.0.1:8484 by default. bind_addr = "0.0.0.0:8484" ## If you use a reverse proxy, use this header instead of the actual connecting ip. ## Make sure the reverse proxy has filters in place to prevent spoofing issues. #real_ip_header = "X-Real-IP" ## Set a default endpoint for rebuildctl. This is especially useful for the sync timer. #endpoint = "http://127.0.0.1:8484" ## A random cookie for administration is generated at startup and written to /var/lib/rebuilderd/auth-cookie ## You can set this to a fixed value here. Use `pwgen -1s 32` to generate one. ## rebuildctl is searching for this cookie in ~/.config/rebuilderd.conf, /etc/rebuilderd.conf and ## /var/lib/rebuilderd/auth-cookie in that order. #[auth] #cookie = "INSECURE" ## The auth cookie above is only used for the default endpoint. ## You can use different endpoints with `rebuildctl -H https://rebuilder.example.com status`, ## In that case you need to configure a section below if you want to attach a cookie. #[endpoints."https://rebuilder.example.com"] #cookie = "INSECURE" ## IMPORTANT: in production, make sure either `authorized_workers` or `signup_secret` is configured. [worker] ## If we have a fixed set of workers we can allow-list the keys here. #authorized_workers = ["key1", "key2"] ## If we want to spawn new workers dynamically we can configure a sign up secret below. ## Use `pwgen -1s 32` to generate one. signup_secret = "INSECURE" #[schedule] ## Configure the delay to automatically retry failed rebuilds in hours. The ## default is 24h, this base is multiplied with the number of rebuilds, so the ## first retry would happen after 24h, the second retry would happen 48h after the ## first retry and the third retry would happen 72h after the second retry. There ## is no upper limit of retries, if you can't afford frequent retries it's ## recommended to set this to a high value like 168 (1 week) or higher. ## Successful rebuilds are not retried. #retry_delay_base = 24
Start and enable rebuilderd.service.
Next we head over to our worker machines to set them up. Install rebuilderd as well and edit /etc/rebuilderd-worker.conf:
/etc/rebuilderd-worker.conf
## The rebuilderd to connect to endpoint = "https://rebuilder.example.com" # or http://rebuilder.example.com:8484 ## The server would either allowlist our key or require a signup secret signup_secret = "INSECURE-CHANGE-ME" [build] #timeout = 86400 # 24 hours ## Set a maximum build log limit in bytes (default: none). ## When reaching this limit the log is truncated but the rebuilder backend is *not* terminated. max_bytes = 10485760 # 10 MiB ## By default build output is forwarded to stdout/stderr. ## This can be disabled by settings this to true. #silent = true [diffoscope] ## Generate and attach diffs with diffoscope when rebuilding enabled = true ## Pass additional arguments to diffoscope. Use wisely, some options might not work well. #args = ["--max-container-depth", "2", "--fuzzy-threshold", "0"] ## Set a timeout in seconds after which diffoscope is terminated (default: 3600) #timeout = 600 # 10 minutes ## Set a maximum diffoscope output limit in bytes (default: none). ## When reaching this limit, diffoscope is terminated and the output is truncated. max_bytes = 41943040 # 40 MiB
We can start the worker unit multiple times for concurrent rebuilds (if the machine has enough resources). We are going to start with just one worker (and call it 0) by starting/enabling rebuilderd-worker@0.service
Syncing packages to rebuild
The rebuilder setup will not do anything by default until you explicitly configure where to sync packages from. The sync profiles are configured in /etc/rebuilderd-sync.conf. The profile names are supposed to be unique. You configure it to only build packages of a specific maintainer with the maintainer option. By default it is importing packages of all maintainers.
## rebuild all of core [profile."archlinux-core"] distro = "archlinux" suite = "core" architecture = "x86_64" source = "https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch" ## rebuild extra packages of specific maintainers, or whitelist packages by name. ## If no filter is set, all packages are imported, if both filters are set the package only ## has to match one of them to be included. #[profile."archlinux-extra"] #distro = "archlinux" #suite = "extra" #architecture = "x86_64" #source = "https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch" #maintainers = ["somebody"] #pkgs = ["some-pkg", "python-*"] #excludes = ["tensorflow*"]
Afterwards you can enable rebuilderd-sync@archlinux-core.timer to automatically sync the profile.
Ansible playbook
An Ansible playbook with a rebuilderd role can be found in Arch Linux's infastructure repository. Use this as inspiration as the Ansible repository is heavily dependent on how the Arch Linux infrastructure is set up.
Tips and tricks
Requeueing failed builds
Starting with rebuilderd 0.5.0 failed builds are retried automatically with increasing delays. You can retry packages immediately with:
# rebuildctl pkgs requeue --suite core --status BAD
In older versions you had to use this command to add all failed builds to the queue again:
# rebuildctl pkgs ls --distro archlinux --suite core --status BAD --json | jq -r '.[].name' | xargs -L1 rebuildctl queue push archlinux core
Removing suites after being synced
Currently, using rebuildctl is not possible to remove whole suites after they have been synced once. However, this can be achieved by setting a filter that does not match on any packages before removing the sync profile. For example, adding the following maintainers filter will effectively remove the core suite from rebuilderd.
[profile."archlinux-core"] distro = "archlinux" suite = "core" architecture = "x86_64" source = "https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch" maintainers = ["nonexistent"]
Package rebuilders
Rebuilders using Rebuilderd.
| URL | Contact | Comment |
|---|---|---|
| https://reproducible.archlinux.org/ | jelle | |
| https://wolfpit.net/rebuild/ | wahrwolf | |
| https://r-b.engineering.nyu.edu/ | NYU Secure Systems Lab | |
| https://rebuilder.pitastrudl.me/ | pitastrudl | Contact on IRC Libera or pitastrudl@chaos.social |
| https://reproducible.crypto-lab.ch/ | Applied Cyber Security Research Lab (HSLU) | Contact iyanmv on #archlinux-reproducible |