373 lines
19 KiB
Markdown
373 lines
19 KiB
Markdown
# Repository manager and build tool for Arch Linux
|
|
|
|
This **experimental** project contains **unofficial** tools to manage custom Arch Linux
|
|
repositories. It is built on top of the official tools provided by the `pacman` and
|
|
`devtools` packages.
|
|
|
|
At this point the project is rather raw and there are many things left to implement and
|
|
to improve (checkout the TODOs section below). Currently builds are exclusively triggered
|
|
manually (although the API brings the possibility for automation), scalability is limited
|
|
through the use of an in-memory database. Builds are only conducted on one host.
|
|
|
|
On the upside, this project can easily be used to together with other build scripts. It
|
|
doesn't care if your repository's DB file is updated by another application; just be sure
|
|
to reload the DB file (see Workflow section below). This should make an easy migration
|
|
path to use this for maintaining an existing repository.
|
|
|
|
So far this project consists of:
|
|
|
|
* libpkg: C++ library to parse Arch Linux packages and databases
|
|
* parse pacman config
|
|
* parse databases
|
|
* extract and parse binary packages (.PKGINFO, dependencies on soname-level)
|
|
* parse package source infos (.SRCINFO)
|
|
* librepomgr: C++ library for managing custom Arch Linux repositories
|
|
* srv: Daemon and web application for building Arch Linux packages and managing custom Arch
|
|
Linux repositories
|
|
* build packages via `makechrootpkg` and add them to a custom repository via `repo-add`
|
|
* check AUR for package updates
|
|
* build from locally stored PKGBUILDs and from the AUR
|
|
* remove packages from a repository
|
|
* move packages from one repository to another
|
|
* check for repository issues (missing packages, package misses rebuild, …)
|
|
* provides an HTTP API and a web UI with live streaming
|
|
* pacfind: Tool to find the package containing a certain file
|
|
* requires `*.files`-databases to be present
|
|
* does not need the full file path and supports regex (in contrast to `pacman -F`)
|
|
* cli: Command line tool to interact with srv
|
|
* search for packages and show package details
|
|
* show and submit build actions
|
|
|
|
Further ideas (not implemented yet):
|
|
* distri: Tool to distribute applications from the packages in a repository
|
|
* bundle an application with its dependencies similar to `linuxdeployqt`
|
|
and `windeployqt`
|
|
|
|
## Setup server
|
|
An example config files can be found in this repository, see the `srv/doc`
|
|
directory. The example config is also installed so one can easily get started.
|
|
|
|
This is an example for using the package
|
|
[`arch-repo-manager-git`](https://github.com/Martchus/PKGBUILDs/blob/master/arch-repo-manager/git/PKGBUILD):
|
|
|
|
```
|
|
mkdir -p /etc/buildservice-git
|
|
cp /usr/share/buildservice-git/skel/server-config-example.conf /etc/buildservice-git/server.conf
|
|
cp /usr/share/buildservice-git/skel/presets-example.json /etc/buildservice-git/presets.json
|
|
systemctl enable --now buildservice-git.service
|
|
```
|
|
|
|
Note that the `-git` package is used at this point because there are no releases yet.
|
|
|
|
### Setting up a working directory
|
|
The server needs a place to temporarily store PKGBUILDs, cache files and other stuff.
|
|
Just create a directory at any place with enough disk space and set the permissions so the
|
|
user you start the server with can read and write there. This directory goes under
|
|
`working_directory` in the settings file. Relative paths within the configuration file (used
|
|
for other locations) will be treated relative to that directory.
|
|
|
|
### Setting up a database/repository directories
|
|
The databases for the official repositories will be synced from the upstream repositories and
|
|
stored in a local directory. The actual packages might be found in the pacman cache directory.
|
|
Example configuration:
|
|
|
|
```
|
|
[database/core]
|
|
arch = x86_64
|
|
sync_from_mirror = on
|
|
dbdir = $sync_db_path/$arch
|
|
mirror = $default_mirror/$repo/os/$arch
|
|
```
|
|
|
|
Own repositories are not synced from a mirror and all packages are expected to be present in
|
|
a dedicated local directory. It makes most sense to store packages and databases in the same
|
|
directory. Example configuration:
|
|
|
|
```
|
|
[database/ownstuff]
|
|
path = $own_regular_db_path
|
|
files = $own_files_db_path
|
|
arch = x86_64
|
|
depends = core extra community multilib
|
|
pkgdir = $local_db_path/$repo/os/$arch
|
|
dbdir = $local_db_path/$repo/os/$arch
|
|
mirror = $local_mirror/$repo/os/$arch
|
|
```
|
|
|
|
The placeholder/variable values come from the previous "definitions" block, e.g.:
|
|
|
|
```
|
|
[definitions]
|
|
sync_db_path = sync-dbs
|
|
local_db_path = /run/media/repo/arch
|
|
own_sync_db_path = $local_db_path/$repo/os/$arch
|
|
own_regular_db_path = $local_db_path/$repo/os/$arch/$repo.db.tar.xz
|
|
own_files_db_path = $local_db_path/$repo/os/$arch/$repo.files.tar.xz
|
|
default_mirror = http://mirror.23media.de/archlinux
|
|
local_mirror = file://$local_db_path
|
|
```
|
|
|
|
The server obviously needs write permissions to add packages to repositories. In my
|
|
setup I've just add it as group and set permissions accordingly:
|
|
|
|
```
|
|
sudo chown -R martchus:buildservice $local_db_path
|
|
find $local_db_path -type d -exec chmod 775 {} \+
|
|
find $local_db_path -type f -exec chmod 664 {} \+
|
|
```
|
|
|
|
### Setting up the chroot for the build
|
|
The overall reasoning and procedure is already outlined
|
|
[in the Wiki](https://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_clean_chroot).
|
|
|
|
I also like to be able to build for other architectures than x86_64 so the server
|
|
expects a directory layout like this:
|
|
|
|
```
|
|
/the/chroot/directory
|
|
├── arch-i686
|
|
│ └── root
|
|
├── arch-x86_64
|
|
│ └── root
|
|
├── config-i686
|
|
│ ├── makepkg.conf
|
|
│ ├── pacman.conf
|
|
├── config-x86_64
|
|
│ ├── makepkg.conf
|
|
│ └── pacman.conf
|
|
```
|
|
|
|
So there's a "root" chroot for every architecture and config files for that architecture
|
|
in dedicated directories. To achieve this one just has to call `mkarchroot` for each architecture.
|
|
It is invoked like this:
|
|
|
|
```
|
|
mkarchroot -C /path/to/pacman.conf -M /path/to/makepkg.conf /directory/to/store/the/tree base-devel
|
|
```
|
|
|
|
Note that `makechrootpkg` will not use the "root" chroot directories directly. It will create a
|
|
working copy using `rsync -a --delete -q -W -x "$chrootdir/root/" "$copydir"` (when using btrfs
|
|
it creates a subvolume instead of using rsync).
|
|
|
|
When following the procedure the "root" chroot is owned by `root` and the working copies by the
|
|
user you start the buildservice/makechrootpkg with.
|
|
|
|
Until I find a better solution the buildservice updates the config files within the "root" chroot
|
|
to configure the databases used during the build. It relies on the configuration files stored within
|
|
the `config-*` directories for that and needs permissions to update them:
|
|
|
|
```
|
|
sudo chown buildservice:buildservice /the/chroot/directory/arch-x86_64/root/etc/{pacman,makepkg}.conf
|
|
```
|
|
|
|
### sudo configuration
|
|
`makechrootpkg` seems to use `sudo` internally so it is required that the user one starts the
|
|
server with is allowed to use `sudo`. Currently it is not possible to supply the password
|
|
automatically so one has to allow access without password like this:
|
|
|
|
```
|
|
buildservice ALL=(ALL) NOPASSWD:ALL
|
|
```
|
|
|
|
### Sample NGINX config
|
|
To make the server publicy accessible one should use a reverse proxy. NGINX example:
|
|
|
|
```
|
|
proxy_http_version 1.1; # this is essential for chunked responses to work
|
|
proxy_read_timeout 1d; # prevent timeouts when serving live stream
|
|
|
|
location /buildservice/api/ {
|
|
proxy_buffering off; # for live streaming of logfiles
|
|
proxy_pass http://localhost:8090/api/;
|
|
proxy_set_header Host $host;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
}
|
|
location ~ ^(?!/buildservice/api/)/buildservice/(.*)$ {
|
|
alias /run/media/devel/projects/c++/cmake/auto-makepkg/buildservice/static/$1;
|
|
}
|
|
|
|
```
|
|
|
|
### Sample GPG config
|
|
This minimal GPG config allows `makepkg` to validate signatures which might be present
|
|
in some PKGBUILDs:
|
|
```
|
|
~/.gnupg/gpg.conf
|
|
---
|
|
utf8-strings
|
|
keyring /etc/pacman.d/gnupg/pubring.gpg
|
|
keyserver-options auto-key-retrieve
|
|
keyserver hkp://keys.gnupg.net
|
|
```
|
|
|
|
#### Notes
|
|
* Adding the pacman keyring is actually not very useful because we need to check signatures
|
|
of any upstream project and not just arch devs.
|
|
* If "auto-key-retrieve" does not work, use `gpg --recv-key <KEYID>` as a workaround
|
|
* Also see http://allanmcrae.com/2015/01/two-pgp-keyrings-for-package-management-in-arch-linux/
|
|
|
|
### ccache configuration
|
|
To use ccache one can set the ccache directory in the config (`ccache_dir`) and install the
|
|
package `ccache` (and `mingw-w64-ccache` for MinGW packages) into the chroot. Make sure the user
|
|
you start the server with has permissions to read and write there. Otherwise the resulting
|
|
configure errors can be confusing. Internally the server is mounting that directory like
|
|
described in [the wiki](https://wiki.archlinux.org/index.php/Ccache#makechrootpkg).
|
|
|
|
If you want to use the existing `ccache` directory owned by your current user, you could do
|
|
the following to grant the `buildservice` user access to it:
|
|
|
|
1. Add new group for accessing the cache and add the users to it:
|
|
`sudo groupadd ccache`, `sudo usermod -a -G ccache "$USER"`, `sudo usermod -a -G ccache buildservice`
|
|
2. Set group ownership: `sudo chown -R $USER:ccache $ccache_dir`
|
|
3. Ensure dirs are readable by the group and that the group is inherited:
|
|
`sudo find "$ccache_dir" -type d -exec chmod 2775 {} \+`
|
|
4. Ensure that all files are readable by the group:
|
|
`sudo find "$ccache_dir" -type f -exec chmod 664 {} \+`
|
|
5. Add `umask = 002` to the ccache config so new files/dirs created by it are fully accessible by
|
|
the group.
|
|
6. Patch `devtools` so additional groups are configured within the container (see
|
|
https://github.com/Martchus/devtools/commit/b981c6afe81219ff4ca1ea34d0be5cc681408962)
|
|
|
|
|
|
Note that ccache *slows* down the initial compilation. It is only useful to enable it if rebuilds
|
|
with only slight changes are expected. Currently it is *not* possible to enable/disable ccache usage
|
|
per package (e.g. via a regex).
|
|
|
|
## Workflow
|
|
So called "build actions" are used to conduct certain tasks, e.g. checking for updates or problems,
|
|
moving packages from one repository to another, preparing the build and conducting the build. The web
|
|
UI contains a form to create a new build action and shows the possible options.
|
|
|
|
It would be tedious to create the same set of build actions over and over again. Hence it is also
|
|
possible to create "pre defined tasks". Such a task contains a list of build actions with pre-filled
|
|
flags and settings. By default, the build actions within a task run one after another but it is also
|
|
possible to specify that certain build actions can run in parallel. The presets are configured by
|
|
editing the presets JSON file (e.g. `/etc/buildservice-git/presets.json` in the example config).
|
|
|
|
### General
|
|
* So far databases are *not* automatically reloaded. Use the "Reload database" build action to reload
|
|
one or more databases after databases have been changes by build actions or manually. This build
|
|
action is similar to `pacman -Sy`.
|
|
* Use the "Reload configuration" build action to apply configuration changes without restarting the
|
|
service.
|
|
|
|
### Building packages
|
|
0. Ensure the databases are up to date via the "Reload databases" build action.
|
|
1. Start the "Prepare building" build action.
|
|
* Specify the names of packages to be built.
|
|
* If a package already exists the `pkgrel` or `epoch` is bumped as needed so the rebuilt
|
|
packages are considered as newer by pacman. A warning is logged when bumping is done.
|
|
* Missing dependencies are pulled into the build automatically. Whether dependencies are
|
|
considered "missing" or "given" depends on the specified databases (see next points).
|
|
* Packages will be split into batches. The first batch contains all packages which can be
|
|
built immediately. The second batch contains all packages which only depend on packages in
|
|
the first batch. The third batch contains all packages which only depend on packages in the
|
|
second batch and so on. That means packages do not need to be specified in the correct order.
|
|
However, the order in which packages are specified is preserved within each batch so it makes
|
|
still sense to specify packages in the order you would prefer to build them.
|
|
* Cyclic dependencies can not be added to a batch. A list of cyclic "leftovers" is emitted
|
|
if those exist and that is considered a failure. If this is the case you need to add a
|
|
bootstrap package to break the cycle. The build system is not clever enough to pull a bootstrap
|
|
package automatically into the build so it must be specified explicitly. E.g. to build
|
|
`mingw-w64-freetype2` and `mingw-w64-harfbuzz` one needs to add `mingw-w64-freetype2-bootstrap`
|
|
explicitly to the list of packages to be built.
|
|
* Specify exactly one destination database. The built packages are added to this database.
|
|
* Specify source databases.
|
|
* All packages contained by the source databases are considered as "given" and not pulled
|
|
into the build as dependency.
|
|
* If no source databases are specified the destination database and all databases the
|
|
destination database is based on are used as source databases.
|
|
* Source databases must be the destination database or a database the destination database
|
|
is based on.
|
|
* Specify a "directory" to store meta-data, logs, PKGBUILDs, sources and packages.
|
|
* Check "Clean source directories" when previously prepared sources which are still present in
|
|
the specified "directory" should be overridden.
|
|
* When the build action has finished, have a look at the results to check whether they are as
|
|
expected. If not, just restart the build action with adjusted parameters. The "directory"
|
|
can generally be re-used. Use "Clean source directories" as needed. To override only a single
|
|
source directory, simply delete it manually before restarting the build action.
|
|
* You can amend PKGBUILDs created in the "directory" as needed.
|
|
2. Start the "Conduct building" build action.
|
|
* Specify the same "directory" as in 1.
|
|
* Do *not* specify any databases. The databases as specified in 1. will be used.
|
|
* One can optionally specify package names to build only a subset of the initially prepared
|
|
build.
|
|
* When starting the build, the following steps are performed:
|
|
1. All sources of all packages are downloaded. The build is stopped when not all sources
|
|
can be downloaded.
|
|
2. Packages will now be built in the same order as computed when preparing. Building of
|
|
the next batch is stopped when a failure happened within the previous batch.
|
|
* When using "Build as far as possible" the build is not stopped as previously explained.
|
|
* When using "Save chroot of failures" the chroot directory is renamed when a build failure
|
|
happens and therefore not overridden by the next build. This is useful to investigate the
|
|
failure later without starting the build from scratch.
|
|
* When the build has been stopped due to failures one can simply restart the build action. Packages
|
|
which have already been built will be skipped.
|
|
* It is useful to modify `build-progress.json` in the "directory".
|
|
* Set `finished` back to `"1"` and `addedToRepo` back to `false` to let the build system think
|
|
a package has not been built yet. This is useful to build a package again after a non-fatal
|
|
packaging mistake.
|
|
* Set `addedToRepo` to `true` let the build system think a package has already been built.
|
|
This effectively skips a package completely.
|
|
* Set `hasSources` back to `false` to recreate the source package and download sources
|
|
again as needed. This will also copy over contents of the `src` directory to the `pkg`
|
|
directory again.
|
|
3. Amend a PKGBUILD file to fix a failure within a "big rebuild".
|
|
* Fix the package at the source, e.g. the AUR or the local package directory, then
|
|
1. Clean the `src` directory within "directory" for the package(s) or use "Clean source directory" if
|
|
all packages need fixing.
|
|
2. Restart step 1. Existing sources are reused (except for the removed `src` directories) so it shouldn't
|
|
take long. The build progress (`hasSources` in particular) of the affected package(s) should be reset.
|
|
3. Restart step 2.
|
|
* or: Amend the PKGBUILD within the `src` directory.
|
|
1. Set `hasSources` back to `false` as mentioned under 2. to retrigger building the source directory.
|
|
2. Optionally restart step 1. to reevaluate possibly adjusted versions, split packages and dependencies.
|
|
Be sure *not* to check "Clean source directory" as this would override your amendment.
|
|
2. Restart step 2.
|
|
|
|
#### Further notes
|
|
* One can keep using the same "directory" for different builds. Since the `pkg` dirs will not be cleaned
|
|
up (existing files are only updated/overridden) previously downloaded source files can be reused (useful if
|
|
only `pkgrel` changes or when building from VCS sources or when some sources just remain the same).
|
|
|
|
## TODOs and further ideas for improvement
|
|
* [ ] Use persistent, non-in-memory database (e.g. lmdb or leveldb) to improve scalability
|
|
* [ ] Allow triggering tasks automatically/periodically
|
|
* [ ] Allow to run `makechrootpkg` on a remote host (e.g. via SSH) to work can be spread across multiple hosts
|
|
* [ ] More advanced search options
|
|
* [ ] Refresh build action details on the web UI automatically while an action is pending
|
|
* [ ] Fix permission error when stopping a process
|
|
* [ ] Keep the possibility for a "soft stop" where the build action would finish the current item
|
|
* [ ] Show statistics like CPU and RAM usage about ongoing build processes
|
|
* [ ] Stop a build process which doesn't produce output after a certain time
|
|
* [ ] Find out why the web service sometimes gets stuck
|
|
* Restarting the client (browser) helps in most cases, it likely just hits
|
|
[a limitation to the maximum number of open connections](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#receiving_events_from_the_server)
|
|
* Add "stress" test for live-streaming
|
|
* Start process producing lots of output very fast
|
|
* Let different clients connect and disconnect fast
|
|
* [ ] Improve test coverage
|
|
* [ ] Add fancy graphs for dependencies on the web UI
|
|
* [ ] Include `xterm.js` properly
|
|
|
|
## Build instructions and dependencies
|
|
For a PKGBUILD checkout my [PKGBUILDs repository](https://github.com/Martchus/PKGBUILDs).
|
|
|
|
### C++ stuff
|
|
The application depends on [c++utilities](https://github.com/Martchus/cpp-utilities),
|
|
[reflective-rapidjson](https://github.com/Martchus/reflective-rapidjson), some Boost modules and OpenSSL.
|
|
|
|
For basic instructions checkout the README file of [c++utilities](https://github.com/Martchus/cpp-utilities).
|
|
|
|
### Web stuff
|
|
The only dependency is `xterm.js` which is bundled. However, only the JavaScript and CSS files are
|
|
bundled. For development the full checkout might be useful (e.g. for TypeScript mapping). It can be
|
|
retrieved using npm:
|
|
|
|
```
|
|
cd srv/static/node_modules
|
|
npm install xterm
|
|
```
|