edb014011c
Make package search more efficient
...
* Avoid keeping list of all packages in memory by pushing each package
directly to the JSON array
* Avoid check for database before pushing each package; do it only before
reaching the next database
* Remove no longer required functions
2022-04-17 23:39:25 +02:00
4578b30ea2
Remove hard-coded build directories in test data
2022-04-15 19:58:35 +02:00
eae8af6571
Avoid big build action IDs by reusing older/lower ones
2022-04-15 19:20:01 +02:00
e15f65ac16
Extend tests for conducting build
2022-04-07 23:18:14 +02:00
08421fa719
Don't rely on local pkg dir when reloading library deps
...
Since 11f1849fde
2022-04-07 21:35:05 +02:00
7c627e6a12
Use auto in getPackages()
2022-03-22 21:03:56 +01:00
56afe0c710
Fix returning newly retrieved AUR packages
2022-03-22 21:03:01 +01:00
d163f56abf
Log whether io_uring is used/configured
2022-03-20 17:37:33 +01:00
d5e2b5489d
Provide filename for artefact downloads
2022-03-15 23:22:34 +01:00
69a81f7583
Remove unused function declaration
2022-03-15 22:27:50 +01:00
83c7426fce
Allow specifying output path for license info
2022-03-15 22:23:34 +01:00
13cccd3bc9
Improve creating license info
2022-03-15 01:10:15 +01:00
f87829ba85
Save state after reloading configuration
2022-03-15 01:07:51 +01:00
11f1849fde
Keep local pkg paths deduced from pacman config
...
It makes the setup easier and shouldn't hurt otherwise.
2022-03-15 01:07:36 +01:00
b0234ff02e
Improve MIME-type when serving artefacts
2022-03-15 00:21:08 +01:00
59cd593a3b
Display template name
2022-03-12 22:46:42 +01:00
70a98ffedc
Make name of build action to execute custom command shorter
2022-03-12 21:37:59 +01:00
ac0d64fe3e
Allow everyone to see build action details
2022-03-10 23:19:02 +01:00
55f081153e
Add command to delete invalid build actions from DB
2022-03-07 23:36:48 +01:00
7ade757c8d
Avoid locking whole config when updating DBs
...
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
should be safe to do so even when a write operation happens because
* LMDB read and write transactions can happen at the same time
* The package cache has its own mutex anyways
* Write ops to the package cache try to lock the "update mutex" to
prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
2022-03-05 18:55:52 +01:00
afc61bcad6
Speed up AUR updates using PackageUpdater
2022-03-05 16:10:33 +01:00
8fdf7bfb04
Print certain resource limits on server startup
2022-03-05 12:34:26 +01:00
9acd10fa4c
Catch boost::exceptions
2022-03-05 12:34:05 +01:00
9b6be51dd3
Allow reloading library dependencies of specific packages
2022-03-04 21:51:25 +01:00
4293eb92bf
Ignore libs for cross-compilation when checking any packages for binaries
...
Not really generic yet (e.g. doesn't cover `aarch64-linux-gnu-glibc`) but
good enough for now.
2022-03-04 21:50:56 +01:00
c633eea607
Add any-packages containing binaries as problem when checking for errors
2022-03-03 23:08:14 +01:00
c2e7f40767
Fix typos
2022-03-01 00:43:06 +01:00
79123e4cd9
Improve error handling of server
...
* Return with non-zero exit code when an exception occurred
* Handle exceptions when terminating the service
2022-03-01 00:39:50 +01:00
0343685910
Adapt tests for limiting
2022-02-25 23:18:26 +01:00
67fa8def0b
Name additional threads
2022-02-25 23:09:46 +01:00
2071327a71
Fix building packages using signed git repositories as source
2022-02-25 23:09:16 +01:00
90ff9678fe
Limit results returned by the API for better scalability
2022-02-25 00:29:43 +01:00
ff7f039519
Ensure problems of repos with different archs but the same name aren't mixed up
2022-02-22 22:50:51 +01:00
382169ab6e
Fix varoius build action related problems
...
* Fix crashes in some situations
* Fix aborting reloading library dependencies
2022-02-22 22:49:40 +01:00
5c62014b3b
Fix variable name in fallback code when BOOST_ASIO_HAS_FILE
2022-02-22 20:22:00 +01:00
a25db1f9e0
Log usage of io_uring
2022-02-22 00:08:21 +01:00
839a4b0831
Fix wrong variable name in buildactionlivestreaming.cpp
2022-02-22 00:05:35 +01:00
8ea2f5fd62
Reintroduce cache file as it is still required for "lastUpdate" for DBs
...
Maybe it makes more sense to use LMDB for this as well. On the other hand
it is an easy way of keeping the remaining state persistent.
2022-02-20 20:17:42 +01:00
9c4f0277bd
Unlock output lock early
...
The following things are read-only or guarded by other mutex.
2022-02-20 19:51:46 +01:00
bdbfd42e5e
Avoid package object being reused after 3d7310af3
2022-02-20 19:50:44 +01:00
334bb1faa9
Use io_uring for async file I/O
2022-02-20 18:44:07 +01:00
8115721913
Fix handling error computing db order when reloading dependencies
2022-02-20 00:18:49 +01:00
93afb3883d
Write build action "output" log to a logfile like for sub-processes
2022-02-20 00:11:23 +01:00
a7de520549
Use lazy-deserialization and package cache when searching in package name
2022-02-19 00:11:59 +01:00
b3b396df5f
Allow ignoring copying errors due to existing files when moving packages
2022-02-16 19:47:25 +01:00
8949b5f3c1
Fix resetting running build actions on server startup
2022-02-16 19:17:29 +01:00
557fd1a738
lmdb: Use lmdb to store build actions
2022-02-13 21:04:54 +01:00
4fbe3df095
Allow enabling Boost.Asio's support for using io_uring
...
It won't likely do much regarding file handling as code is currently still
using `boost::asio::posix::stream_descriptor` or Boost.Beast's file
handling (which uses blocking I/O). I suppose one needs to use the new file
API described here under "Asio 1.22.0 / Boost 1.78":
https://www.boost.org/doc/libs/1_78_0/doc/html/boost_asio/history.html
2022-02-05 20:31:11 +01:00
e582d5f406
Remove useless blank line
2022-02-03 22:07:22 +01:00
4ed645be94
Fix handling of head requests
...
* Fix condition for skipping
* Open destination file only if the request is not skipped (when doing a
head request first)
2022-02-02 21:18:07 +01:00
66f59fecb9
Make storing lib deps more efficient and allow for intermediate reads
2022-01-31 21:32:07 +01:00
45922b47ec
lmdb: Allow configuring cache limit
2022-01-26 00:41:53 +01:00
d4d187463a
Fix problems with global locks
...
* Aquire the mutex before setting `m_exclusivelyOwned = false`; otherwise
it might be set after `lock()` has checked it but before `lock()` has
entered `wait()` leading to a deadlock
* Check state again after `wait()` because it may also be unblocked
"spuriously" so it isn't guaranteed that the state will have actually
changed after unblocking
2022-01-25 22:49:20 +01:00
218dfecf56
Avoid loading databases when modification date is <= last update
...
* Do HTTP head request first when loading database from mirror to avoid
downloading the full database all the time
* Use the last modification date of the local database file because with
the persistent storage even local database reloads became a bit expensive
2022-01-25 00:13:10 +01:00
300f74fb10
lmdb: Open LMDB file after restoring cache so DBs added from cache have the storage initialized
2022-01-25 00:01:59 +01:00
1c820fc2f6
lmdb: Fix storage initialization when config file cannot be loaded
2022-01-23 19:47:50 +01:00
f088e54c61
lmdb: Use package cache when querying packages for dependencies
...
* Use package cache when querying packages for dependencies or library
names
* Avoid manual instantiations of `std::shared_ptr<Package>`
2022-01-21 20:35:43 +01:00
e44cbf94e5
lmdb: Enable versioning for objects we'd like to store in the DB
2022-01-18 22:48:34 +01:00
2ffa6629c8
lmdb: Use lmdb to store packages and dependency indices
2022-01-18 21:50:53 +01:00
0a2c41ee0c
Fix shadowing warning
2021-12-07 22:48:22 +01:00
567c3aaf56
Support "trunk" subdirectory for using official svntogit repos as source
2021-11-16 15:00:19 +01:00
0f7718c934
Locate PKGBUILDs locally also without variant dir
...
This makes it possible to checkout
https://github.com/archlinux/svntogit-packages.git and
https://github.com/archlinux/svntogit-community.git locally to build
official packages.
2021-11-14 23:37:33 +01:00
6f652922cd
Add lookup for repo-specific config file path
...
This could be useful to create another repository with packages that are
built using different compile flags, e.g. optimized for certain processors.
2021-11-14 22:59:46 +01:00
2adc80e5fc
Override existing symlinks when adding signatures to repo
2021-11-03 16:32:38 +01:00
893a8473ad
Override existing symlinks when adding packages to repo
2021-11-03 16:06:16 +01:00
755e1958d3
Allow overriding existing files in repository
...
When `repo-add` fails but files could be copied before refusing to override
is actually very annoying because one has to delete the files before
restarting the build task.
2021-10-07 18:12:44 +02:00
672f3b4e97
Fix staging check for split packages
...
* Consider all split packages when skipping an affected package
2021-09-30 22:23:54 +02:00
ace478be22
Fix tests when test working directory is on another device
...
* Fallback to copying if hard link creation fails due to cross device link
2021-09-30 22:22:22 +02:00
a93924bb3a
Fix cleanup of signatures
...
* Record full path for the deletion of orphaned signatures
* Test for the existence of the signature symlink itself and not the target
because the symlink target (in the any directory) might not exist anymore
but the symlink should be removed in any case
2021-08-29 00:32:19 +02:00
c68202bdc1
Add test for repository cleanup
2021-08-29 00:29:26 +02:00
1cc8a3d6db
Add warnings about missing DLLs to the build action's result
2021-08-28 15:06:24 +02:00
2ad4a39aea
Handle case when a split package overrides the archs from the base
2021-08-28 00:25:09 +02:00
95ae827c74
Fix applying relative working directory multiple times
2021-07-31 14:55:49 +02:00
1ce736d66f
Fix passing the body limit to nested cachePackages() calls
2021-07-25 01:25:30 +02:00
075f539480
Avoid copying strings when splitting lock names for custom command
2021-07-25 00:41:46 +02:00
f3ec908bcc
Hold lock for named lock table not while acquiring named lock
...
Otherwise all other attempts to acquire named locks are blocked. It should
be ok because references are not invalidated when inserting/accessing
elements of an `std::unique_map`. When cleaning locks up elements are
erased, though. Hence an additional cleanup lock is required to prevent
acquiring a named lock which has already been cleaned up.
2021-07-25 00:40:39 +02:00
bbc7c3a533
Ignore empty packages within the cache dir
...
Those are most likely leftovers from previous failed download attempts and
can be ignored. (Existing files are overridden.)
2021-07-17 23:58:37 +02:00
e5f4057c61
Use BufferSearch from c++utilities
2021-07-16 16:01:23 +02:00
bc8a0549b2
Log package download limit
2021-07-14 15:27:49 +02:00
909f7bfab0
Use default package download size limit if it is set to zero
2021-07-14 15:23:18 +02:00
e0a913b16c
Apply clang-format
2021-07-13 00:41:49 +02:00
c05a581322
Increase the max. download size for packages and allow configuring it
2021-07-12 15:00:43 +02:00
a2fbba9713
Allow constructing BadRequest from std::string
2021-07-07 19:46:43 +02:00
b5ca815e4c
Allow excluding packages via regex when reloading library dependencies
...
This recently failed because the cuda package exceeded the max. body size.
I suppose it makes generally sense to be able to exclude packages as I
don't need that package anyways.
2021-07-07 19:14:03 +02:00
d006c9ce4f
Fix error handling when instantiating std::regex
2021-07-07 19:09:32 +02:00
17c5f50440
Do not exit process if config values are invalid
...
This is not a good idea when re-reading the config after startup.
2021-07-07 19:08:47 +02:00
49aebdd8a5
Fix typos found via codespell --skip .git -w
2021-07-03 19:59:57 +02:00
8b70cc5f8e
Clear added/removed provides from previous builds
2021-06-22 00:02:41 +02:00
f685fddd20
Add .pid-equivalent for Boost.Asio's Windows-executor
2021-06-22 00:01:55 +02:00
0a55412625
Remove unused #include
2021-06-21 23:58:34 +02:00
d9203689a0
Fix param for PrepareBuildFlags::ResetChrootSettings
2021-06-04 16:47:24 +02:00
3a797bc8b6
Adapt test to fixed behavior of splitString*() to preserve last empty part
2021-05-16 19:43:16 +02:00
9359004a10
Allow resetting chroot settings as well when preparing build
2021-05-16 16:49:09 +02:00
efbcef5e6e
Fix error handling when makechrootpkg can not be invoked
2021-05-16 13:49:57 +02:00
b9c7f05ef1
Add missing include for condition_variable
2021-05-15 21:04:21 +02:00
05ff6cd00f
Do not set packages in conduct build actions when creating it form task
...
Usually this doesn't make sense because we want to build all packages which
were determined in the previous prepare action. When specifying the
packages again in the conduct build action we'd skip packages which are
pulled into the build as dependencies which is normally not desired.
2021-05-01 23:28:30 +02:00
6101eaa6b6
Ensure staging flag is not unset
2021-04-28 22:19:10 +02:00
a96ba707cd
Populate list of packages to be re-built also for removed libraries
2021-04-28 21:58:33 +02:00
12050364c8
Handle signature cleanup alongside related package even if package symlink broken
2021-04-27 17:06:47 +02:00
e4755a283b
Prevent running out of memory when signing split packages
...
* Launch only up to 4 gpg processes at a time
* Prevent `gpg: signing failed: Nicht genügend Hauptspeicher verfügbar`
2021-04-17 17:30:18 +02:00
