66f59fecb9
Make storing lib deps more efficient and allow for intermediate reads
2022-01-31 21:32:07 +01:00
45922b47ec
lmdb: Allow configuring cache limit
2022-01-26 00:41:53 +01:00
d4d187463a
Fix problems with global locks
...
* Aquire the mutex before setting `m_exclusivelyOwned = false`; otherwise
it might be set after `lock()` has checked it but before `lock()` has
entered `wait()` leading to a deadlock
* Check state again after `wait()` because it may also be unblocked
"spuriously" so it isn't guaranteed that the state will have actually
changed after unblocking
2022-01-25 22:49:20 +01:00
218dfecf56
Avoid loading databases when modification date is <= last update
...
* Do HTTP head request first when loading database from mirror to avoid
downloading the full database all the time
* Use the last modification date of the local database file because with
the persistent storage even local database reloads became a bit expensive
2022-01-25 00:13:10 +01:00
300f74fb10
lmdb: Open LMDB file after restoring cache so DBs added from cache have the storage initialized
2022-01-25 00:01:59 +01:00
1c820fc2f6
lmdb: Fix storage initialization when config file cannot be loaded
2022-01-23 19:47:50 +01:00
f088e54c61
lmdb: Use package cache when querying packages for dependencies
...
* Use package cache when querying packages for dependencies or library
names
* Avoid manual instantiations of `std::shared_ptr<Package>`
2022-01-21 20:35:43 +01:00
e44cbf94e5
lmdb: Enable versioning for objects we'd like to store in the DB
2022-01-18 22:48:34 +01:00
2ffa6629c8
lmdb: Use lmdb to store packages and dependency indices
2022-01-18 21:50:53 +01:00
0a2c41ee0c
Fix shadowing warning
2021-12-07 22:48:22 +01:00
567c3aaf56
Support "trunk" subdirectory for using official svntogit repos as source
2021-11-16 15:00:19 +01:00
0f7718c934
Locate PKGBUILDs locally also without variant dir
...
This makes it possible to checkout
https://github.com/archlinux/svntogit-packages.git and
https://github.com/archlinux/svntogit-community.git locally to build
official packages.
2021-11-14 23:37:33 +01:00
6f652922cd
Add lookup for repo-specific config file path
...
This could be useful to create another repository with packages that are
built using different compile flags, e.g. optimized for certain processors.
2021-11-14 22:59:46 +01:00
2adc80e5fc
Override existing symlinks when adding signatures to repo
2021-11-03 16:32:38 +01:00
893a8473ad
Override existing symlinks when adding packages to repo
2021-11-03 16:06:16 +01:00
755e1958d3
Allow overriding existing files in repository
...
When `repo-add` fails but files could be copied before refusing to override
is actually very annoying because one has to delete the files before
restarting the build task.
2021-10-07 18:12:44 +02:00
672f3b4e97
Fix staging check for split packages
...
* Consider all split packages when skipping an affected package
2021-09-30 22:23:54 +02:00
ace478be22
Fix tests when test working directory is on another device
...
* Fallback to copying if hard link creation fails due to cross device link
2021-09-30 22:22:22 +02:00
a93924bb3a
Fix cleanup of signatures
...
* Record full path for the deletion of orphaned signatures
* Test for the existence of the signature symlink itself and not the target
because the symlink target (in the any directory) might not exist anymore
but the symlink should be removed in any case
2021-08-29 00:32:19 +02:00
c68202bdc1
Add test for repository cleanup
2021-08-29 00:29:26 +02:00
1cc8a3d6db
Add warnings about missing DLLs to the build action's result
2021-08-28 15:06:24 +02:00
2ad4a39aea
Handle case when a split package overrides the archs from the base
2021-08-28 00:25:09 +02:00
95ae827c74
Fix applying relative working directory multiple times
2021-07-31 14:55:49 +02:00
1ce736d66f
Fix passing the body limit to nested cachePackages()
calls
2021-07-25 01:25:30 +02:00
075f539480
Avoid copying strings when splitting lock names for custom command
2021-07-25 00:41:46 +02:00
f3ec908bcc
Hold lock for named lock table not while acquiring named lock
...
Otherwise all other attempts to acquire named locks are blocked. It should
be ok because references are not invalidated when inserting/accessing
elements of an `std::unique_map`. When cleaning locks up elements are
erased, though. Hence an additional cleanup lock is required to prevent
acquiring a named lock which has already been cleaned up.
2021-07-25 00:40:39 +02:00
bbc7c3a533
Ignore empty packages within the cache dir
...
Those are most likely leftovers from previous failed download attempts and
can be ignored. (Existing files are overridden.)
2021-07-17 23:58:37 +02:00
e5f4057c61
Use BufferSearch from c++utilities
2021-07-16 16:01:23 +02:00
bc8a0549b2
Log package download limit
2021-07-14 15:27:49 +02:00
909f7bfab0
Use default package download size limit if it is set to zero
2021-07-14 15:23:18 +02:00
e0a913b16c
Apply clang-format
2021-07-13 00:41:49 +02:00
c05a581322
Increase the max. download size for packages and allow configuring it
2021-07-12 15:00:43 +02:00
a2fbba9713
Allow constructing BadRequest from std::string
2021-07-07 19:46:43 +02:00
b5ca815e4c
Allow excluding packages via regex when reloading library dependencies
...
This recently failed because the cuda package exceeded the max. body size.
I suppose it makes generally sense to be able to exclude packages as I
don't need that package anyways.
2021-07-07 19:14:03 +02:00
d006c9ce4f
Fix error handling when instantiating std::regex
2021-07-07 19:09:32 +02:00
17c5f50440
Do not exit process if config values are invalid
...
This is not a good idea when re-reading the config after startup.
2021-07-07 19:08:47 +02:00
49aebdd8a5
Fix typos found via codespell --skip .git -w
2021-07-03 19:59:57 +02:00
8b70cc5f8e
Clear added/removed provides from previous builds
2021-06-22 00:02:41 +02:00
f685fddd20
Add .pid
-equivalent for Boost.Asio's Windows-executor
2021-06-22 00:01:55 +02:00
0a55412625
Remove unused #include
2021-06-21 23:58:34 +02:00
d9203689a0
Fix param for PrepareBuildFlags::ResetChrootSettings
2021-06-04 16:47:24 +02:00
3a797bc8b6
Adapt test to fixed behavior of splitString*() to preserve last empty part
2021-05-16 19:43:16 +02:00
9359004a10
Allow resetting chroot settings as well when preparing build
2021-05-16 16:49:09 +02:00
efbcef5e6e
Fix error handling when makechrootpkg can not be invoked
2021-05-16 13:49:57 +02:00
b9c7f05ef1
Add missing include for condition_variable
2021-05-15 21:04:21 +02:00
05ff6cd00f
Do not set packages in conduct build actions when creating it form task
...
Usually this doesn't make sense because we want to build all packages which
were determined in the previous prepare action. When specifying the
packages again in the conduct build action we'd skip packages which are
pulled into the build as dependencies which is normally not desired.
2021-05-01 23:28:30 +02:00
6101eaa6b6
Ensure staging flag is not unset
2021-04-28 22:19:10 +02:00
a96ba707cd
Populate list of packages to be re-built also for removed libraries
2021-04-28 21:58:33 +02:00
12050364c8
Handle signature cleanup alongside related package even if package symlink broken
2021-04-27 17:06:47 +02:00
e4755a283b
Prevent running out of memory when signing split packages
...
* Launch only up to 4 gpg processes at a time
* Prevent `gpg: signing failed: Nicht genügend Hauptspeicher verfügbar`
2021-04-17 17:30:18 +02:00
56ad086c0c
Start the next job only if all start after actions were successful
2021-04-16 14:06:18 +02:00
8e904ebfbf
Fix starting first build action
...
* Prevent endless recursion
* Consider starting parallel sequence also as having started an action to
prevent starting also actions which should only start after that parallel
sequence
2021-04-16 14:05:25 +02:00
80c99cb179
Support receiving a body as well when handling chunks individually
2021-04-14 17:56:29 +02:00
da9ddc2079
Apply clang-format
2021-04-14 17:56:29 +02:00
e47edcc09e
Allow live-streaming via CLI
2021-04-12 14:03:47 +02:00
338674a9e6
Fix returning output when no offset present
2021-04-05 21:11:13 +02:00
7ab50a121f
Split routes.cpp
2021-04-05 20:02:00 +02:00
b73bee87b5
Add test for posting build actions from pre-defined build task
2021-04-05 19:49:04 +02:00
ff1e955bde
Consider concurrent flag when starting build actions from task
...
This allows concurrent build actions within the same task.
2021-04-05 12:10:42 +02:00
7f0ec081c6
Allow specifying build tasks with more complicated build actions
...
The new data structures (e.g. concurrent flag) aren't actually used yet.
2021-04-02 00:14:00 +02:00
6f2c6cabc2
cli: Add more build action operations
2021-03-27 21:37:28 +01:00
bfbde3f957
Refactor client code
2021-03-27 19:55:24 +01:00
fa969270c1
Fix warnings
2021-03-22 15:08:41 +01:00
ac7f38e8fa
Do not access moved-from object in CleanRepository::run()
2021-03-16 15:36:57 +01:00
9581716d8b
Delete signature in the src repo when moving package
2021-03-16 15:16:03 +01:00
307322ab11
Handle signatures when cleaning up repository
2021-03-15 19:22:28 +01:00
1c75e8f957
Allow signing packages manually after the build
...
Simply adding `--sign` to the `makepkg` flags doesn't work because it would
require setting up GPG within the chroot environment (of `makechrootpkg`).
When debugging it is anyways annoying that `makepkg` sends the `gpg` output
to `/dev/null`. This way the logs are preserved.
2021-03-14 21:53:51 +01:00
45bf4fa234
Handle signature file when adding/removing/moving/checking/cleaning packages
2021-03-14 01:53:40 +01:00
94b39f55dc
Lock working copy directory as well (as root directory)
2021-03-06 17:08:04 +01:00
29c0741984
Allow registration of multiple data handlers
2021-03-06 16:44:42 +01:00
cbfa8d8298
Use locks when checking/cleaning repository
2021-03-05 23:11:48 +01:00
556a749362
Improve lock-related code in conductbuild.cpp
2021-03-02 18:30:40 +01:00
4416922397
Fix clearing locks
2021-03-02 18:30:15 +01:00
f147dc3179
Fix locks for global resources by implementing a mutex not tied to a thread
...
* Can not use a normal mutex because we don't want to tie the resources to
a specific thread (and instead e.g. to a build action which might not be
executed by a single thread)
* A semaphore would do that but libstdc++ only supports it as of GCC 11 and
besides it wouldn't distinguish between shared and exclusive locking
2021-02-26 01:39:21 +01:00
930d449333
Fix handling locks in ConductBuild
2021-02-25 23:20:07 +01:00
0760860c6d
Log lock acquisitions/releases
2021-02-25 22:12:56 +01:00
aae5bf81f3
Allow specifying locks when executing custom command
2021-02-23 20:57:07 +01:00
b7f27cb0a0
Use locks to prevent multiple actions accessing the same db files and chroot dirs
2021-02-22 23:44:06 +01:00
bc993f1d78
Add locking system so build actions can acquire named locks
2021-02-20 23:25:31 +01:00
120fbf993b
Fix param for ignoring dependencies
2021-02-20 12:25:08 +01:00
046f963f4b
Improve error handling when parsing config, fix specifying IP address
2021-02-18 15:21:05 +01:00
f040418bdd
Avoid checking for failed dependencies when processing the last package of batch
2021-02-13 12:54:01 +01:00
97001740e2
Do enable staging in the current batch when processing the very first package
...
Likely not really relevant because the function should only be called when
processing the 2nd or further packages. However, this way it seems a little
bit less error prone.
2021-02-13 12:52:54 +01:00
b556c39cd7
Skip the current package when checking for failed dependencies
...
We usually wouldn't expect a package to depend on itself but the different
packages build as part of one split-package might do.
2021-02-13 12:30:13 +01:00
3fe15fb081
Add missing newline when logging failed makechrootpkg invocation
2021-02-13 12:28:42 +01:00
ab6b940d8e
Acknowledge restoring config from cache file correctly
2021-02-09 14:59:40 +01:00
5127ca351f
Allow ignoring certain dependencies/libraries when checking for problems
2021-02-08 23:54:03 +01:00
b06a1dfcd2
Allow considering the regular package when checking for updates
2021-02-08 23:54:03 +01:00
3894d911d7
Fix posting build action to start it later; add basic test for the route
2021-02-08 23:54:03 +01:00
0c8f0b04c2
Experiment with Boost.Beast to use version of async_accept accepting IO ctx
...
This would make more sense instead of using the (copy of the) same socket
object all the time.
2021-02-08 23:54:03 +01:00
6b274ccc88
Improve log messages
2021-02-08 23:54:03 +01:00
17ab860c8b
Don't log all build-action specific output to stderr as well
2021-02-08 23:54:03 +01:00
cfdf980511
Avoid constructing extra string in ConductBuild::dumpBuildProgress()
2021-02-08 23:54:03 +01:00
aa356944da
Log process IDs of started sub processes
2021-02-08 23:54:02 +01:00
b647fc79ab
Fix parameter of "Keep pkgrel-epoch" option
2021-02-08 23:54:02 +01:00
c1554f4c87
Initial import
2021-02-08 23:53:56 +01:00