Martchus
ecebf9fbde
Fix showing arch of packages after optimization
...
* Move the binary package arch truly into the base package
2022-05-31 20:13:15 +02:00
Martchus
68d67f543f
Refactor to optimize deserialization of base-data
...
* Break backwards compatibility
* Allow to deserialize only base-data of packages and build
actions to potentially speed up showing tables
* Speed up package search in many cases by only deserializing
base-data (unless details are actually wanted)
2022-05-31 19:04:57 +02:00
Martchus
40cfe9883c
Ensure there are no more leftovers in "clean-repository-" DBs
2022-05-15 00:07:39 +02:00
Martchus
5a6d68b318
Fix using files DB in CleanRepository
2022-05-07 01:59:14 +02:00
Martchus
c2d5e2f749
Use DB path when checking timestamps in repo cleanup
...
Otherwise it can break if files DBs are used.
2022-05-02 22:23:35 +02:00
Martchus
26caa78956
Reduce memory usage when loading packages
...
* Avoid creation of map with all archive contents; instead parse packages
while walking though the archive
* Avoid instantiation of std::string in come cases (using std::string_view)
* Reuse libarchive's archive entry when walking though archive
* Use visitor-based database parser in all places to avoid intermediate big
array with all package objects
2022-04-20 22:47:10 +02:00
Martchus
1f3f0b0df1
Avoid high memory usage when reloading database
...
Especially when enabling files DBs it is quite problematic to store all
package objects of a database within one big array. This change avoids the
array and instead adds the packages directly. The disadvantage is that
clearing the database isn't as simple anymore.
2022-04-19 00:23:46 +02:00
Martchus
08421fa719
Don't rely on local pkg dir when reloading library deps
...
Since 11f1849fde
the cache dir is also set as local pkg dir
so we cannot rely on packages really existing there.
2022-04-07 21:35:05 +02:00
Martchus
d5e2b5489d
Provide filename for artefact downloads
2022-03-15 23:22:34 +01:00
Martchus
69a81f7583
Remove unused function declaration
2022-03-15 22:27:50 +01:00
Martchus
83c7426fce
Allow specifying output path for license info
2022-03-15 22:23:34 +01:00
Martchus
13cccd3bc9
Improve creating license info
2022-03-15 01:10:15 +01:00
Martchus
f87829ba85
Save state after reloading configuration
2022-03-15 01:07:51 +01:00
Martchus
59cd593a3b
Display template name
2022-03-12 22:46:42 +01:00
Martchus
70a98ffedc
Make name of build action to execute custom command shorter
2022-03-12 21:37:59 +01:00
Martchus
55f081153e
Add command to delete invalid build actions from DB
2022-03-07 23:36:48 +01:00
Martchus
7ade757c8d
Avoid locking whole config when updating DBs
...
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
should be safe to do so even when a write operation happens because
* LMDB read and write transactions can happen at the same time
* The package cache has its own mutex anyways
* Write ops to the package cache try to lock the "update mutex" to
prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
2022-03-05 18:55:52 +01:00
Martchus
9b6be51dd3
Allow reloading library dependencies of specific packages
2022-03-04 21:51:25 +01:00
Martchus
4293eb92bf
Ignore libs for cross-compilation when checking any packages for binaries
...
Not really generic yet (e.g. doesn't cover `aarch64-linux-gnu-glibc`) but
good enough for now.
2022-03-04 21:50:56 +01:00
Martchus
c633eea607
Add any-packages containing binaries as problem when checking for errors
2022-03-03 23:08:14 +01:00
Martchus
c2e7f40767
Fix typos
2022-03-01 00:43:06 +01:00
Martchus
2071327a71
Fix building packages using signed git repositories as source
2022-02-25 23:09:16 +01:00
Martchus
ff7f039519
Ensure problems of repos with different archs but the same name aren't mixed up
2022-02-22 22:50:51 +01:00
Martchus
382169ab6e
Fix varoius build action related problems
...
* Fix crashes in some situations
* Fix aborting reloading library dependencies
2022-02-22 22:49:40 +01:00
Martchus
5c62014b3b
Fix variable name in fallback code when BOOST_ASIO_HAS_FILE
2022-02-22 20:22:00 +01:00
Martchus
839a4b0831
Fix wrong variable name in `buildactionlivestreaming.cpp`
2022-02-22 00:05:35 +01:00
Martchus
8ea2f5fd62
Reintroduce cache file as it is still required for "lastUpdate" for DBs
...
Maybe it makes more sense to use LMDB for this as well. On the other hand
it is an easy way of keeping the remaining state persistent.
2022-02-20 20:17:42 +01:00
Martchus
9c4f0277bd
Unlock output lock early
...
The following things are read-only or guarded by other mutex.
2022-02-20 19:51:46 +01:00
Martchus
bdbfd42e5e
Avoid package object being reused after 3d7310af3
2022-02-20 19:50:44 +01:00
Martchus
334bb1faa9
Use io_uring for async file I/O
2022-02-20 18:44:07 +01:00
Martchus
8115721913
Fix handling error computing db order when reloading dependencies
2022-02-20 00:18:49 +01:00
Martchus
93afb3883d
Write build action "output" log to a logfile like for sub-processes
2022-02-20 00:11:23 +01:00
Martchus
b3b396df5f
Allow ignoring copying errors due to existing files when moving packages
2022-02-16 19:47:25 +01:00
Martchus
557fd1a738
lmdb: Use lmdb to store build actions
2022-02-13 21:04:54 +01:00
Martchus
e582d5f406
Remove useless blank line
2022-02-03 22:07:22 +01:00
Martchus
66f59fecb9
Make storing lib deps more efficient and allow for intermediate reads
2022-01-31 21:32:07 +01:00
Martchus
218dfecf56
Avoid loading databases when modification date is <= last update
...
* Do HTTP head request first when loading database from mirror to avoid
downloading the full database all the time
* Use the last modification date of the local database file because with
the persistent storage even local database reloads became a bit expensive
2022-01-25 00:13:10 +01:00
Martchus
f088e54c61
lmdb: Use package cache when querying packages for dependencies
...
* Use package cache when querying packages for dependencies or library
names
* Avoid manual instantiations of `std::shared_ptr<Package>`
2022-01-21 20:35:43 +01:00
Martchus
e44cbf94e5
lmdb: Enable versioning for objects we'd like to store in the DB
2022-01-18 22:48:34 +01:00
Martchus
2ffa6629c8
lmdb: Use lmdb to store packages and dependency indices
2022-01-18 21:50:53 +01:00
Martchus
0a2c41ee0c
Fix shadowing warning
2021-12-07 22:48:22 +01:00
Martchus
567c3aaf56
Support "trunk" subdirectory for using official svntogit repos as source
2021-11-16 15:00:19 +01:00
Martchus
0f7718c934
Locate PKGBUILDs locally also without variant dir
...
This makes it possible to checkout
https://github.com/archlinux/svntogit-packages.git and
https://github.com/archlinux/svntogit-community.git locally to build
official packages.
2021-11-14 23:37:33 +01:00
Martchus
6f652922cd
Add lookup for repo-specific config file path
...
This could be useful to create another repository with packages that are
built using different compile flags, e.g. optimized for certain processors.
2021-11-14 22:59:46 +01:00
Martchus
2adc80e5fc
Override existing symlinks when adding signatures to repo
2021-11-03 16:32:38 +01:00
Martchus
893a8473ad
Override existing symlinks when adding packages to repo
2021-11-03 16:06:16 +01:00
Martchus
755e1958d3
Allow overriding existing files in repository
...
When `repo-add` fails but files could be copied before refusing to override
is actually very annoying because one has to delete the files before
restarting the build task.
2021-10-07 18:12:44 +02:00
Martchus
672f3b4e97
Fix staging check for split packages
...
* Consider all split packages when skipping an affected package
2021-09-30 22:23:54 +02:00
Martchus
a93924bb3a
Fix cleanup of signatures
...
* Record full path for the deletion of orphaned signatures
* Test for the existence of the signature symlink itself and not the target
because the symlink target (in the any directory) might not exist anymore
but the symlink should be removed in any case
2021-08-29 00:32:19 +02:00
Martchus
1cc8a3d6db
Add warnings about missing DLLs to the build action's result
2021-08-28 15:06:24 +02:00
Martchus
2ad4a39aea
Handle case when a split package overrides the archs from the base
2021-08-28 00:25:09 +02:00
Martchus
075f539480
Avoid copying strings when splitting lock names for custom command
2021-07-25 00:41:46 +02:00
Martchus
bbc7c3a533
Ignore empty packages within the cache dir
...
Those are most likely leftovers from previous failed download attempts and
can be ignored. (Existing files are overridden.)
2021-07-17 23:58:37 +02:00
Martchus
e5f4057c61
Use BufferSearch from c++utilities
2021-07-16 16:01:23 +02:00
Martchus
909f7bfab0
Use default package download size limit if it is set to zero
2021-07-14 15:23:18 +02:00
Martchus
e0a913b16c
Apply clang-format
2021-07-13 00:41:49 +02:00
Martchus
c05a581322
Increase the max. download size for packages and allow configuring it
2021-07-12 15:00:43 +02:00
Martchus
b5ca815e4c
Allow excluding packages via regex when reloading library dependencies
...
This recently failed because the cuda package exceeded the max. body size.
I suppose it makes generally sense to be able to exclude packages as I
don't need that package anyways.
2021-07-07 19:14:03 +02:00
Martchus
49aebdd8a5
Fix typos found via `codespell --skip .git -w`
2021-07-03 19:59:57 +02:00
Martchus
8b70cc5f8e
Clear added/removed provides from previous builds
2021-06-22 00:02:41 +02:00
Martchus
f685fddd20
Add `.pid`-equivalent for Boost.Asio's Windows-executor
2021-06-22 00:01:55 +02:00
Martchus
0a55412625
Remove unused #include
2021-06-21 23:58:34 +02:00
Martchus
d9203689a0
Fix param for PrepareBuildFlags::ResetChrootSettings
2021-06-04 16:47:24 +02:00
Martchus
9359004a10
Allow resetting chroot settings as well when preparing build
2021-05-16 16:49:09 +02:00
Martchus
efbcef5e6e
Fix error handling when makechrootpkg can not be invoked
2021-05-16 13:49:57 +02:00
Martchus
05ff6cd00f
Do not set packages in conduct build actions when creating it form task
...
Usually this doesn't make sense because we want to build all packages which
were determined in the previous prepare action. When specifying the
packages again in the conduct build action we'd skip packages which are
pulled into the build as dependencies which is normally not desired.
2021-05-01 23:28:30 +02:00
Martchus
6101eaa6b6
Ensure staging flag is not unset
2021-04-28 22:19:10 +02:00
Martchus
a96ba707cd
Populate list of packages to be re-built also for removed libraries
2021-04-28 21:58:33 +02:00
Martchus
12050364c8
Handle signature cleanup alongside related package even if package symlink broken
2021-04-27 17:06:47 +02:00
Martchus
e4755a283b
Prevent running out of memory when signing split packages
...
* Launch only up to 4 gpg processes at a time
* Prevent `gpg: signing failed: Nicht genügend Hauptspeicher verfügbar`
2021-04-17 17:30:18 +02:00
Martchus
56ad086c0c
Start the next job only if all start after actions were successful
2021-04-16 14:06:18 +02:00
Martchus
ff1e955bde
Consider concurrent flag when starting build actions from task
...
This allows concurrent build actions within the same task.
2021-04-05 12:10:42 +02:00
Martchus
7f0ec081c6
Allow specifying build tasks with more complicated build actions
...
The new data structures (e.g. concurrent flag) aren't actually used yet.
2021-04-02 00:14:00 +02:00
Martchus
6f2c6cabc2
cli: Add more build action operations
2021-03-27 21:37:28 +01:00
Martchus
fa969270c1
Fix warnings
2021-03-22 15:08:41 +01:00
Martchus
ac7f38e8fa
Do not access moved-from object in CleanRepository::run()
2021-03-16 15:36:57 +01:00
Martchus
9581716d8b
Delete signature in the src repo when moving package
2021-03-16 15:16:03 +01:00
Martchus
307322ab11
Handle signatures when cleaning up repository
2021-03-15 19:22:28 +01:00
Martchus
1c75e8f957
Allow signing packages manually after the build
...
Simply adding `--sign` to the `makepkg` flags doesn't work because it would
require setting up GPG within the chroot environment (of `makechrootpkg`).
When debugging it is anyways annoying that `makepkg` sends the `gpg` output
to `/dev/null`. This way the logs are preserved.
2021-03-14 21:53:51 +01:00
Martchus
45bf4fa234
Handle signature file when adding/removing/moving/checking/cleaning packages
2021-03-14 01:53:40 +01:00
Martchus
94b39f55dc
Lock working copy directory as well (as root directory)
2021-03-06 17:08:04 +01:00
Martchus
29c0741984
Allow registration of multiple data handlers
2021-03-06 16:44:42 +01:00
Martchus
cbfa8d8298
Use locks when checking/cleaning repository
2021-03-05 23:11:48 +01:00
Martchus
556a749362
Improve lock-related code in conductbuild.cpp
2021-03-02 18:30:40 +01:00
Martchus
f147dc3179
Fix locks for global resources by implementing a mutex not tied to a thread
...
* Can not use a normal mutex because we don't want to tie the resources to
a specific thread (and instead e.g. to a build action which might not be
executed by a single thread)
* A semaphore would do that but libstdc++ only supports it as of GCC 11 and
besides it wouldn't distinguish between shared and exclusive locking
2021-02-26 01:39:21 +01:00
Martchus
930d449333
Fix handling locks in ConductBuild
2021-02-25 23:20:07 +01:00
Martchus
0760860c6d
Log lock acquisitions/releases
2021-02-25 22:12:56 +01:00
Martchus
aae5bf81f3
Allow specifying locks when executing custom command
2021-02-23 20:57:07 +01:00
Martchus
b7f27cb0a0
Use locks to prevent multiple actions accessing the same db files and chroot dirs
2021-02-22 23:44:06 +01:00
Martchus
bc993f1d78
Add locking system so build actions can acquire named locks
2021-02-20 23:25:31 +01:00
Martchus
120fbf993b
Fix param for ignoring dependencies
2021-02-20 12:25:08 +01:00
Martchus
f040418bdd
Avoid checking for failed dependencies when processing the last package of batch
2021-02-13 12:54:01 +01:00
Martchus
97001740e2
Do enable staging in the current batch when processing the very first package
...
Likely not really relevant because the function should only be called when
processing the 2nd or further packages. However, this way it seems a little
bit less error prone.
2021-02-13 12:52:54 +01:00
Martchus
b556c39cd7
Skip the current package when checking for failed dependencies
...
We usually wouldn't expect a package to depend on itself but the different
packages build as part of one split-package might do.
2021-02-13 12:30:13 +01:00
Martchus
3fe15fb081
Add missing newline when logging failed makechrootpkg invocation
2021-02-13 12:28:42 +01:00
Martchus
5127ca351f
Allow ignoring certain dependencies/libraries when checking for problems
2021-02-08 23:54:03 +01:00
Martchus
b06a1dfcd2
Allow considering the regular package when checking for updates
2021-02-08 23:54:03 +01:00
Martchus
6b274ccc88
Improve log messages
2021-02-08 23:54:03 +01:00
Martchus
17ab860c8b
Don't log all build-action specific output to stderr as well
2021-02-08 23:54:03 +01:00
Martchus
cfdf980511
Avoid constructing extra string in ConductBuild::dumpBuildProgress()
2021-02-08 23:54:03 +01:00