Commit Graph

268 Commits

Author SHA1 Message Date
Martchus 30a80e7aca Add pkgctl tool 2023-05-28 22:05:02 +02:00
Martchus cf6b90407e Workaround compilation error happening in reflection generation
The latest version of libstdc++ in Arch Linux at this point (13.1.1)
pulls in the following code via `#iterator <iterator>` when
`-std=gnu++2b` is used:

```
    template<forward_iterator _It>
      struct __basic_const_iterator_iter_cat<_It>
{ using iterator_category =
iterator_traits<_It>::iterator_category; };
```

The latest version of Clang in Arch Linux at this point (15.0.7)
complains about a missing `typename` here. To avoid this error
from preventing the build, this changes enables error resilience
for the reflection generation to turn this error into a warning.
2023-05-28 21:05:29 +02:00
Martchus 72d449977f Fix note about deleted build actions in cleanup action 2023-05-28 20:37:45 +02:00
Martchus 55e19d9041 Relax condition for adding `--skippgpcheck`
The `signed` parameter might be used like this:
```
"$pkgname::git+https://github.com/ankitects/anki#tag=${_tag}?signed"
```
So we can't rely on `.git` preceding it.
2023-04-08 19:26:46 +02:00
Martchus cce83f1f9e Allow considering pulling-in further dependencies unexpected
This allows one to review further dependencies if pulled-in automatically
during the build preparation first before conducting the build.
2023-04-08 19:10:59 +02:00
Martchus 86581ec002 Add note in cleanup action stating how many build actions have been deleted 2023-04-08 18:50:46 +02:00
Martchus 5af8de2468 Fix catching exception from `boost::filesystem` in `BuildServiceCleanup` and adapt tests 2023-04-08 18:43:58 +02:00
Martchus 87fd9bb4b1 Fix unqualified calls in `preparebuild.cpp` 2023-04-07 19:30:08 +02:00
Martchus 00edfed48b Improve error handling in `BuildServiceCleanup` 2023-04-07 19:23:13 +02:00
Martchus 836b7da8a3 Invoke `paccache` as part of the build service cleanup 2023-04-07 19:15:16 +02:00
Martchus 08af21ddb5 Fix warnings about unqualified calls in `buildaction.cpp` 2023-04-07 17:55:13 +02:00
Martchus 9702d4f207 Add paths for pacman/paccache to server setup 2023-04-07 17:52:52 +02:00
Martchus e477613a8b Implement preserving chroot for containerized builds as well
This keeps the container around by passing the `DEBUG´ environment variable
recognized by `makechrootpkg`.
2023-02-03 23:06:43 +01:00
Martchus e5f4dc156c Avoid unqualified call to `std::move` 2023-02-03 21:30:42 +01:00
Martchus e14eb17c60 Apply clang-format 2023-01-23 22:42:45 +01:00
Martchus 2fd30f5a0f Adapt tests to boost 1.81 2023-01-23 22:41:01 +01:00
Martchus 31d1025b85 Fix compilation against boost 1.81.0 2023-01-22 22:31:20 +01:00
Martchus 5f25e6e8b7 Avoid unqualified calls to `std::move` 2023-01-22 22:30:07 +01:00
Martchus 599185d3ae Use async locks when invoking custom command 2023-01-17 23:27:36 +01:00
Martchus 80b5094659 Use async locks in repo actions 2023-01-17 22:20:10 +01:00
Martchus bb9da06950 Use async lock to lock repo when adding package 2023-01-17 22:07:00 +01:00
Martchus 4f413e86ba Allow to abort build action immediately while waiting for chroot locks
* Avoid aborted build actions from being stuck in running state while
  waiting until the chroot locks can be acquired; allow aborting action
  immediately instead
* Use async locks when invoking makechrootpkg and split affected function
  accordingly
2023-01-17 18:40:05 +01:00
Martchus b8f211e1d1 Make functions to invoke build async 2023-01-17 18:40:05 +01:00
Martchus 8dfa948e03 Implement helper functions for async global locks 2023-01-17 18:40:05 +01:00
Martchus 036aa2c090 Fix "invalid escape" in regular expression in test suite 2022-12-29 17:33:01 +01:00
Martchus c404a092cc Add async locks
This might be useful to avoid blocking threads in the thread pool just for
waiting on a global lock. It might also be useful to allow stopping build
actions while they're waiting for a lock.
2022-12-04 00:28:47 +01:00
Martchus b1acbee127 Add very basic test for build service cleanup 2022-11-24 23:12:01 +01:00
Martchus 218e1f6b17 Add simple cleanup of old build actions 2022-11-24 21:33:42 +01:00
Martchus 9cf0fed450 Avoid duplicating "Clone of" in task name 2022-11-13 22:44:41 +01:00
Martchus aff7853f94 Set task name of clones 2022-11-13 22:14:29 +01:00
Martchus 962dd0088e Fix computing source package name
The `packageName` (key in build preparation) might not actually be
`pkgbase` when rebuilding a binary package and `pkgbase` changes compared
to what it was in the before. Then `packageName` is still set to the old
`pkgbase` and we need to use the name from the actual source info (which is
the name `pkgbase` value).
2022-11-13 22:09:44 +01:00
Martchus 6e7e0c01e3 Avoid using deprecated OpenSSL functions 2022-11-03 22:34:31 +01:00
Martchus 809bafa6ee Log "Loading …" message when reloading DBs also without force-flag 2022-11-01 14:35:00 +01:00
Martchus a205ffae38 Fix adding last package of current batch already to staging repository 2022-09-20 23:00:36 +02:00
Martchus c7d8e71978 Add staging test with multiple batches 2022-09-15 22:59:18 +02:00
Martchus 598b931e9f Move staging tests into their own block 2022-09-15 22:24:12 +02:00
Martchus 69ee493b8f Fix out-commented check in tests 2022-09-15 22:06:48 +02:00
Martchus bde385ba6b Allow auth without access to secrets (and then just don't access secrets) 2022-07-29 22:07:24 +02:00
Martchus 662c924321 Use `flagenumclass.h` for auth flags 2022-07-29 22:06:40 +02:00
Martchus aece080986 Pass sudo password and GPG passphrase from encrypted file
Not tested yet; so far just an idea to make everything at least a little
more secure
2022-07-25 21:48:24 +02:00
Martchus 255da5b091 Allow passing secrets to build action 2022-07-11 23:33:28 +02:00
Martchus df2b5ba9f6 Allow secrets stored in encrypted files to be loaded from route handlers 2022-07-10 19:47:41 +02:00
Martchus 9c83ba8ea7 Extend tests for conducting build
* Test further error cases
* Improve checks for error messages
* Use better scoping
2022-07-01 19:28:49 +02:00
Martchus 1e7e604917 Fix skipping package build due to failed dependency
Skip package as expected when a dependency failed that was attempted to be
built as last package of the previous batch (and building as far as
possible is enabled).
2022-07-01 19:27:39 +02:00
Martchus 6c1782ba42 Apply clang-format 2022-06-28 16:08:53 +02:00
Martchus 0e3be7ce69 Fix conduct build action from getting stuck when configuring chroot fails
by moving the creation of the process session below configuring the chroot
2022-06-27 12:49:47 +02:00
Martchus 9f5474e128 Fix passing architecture when creating database
When adding a new database when reloading the config at runtime the
architecture must be passed when creating the database. Otherwise we end up
with the new database sharing the internal storage with the x86_64 database
which leads to very confusing behavior.
2022-06-19 23:06:25 +02:00
Martchus 0a536a992a Fix missing include for std::ifstream 2022-06-15 23:31:14 +02:00
Martchus 3b150428ae Allow building with `makecontainerpkg` 2022-06-12 20:42:14 +02:00
Martchus 294dba0e2a Fix URL and show it in the about dialog 2022-06-09 19:50:43 +02:00
Martchus c155176098 Fix missing build date in package search
* Avoid using buildDate of PackageInfo in favor of the same field in
  PackageBase
    * PackageInfo::buildDate should be removed on the next breaking change
* Do not really use the deprecated attribute because it is still used by
  (de)serialization code until it is removed for good
2022-06-08 22:44:58 +02:00
Martchus 94c4b8addc Improve misleading test name 2022-05-31 21:34:20 +02:00
Martchus 55c7c62c7c Speed up returning build actions table by avoiding deserializing whole obj 2022-05-31 20:54:02 +02:00
Martchus 645bb0fe54 Use arch-specific sub-directory within cache dir when conducting build 2022-05-31 20:27:42 +02:00
Martchus f2902f1d08 Apply clang-format 2022-05-31 20:13:21 +02:00
Martchus ecebf9fbde Fix showing arch of packages after optimization
* Move the binary package arch truly into the base package
2022-05-31 20:13:15 +02:00
Martchus b74392c34d Speed up deserialization when serving providing packages as well 2022-05-31 19:40:12 +02:00
Martchus 68d67f543f Refactor to optimize deserialization of base-data
* Break backwards compatibility
* Allow to deserialize only base-data of packages and build
  actions to potentially speed up showing tables
* Speed up package search in many cases by only deserializing
  base-data (unless details are actually wanted)
2022-05-31 19:04:57 +02:00
Martchus ae44624989 Rebuild package DBs as well when fixing DBs 2022-05-15 00:32:46 +02:00
Martchus 40cfe9883c Ensure there are no more leftovers in "clean-repository-" DBs 2022-05-15 00:07:39 +02:00
Martchus f37739ec07 Avoid compilation error with GCC 12
Make operator<< for iteratable objects visible prior to call site like it
is already done in other places.
2022-05-12 21:01:58 +02:00
Martchus 438fccd6cd Avoid fatal warning about potential null pointer deref from GCC 12
It is about code included from Boost.Asio (some io_uring related code) so
let's just not treat it as fatal error.
2022-05-12 20:45:25 +02:00
Martchus 4c98887b02 Fix warning about possibly unused value 2022-05-12 20:33:23 +02:00
Martchus 5a6d68b318 Fix using files DB in CleanRepository 2022-05-07 01:59:14 +02:00
Martchus c2d5e2f749 Use DB path when checking timestamps in repo cleanup
Otherwise it can break if files DBs are used.
2022-05-02 22:23:35 +02:00
Martchus 1950619f21 Add link to UI in root page 2022-05-01 22:21:42 +02:00
Martchus e343b3ca44 Split `ResourceUsage` into `MemoryUsage` and setup specific stats 2022-04-24 18:25:22 +02:00
Martchus 142f6466ae Show resource usage in UI 2022-04-23 19:56:22 +02:00
Martchus a780b26ef6 Log package cache size on startup 2022-04-20 22:47:29 +02:00
Martchus 26caa78956 Reduce memory usage when loading packages
* Avoid creation of map with all archive contents; instead parse packages
  while walking though the archive
* Avoid instantiation of std::string in come cases (using std::string_view)
* Reuse libarchive's archive entry when walking though archive
* Use visitor-based database parser in all places to avoid intermediate big
  array with all package objects
2022-04-20 22:47:10 +02:00
Martchus 2ca6ce1eec Prevent crash in package search when index is broken 2022-04-19 00:24:31 +02:00
Martchus 1f3f0b0df1 Avoid high memory usage when reloading database
Especially when enabling files DBs it is quite problematic to store all
package objects of a database within one big array. This change avoids the
array and instead adds the packages directly. The disadvantage is that
clearing the database isn't as simple anymore.
2022-04-19 00:23:46 +02:00
Martchus edb014011c Make package search more efficient
* Avoid keeping list of all packages in memory by pushing each package
  directly to the JSON array
* Avoid check for database before pushing each package; do it only before
  reaching the next database
* Remove no longer required functions
2022-04-17 23:39:25 +02:00
Martchus 4578b30ea2 Remove hard-coded build directories in test data 2022-04-15 19:58:35 +02:00
Martchus eae8af6571 Avoid big build action IDs by reusing older/lower ones 2022-04-15 19:20:01 +02:00
Martchus e15f65ac16 Extend tests for conducting build 2022-04-07 23:18:14 +02:00
Martchus 08421fa719 Don't rely on local pkg dir when reloading library deps
Since 11f1849fde the cache dir is also set as local pkg dir
so we cannot rely on packages really existing there.
2022-04-07 21:35:05 +02:00
Martchus 7c627e6a12 Use `auto` in `getPackages()` 2022-03-22 21:03:56 +01:00
Martchus 56afe0c710 Fix returning newly retrieved AUR packages 2022-03-22 21:03:01 +01:00
Martchus d163f56abf Log whether `io_uring` is used/configured 2022-03-20 17:37:33 +01:00
Martchus d5e2b5489d Provide filename for artefact downloads 2022-03-15 23:22:34 +01:00
Martchus 69a81f7583 Remove unused function declaration 2022-03-15 22:27:50 +01:00
Martchus 83c7426fce Allow specifying output path for license info 2022-03-15 22:23:34 +01:00
Martchus 13cccd3bc9 Improve creating license info 2022-03-15 01:10:15 +01:00
Martchus f87829ba85 Save state after reloading configuration 2022-03-15 01:07:51 +01:00
Martchus 11f1849fde Keep local pkg paths deduced from pacman config
It makes the setup easier and shouldn't hurt otherwise.
2022-03-15 01:07:36 +01:00
Martchus b0234ff02e Improve MIME-type when serving artefacts 2022-03-15 00:21:08 +01:00
Martchus 59cd593a3b Display template name 2022-03-12 22:46:42 +01:00
Martchus 70a98ffedc Make name of build action to execute custom command shorter 2022-03-12 21:37:59 +01:00
Martchus ac0d64fe3e Allow everyone to see build action details 2022-03-10 23:19:02 +01:00
Martchus 55f081153e Add command to delete invalid build actions from DB 2022-03-07 23:36:48 +01:00
Martchus 7ade757c8d Avoid locking whole config when updating DBs
* Only lock the config for writing the reloading the config file
* Make sure all write operations to the database acquire an "update mutex"
  to ensure only one write operation happens at a time
* Do *not* acquire any additional locks when reading from a database as it
  should be safe to do so even when a write operation happens because
    * LMDB read and write transactions can happen at the same time
    * The package cache has its own mutex anyways
    * Write ops to the package cache try to lock the "update mutex" to
      prevent writing "old" data to the cache during updates
* Make "lastUpdate" atomic to avoid locking the config when accessing it
2022-03-05 18:55:52 +01:00
Martchus afc61bcad6 Speed up AUR updates using `PackageUpdater` 2022-03-05 16:10:33 +01:00
Martchus 8fdf7bfb04 Print certain resource limits on server startup 2022-03-05 12:34:26 +01:00
Martchus 9acd10fa4c Catch `boost::exception`s 2022-03-05 12:34:05 +01:00
Martchus 9b6be51dd3 Allow reloading library dependencies of specific packages 2022-03-04 21:51:25 +01:00
Martchus 4293eb92bf Ignore libs for cross-compilation when checking any packages for binaries
Not really generic yet (e.g. doesn't cover `aarch64-linux-gnu-glibc`) but
good enough for now.
2022-03-04 21:50:56 +01:00
Martchus c633eea607 Add any-packages containing binaries as problem when checking for errors 2022-03-03 23:08:14 +01:00
Martchus c2e7f40767 Fix typos 2022-03-01 00:43:06 +01:00
Martchus 79123e4cd9 Improve error handling of server
* Return with non-zero exit code when an exception occurred
* Handle exceptions when terminating the service
2022-03-01 00:39:50 +01:00