syncthing/lib
Jakob Borg 780fb3bac1
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.

Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.

I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.

This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...

The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)

```
name                         old time/op         new time/op         delta
WalkCaseFakeFS100k/rawfs-8           306ms ± 1%          305ms ± 2%     ~     (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8          579ms ± 5%          557ms ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old B/entry         new B/entry         delta
WalkCaseFakeFS100k/rawfs-8             590 ± 0%            590 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          1.09k ± 0%          0.87k ± 0%  -19.98%  (p=0.000 n=10+10)

name                         old DirNames/entry  new DirNames/entry  delta
WalkCaseFakeFS100k/rawfs-8            0.51 ± 0%           0.51 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           1.02 ± 0%           1.02 ± 0%     ~     (all equal)

name                         old DirNames/op     new DirNames/op     delta
WalkCaseFakeFS100k/rawfs-8           51.2k ± 0%          51.2k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           102k ± 0%           102k ± 0%     ~     (all equal)

name                         old Lstat/entry     new Lstat/entry     delta
WalkCaseFakeFS100k/rawfs-8            3.02 ± 0%           3.02 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           3.02 ± 0%           3.02 ± 0%     ~     (all equal)

name                         old Lstat/op        new Lstat/op        delta
WalkCaseFakeFS100k/rawfs-8            302k ± 0%           302k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           302k ± 0%           302k ± 0%     ~     (all equal)

name                         old allocs/entry    new allocs/entry    delta
WalkCaseFakeFS100k/rawfs-8            15.7 ± 0%           15.7 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           27.5 ± 0%           26.1 ± 0%   -5.09%  (p=0.000 n=10+10)

name                         old ns/entry        new ns/entry        delta
WalkCaseFakeFS100k/rawfs-8           2.02k ± 1%          2.02k ± 2%     ~     (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8          3.83k ± 5%          3.68k ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old alloc/op        new alloc/op        delta
WalkCaseFakeFS100k/rawfs-8          89.2MB ± 0%         89.2MB ± 0%     ~     (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8          164MB ± 0%          131MB ± 0%  -19.97%  (p=0.000 n=10+10)

name                         old allocs/op       new allocs/op       delta
WalkCaseFakeFS100k/rawfs-8           2.38M ± 0%          2.38M ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          4.16M ± 0%          3.95M ± 0%   -5.05%  (p=0.000 n=10+10)
```
2020-09-09 14:38:39 +02:00
..
api all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
assets lib/assets: Allow assets to remain uncompressed (#6661) 2020-05-25 08:51:27 +02:00
beacon lib/beacon: Only send to appropriately flagged interfaces (ref #5957) (#6404) 2020-03-17 09:40:37 +01:00
build build: Actually set build tags (#6866) 2020-07-30 10:58:43 +02:00
config lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
connections lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
db lib/model: Prevent duplicate need count on recalculating metadata (#6964) 2020-09-07 20:19:03 +02:00
dialer lib/dialer: Try dialing without reuse in parallel (fixes #6892) (#6893) 2020-08-18 13:11:12 +02:00
discover lib/connections: Announce LAN addresses by default (fixes #6928) (#6896) 2020-08-25 11:48:14 +02:00
events lib/api, lib/model: Fixes around event request tracking (#6774) 2020-06-25 21:48:48 +02:00
fs lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
ignore gui, lib/ignore: Handle editing ignores with error (fixes #5425) (#6757) 2020-06-18 11:04:00 +02:00
locations lib/db: Add Badger backend (fixes #5910) (#6250) 2020-05-29 13:43:02 +02:00
logger all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
model lib/model: Don't fail over case-conflict on tempfile (fixes #6973) (#6975) 2020-09-09 11:47:14 +02:00
nat lib/nat: Don't hang on draining timer chan (fixes #6908) (#6912) 2020-08-19 15:58:44 +01:00
osutil lib/osutil: Check returned error instead of info (ref #6885) (#6887) 2020-08-12 11:33:56 +02:00
pmp lib/nat: Make service termination faster (#6777) 2020-06-22 09:01:57 +01:00
protocol lib/db, lib/protocol: Never need empty-version entries (fixes #6961) (#6962) 2020-09-07 20:18:25 +02:00
rand lib/rand: Various minor fixes (#6752) 2020-06-17 10:43:58 +02:00
rc test, lib/model: Various integration test updates & improvements (#6956) 2020-09-07 09:35:37 +02:00
relay all: Make all error implementations pointer types (#6726) 2020-06-16 09:27:34 +02:00
scanner lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708) 2020-07-28 11:15:11 +02:00
sha256 Add indirection for large version vectors. (#6376) 2020-05-13 14:28:42 +02:00
signature all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
stats all: Propagate errors from NamespacedKV (#6203) 2019-11-30 13:03:24 +01:00
stun lib: More contextification (#6343) 2020-02-24 21:57:15 +01:00
sync lib/sync: Cleanly fail instead of panic in tests (#6088) 2019-10-16 10:11:11 +02:00
syncthing lib/syncthing: Also cleanup on startup error (#6926) 2020-08-27 15:52:51 +02:00
testutils lib/model, lib/testutils: Test closing a connection on folder restart (#5707) 2019-05-18 08:53:59 +02:00
tlsutil lib/connections, lib/tlsutil: Handle certName in Go 1.15 (fixes #6867) (#6868) 2020-07-30 13:36:11 +02:00
upgrade all: Tweak error creation (#6391) 2020-03-03 22:40:00 +01:00
upnp lib/nat: Make service termination faster (#6777) 2020-06-22 09:01:57 +01:00
ur all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
util all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
versioner Created cleanup functionality for syncthing (#6884) 2020-08-24 12:14:30 +01:00
watchaggregator lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
weakhash lib/weakhash: Fix speed reporting in benchmark (#6470) 2020-03-29 17:07:25 +02:00