syncthing/lib/fs
Jakob Borg 780fb3bac1
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required
efficiency for our walk usecase. This uses less memory.

Specifically, instead of keeping result and child caches for each path
level, only keep a single cached child. In practice our operations are
depth-first, or almost depth-first, and then we retain the same hit
ratio for a smaller cache size.

I improved the benchmark so that it counts the Lstat and DirNames
operations performed, and they do not change significantly. The amount
of allocated memory is reduced by 20% and the walk itself is actually
slightly faster.

This also removes the clear based on number of cached names (as that is
not a thing any more) and the timer based clear (which was unused). This
means we'll retain the last cache state forever until it's cleared by a
write operation, but we did that before too and that state is now a lot
smaller...

The overhead compared to not using a casefs, for our typical "double
walk" (walk the tree then stat everything again) is 2x the dirnames we
would otherwise call, and no overhead on the stats (unchanged from old
implementation)

```
name                         old time/op         new time/op         delta
WalkCaseFakeFS100k/rawfs-8           306ms ± 1%          305ms ± 2%     ~     (p=0.182 n=9+10)
WalkCaseFakeFS100k/casefs-8          579ms ± 5%          557ms ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old B/entry         new B/entry         delta
WalkCaseFakeFS100k/rawfs-8             590 ± 0%            590 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          1.09k ± 0%          0.87k ± 0%  -19.98%  (p=0.000 n=10+10)

name                         old DirNames/entry  new DirNames/entry  delta
WalkCaseFakeFS100k/rawfs-8            0.51 ± 0%           0.51 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           1.02 ± 0%           1.02 ± 0%     ~     (all equal)

name                         old DirNames/op     new DirNames/op     delta
WalkCaseFakeFS100k/rawfs-8           51.2k ± 0%          51.2k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           102k ± 0%           102k ± 0%     ~     (all equal)

name                         old Lstat/entry     new Lstat/entry     delta
WalkCaseFakeFS100k/rawfs-8            3.02 ± 0%           3.02 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           3.02 ± 0%           3.02 ± 0%     ~     (all equal)

name                         old Lstat/op        new Lstat/op        delta
WalkCaseFakeFS100k/rawfs-8            302k ± 0%           302k ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           302k ± 0%           302k ± 0%     ~     (all equal)

name                         old allocs/entry    new allocs/entry    delta
WalkCaseFakeFS100k/rawfs-8            15.7 ± 0%           15.7 ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8           27.5 ± 0%           26.1 ± 0%   -5.09%  (p=0.000 n=10+10)

name                         old ns/entry        new ns/entry        delta
WalkCaseFakeFS100k/rawfs-8           2.02k ± 1%          2.02k ± 2%     ~     (p=0.163 n=9+10)
WalkCaseFakeFS100k/casefs-8          3.83k ± 5%          3.68k ± 1%   -3.77%  (p=0.000 n=10+10)

name                         old alloc/op        new alloc/op        delta
WalkCaseFakeFS100k/rawfs-8          89.2MB ± 0%         89.2MB ± 0%     ~     (p=0.364 n=9+10)
WalkCaseFakeFS100k/casefs-8          164MB ± 0%          131MB ± 0%  -19.97%  (p=0.000 n=10+10)

name                         old allocs/op       new allocs/op       delta
WalkCaseFakeFS100k/rawfs-8           2.38M ± 0%          2.38M ± 0%     ~     (all equal)
WalkCaseFakeFS100k/casefs-8          4.16M ± 0%          3.95M ± 0%   -5.05%  (p=0.000 n=10+10)
```
2020-09-09 14:38:39 +02:00
..
basicfs_copy_range_copyfilerange.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
basicfs_copy_range_duplicateextents.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
basicfs_copy_range_ioctl.go lib/fs: Disable ioctl on ppc (fixes #6898) (#6901) 2020-08-19 07:56:35 +02:00
basicfs_copy_range_sendfile.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
basicfs_copy_range.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
basicfs_fileinfo_unix.go lib/fs: Treat Windows junctions as normal directories (#6606) 2020-05-13 21:46:24 +02:00
basicfs_fileinfo_windows.go lib/fs: Treat Windows junctions as normal directories (#6606) 2020-05-13 21:46:24 +02:00
basicfs_lstat_broken.go lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
basicfs_lstat_regular.go lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
basicfs_lstat_windows.go lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
basicfs_test.go lib/fs: Treat Windows junctions as normal directories (#6606) 2020-05-13 21:46:24 +02:00
basicfs_unix.go lib/fs: Improve root check (#6033) 2019-09-29 23:38:11 +08:00
basicfs_watch_errors_linux.go lib/fs: Fix and update error about inotify watch limit (fixes #4833) (#4835) 2018-03-23 12:56:38 +01:00
basicfs_watch_errors_others.go all: Add filesystem notification support 2017-10-20 14:52:55 +00:00
basicfs_watch_eventtypes_fen.go vendor: github.com/Zillode/notify is now github.com/syncthing/notify (#4813) 2018-03-14 14:48:22 +01:00
basicfs_watch_eventtypes_inotify.go vendor: github.com/Zillode/notify is now github.com/syncthing/notify (#4813) 2018-03-14 14:48:22 +01:00
basicfs_watch_eventtypes_kqueue.go lib/fs: Fix kqueue event list (fixes #5308) (#5885) 2019-07-23 14:11:15 +02:00
basicfs_watch_eventtypes_other.go vendor: github.com/Zillode/notify is now github.com/syncthing/notify (#4813) 2018-03-14 14:48:22 +01:00
basicfs_watch_eventtypes_readdcw.go lib/fs: Catch size-preserving changes on windows (fixes #5050) (#5056) 2018-07-09 18:29:22 +01:00
basicfs_watch_test.go lib/fs: Fix WatchRename test for FreeBSD (fixes #6613) 2020-08-03 23:24:01 +02:00
basicfs_watch_unsupported.go lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734) 2019-05-25 20:08:26 +01:00
basicfs_watch.go lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
basicfs_windows_test.go lib/fs: Check events against both the user and eval root (#6013) 2019-09-22 08:03:22 +01:00
basicfs_windows.go all: Tweak error creation (#6391) 2020-03-03 22:40:00 +01:00
basicfs.go lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
casefs_test.go lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
casefs.go lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
copyrangemethod.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
copyrangemethod.pb.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
debug_symlink_unix.go lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
debug_symlink_windows.go lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708) 2020-07-28 11:15:11 +02:00
debug.go all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
errorfs.go lib/{fs,scanner}: gofmt from Go 1.14 (#6509) 2020-04-07 09:31:29 +02:00
fakefs_test.go lib/fs, lib/model: Rewrite RecvOnly tests (#6318) 2020-02-12 07:47:05 +01:00
fakefs.go lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
filesystem_copy_range_allwithfallback.go lib/fs: Add support for Windows duplicate extents (#6764) 2020-06-18 22:32:26 +01:00
filesystem_copy_range_standard.go all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
filesystem_copy_range_test.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
filesystem_copy_range.go all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
filesystem_test.go lib/fs: Improve IsParent (#5658) 2019-04-22 11:12:32 +02:00
filesystem.go lib/fs: More efficient casefs cache (#6974) 2020-09-09 14:38:39 +02:00
folding_test.go lib/fs: Add case-insensitive fakefs (#6074) 2019-11-29 09:17:42 +01:00
folding.go lib/fs: Add case insensitivity to MtimeFS 2017-11-17 12:10:16 +00:00
logfs.go lib/fs, lib/model: Add error channel to Watch to avoid panics (fixes #5697) (#5734) 2019-05-25 20:08:26 +01:00
mtimefs_test.go all: Propagate errors from NamespacedKV (#6203) 2019-11-30 13:03:24 +01:00
mtimefs.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
tempname_test.go lib: Folder marker is now a folder 2017-09-02 05:52:38 +00:00
tempname.go all: Revert the underscore sillyness 2019-02-02 12:16:27 +01:00
types.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
types.pb.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
util_test.go lib/versioner: Restore for all versioners, cross-device support (#5514) 2019-04-28 23:30:16 +01:00
util.go all: Cleanups enabled by Go 1.12 2019-11-10 10:16:10 +01:00
walkfs_test.go lib/config, lib/fs: Make junction behaviour configurable (ref #6606) (#6907) 2020-08-19 19:58:51 +02:00
walkfs.go lib/fs: Pass infinite recursion error on instead of warning (#6846) 2020-07-22 22:10:24 +02:00