syncthing/lib/fs
Jakob Borg c1ec9a8826
lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
..
basicfs_copy_range_copyfilerange.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_copy_range_duplicateextents.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_copy_range_ioctl.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_copy_range_sendfile.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_copy_range.go lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
basicfs_fileinfo_bsdish.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs_fileinfo_linuxish.go lib/fs: Ignore inode change time on Android (#9177) 2023-10-21 08:24:29 +02:00
basicfs_fileinfo_unix.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_fileinfo_windows.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs_lstat_broken.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_lstat_regular.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_lstat_windows.go lib/fs: Properly handle Windows deduplicated files (fixes #9120) (#9168) 2023-10-11 14:40:55 +02:00
basicfs_platformdata_unix.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs_platformdata_windows.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs_test.go lib/fs: Let xattr test avoid non-test attributes (fixes #8601) (#8628) 2022-11-03 11:57:30 +01:00
basicfs_unix.go lib/fs: Try to remove read only Windows files (fixes #3744) (#8650) 2022-11-07 21:33:17 +01:00
basicfs_watch_errors_linux.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_watch_errors_others.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_watch_eventtypes_darwin.go lib/fs: Handle permission change events on macos (fixes #7924) (#8150) 2022-01-30 17:21:21 +01:00
basicfs_watch_eventtypes_fen.go lib/fs: Do not follow symlinks in watcher on solaris (fixes #8020) (#8223) 2022-03-24 08:36:43 +01:00
basicfs_watch_eventtypes_inotify.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_watch_eventtypes_kqueue.go lib/fs, lib/model: Add warning about kqueue resource usage (fixes #7855) (#8249) 2022-04-05 21:32:06 +02:00
basicfs_watch_eventtypes_other.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_watch_eventtypes_readdcw.go all: Update build constraints to Go 1.17 style (#7894) 2021-08-17 10:10:41 +02:00
basicfs_watch_notkqueue.go lib/fs, lib/model: Add warning about kqueue resource usage (fixes #7855) (#8249) 2022-04-05 21:32:06 +02:00
basicfs_watch_test.go all: Add build constants for runtime.GOOS comparisons (#8442) 2022-07-28 19:36:39 +02:00
basicfs_watch_unsupported.go lib/fs: Watching is unsupported on android/amd64 (fixes #8709) (#8710) 2022-12-21 22:01:00 +01:00
basicfs_watch.go lib/fs: Watching is unsupported on android/amd64 (fixes #8709) (#8710) 2022-12-21 22:01:00 +01:00
basicfs_windows_test.go build: Tests should run with Go 1.20 on Windows (#8924) 2023-06-05 10:19:47 +02:00
basicfs_windows.go lib/fs: Try to remove read only Windows files (fixes #3744) (#8650) 2022-11-07 21:33:17 +01:00
basicfs_xattr_bsdish.go all: Fix typos found by codespell (#8833) 2023-03-21 08:07:28 +01:00
basicfs_xattr_linuxish.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs_xattr_unix.go lib/fs: Reduce memory usage in xattrs handling (#9251) 2023-12-04 12:48:17 +01:00
basicfs_xattr_unsupported.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
basicfs.go lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969) 2023-07-07 11:00:40 +00:00
casefs_test.go all: Fix various user-facing and non-user-facing typos (#8509) 2022-08-23 15:44:11 +02:00
casefs.go build: Update all dependencies (fixes #8679) (#8680) 2022-11-17 21:26:10 +01:00
copyrangemethod.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
copyrangemethod.pb.go all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
debug.go all, lib/logger: Refactor SetDebug calls (#6054) 2019-10-04 13:03:34 +02:00
errorfs.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
fakefs_test.go all: Add build constants for runtime.GOOS comparisons (#8442) 2022-07-28 19:36:39 +02:00
fakefs.go all: Grand test refactor (fixes #8779, fixes #8799) 2023-05-09 10:01:57 +00:00
filesystem_copy_range_allwithfallback.go lib/fs: Add support for Windows duplicate extents (#6764) 2020-06-18 22:32:26 +01:00
filesystem_copy_range_standard.go all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
filesystem_copy_range_test.go all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
filesystem_copy_range.go all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
filesystem_test.go all: Correct various typos (#8870) 2023-05-09 08:54:02 +02:00
filesystem.go lib/versioner: Minor fixes in comments and error message (#9031) 2023-08-09 07:10:06 +00:00
folding_test.go lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692) 2021-05-17 20:43:07 +02:00
folding.go lib/fs: Fix UnicodeLowercaseNormalized on lowercase NFD (#7692) 2021-05-17 20:43:07 +02:00
logfs.go all: Remove unused method receivers (#8462) 2022-07-28 17:32:45 +02:00
metrics.go lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113) 2023-09-20 09:04:47 +02:00
mtimefs_test.go all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
mtimefs.go lib/fs: Better equality comparison in mtimefs 2023-12-03 16:01:46 +01:00
platform_common.go all: Support syncing extended attributes (fixes #2698) (#8513) 2022-09-14 09:50:55 +02:00
tempname_test.go lib/fs: Optimize TempName + some cosmetic changes (#7911) 2021-08-29 10:47:53 +02:00
tempname.go all: Add build constants for runtime.GOOS comparisons (#8442) 2022-07-28 19:36:39 +02:00
types.go all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
types.pb.go all: Move remaining protos to use the vanity plugin (#7009) 2020-10-02 08:07:05 +02:00
util_test.go lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969) 2023-07-07 11:00:40 +00:00
util.go lib/fs: Clarify errors for Windows filenames (fixes #8968) (#8969) 2023-07-07 11:00:40 +00:00
walkfs_test.go all: Add build constants for runtime.GOOS comparisons (#8442) 2022-07-28 19:36:39 +02:00
walkfs.go all: Remove unused method receivers (#8462) 2022-07-28 17:32:45 +02:00