This skips the warning about "unsupported symlinks" for invalid or
deleted symlinks (and as a side effect also accepts them into the index,
which should be fine). It also prints the affected file paths to the
log. This should be in the hypothetical list of "errored files" we
should present instead of the cryptic "puller stopped" message in the
future...
My theory is that some error condition on the socket results in it
blocking for writes, which maybe also blocks reads... This separates the
two into separate services with their own socket, with restarts and
retries as appropriates on write timeouts and read/write errors. It
should be more robust, hopefully, but I have a hard time testing the
actual error conditions...
We introduced the dbUpdater routine to handle many small files
efficiently, but the folder stats call is almost equally expensive as it
results in two distinct write transactions to the database. This moves
it to the same routine.
(Doesn't make a *huge* difference with leveldb actually, but reduces the
50k-files benchmark time by 25% on my experimental bolt branch...)
This is surprisingly memory expensive when Poll gets called a lot, such
as when syncing lots of small files generating itemstarted/itemfinished
events. It's line number three in this heap profile on the
TestBenchmarkManyFiles test:
jb@syno:~/src/github.com/syncthing/syncthing/test (master) $ go tool pprof ../bin/syncthing heap-13194.pprof
Entering interactive mode (type "help" for commands)
(pprof) top
80.91MB of 83.05MB total (97.42%)
Dropped 1024 nodes (cum <= 0.42MB)
Showing top 10 nodes out of 85 (cum >= 1.75MB)
flat flat% sum% cum cum%
32MB 38.53% 38.53% 32.01MB 38.54% github.com/syndtr/goleveldb/leveldb/memdb.New
22.16MB 26.68% 65.21% 22.16MB 26.68% github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
13.02MB 15.68% 80.89% 13.02MB 15.68% time.NewTimer
6.94MB 8.35% 89.24% 6.94MB 8.35% github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put
3.18MB 3.82% 93.06% 3.18MB 3.82% github.com/calmh/xdr.(*Reader).ReadBytesMaxInto
With this change the allocation is removed entirely.
Jakob Borg
Sergey Mishin
Audrius Butkevicius
Bart De Vries
Lode Hoste