Martchus
/
syncthing
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4,094 Commits 71 Branches 0 Tags 113 MiB
Commit Graph

56 Commits

Author SHA1 Message Date
Audrius Butkevicius 0582836820 lib/model, lib/scanner: Efficient inserts/deletes in the middle of the file 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3527
 2016-12-14 23:30:29 +00:00
Jakob Borg 7b07ed6580 lib/model, lib/protocol, lib/scanner: Include symlink target in index, pull symlinks synchronously 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3792
 2016-12-09 18:02:18 +00:00
Jakob Borg a9b03de99a gui, lib/db: Correct space accounting of symlinks, for "out of sync" status 2016-12-09 10:38:36 +01:00
Jakob Borg 3cde608eda lib/db: Fix ineffassign lint issue 2016-11-24 12:08:44 +01:00
Audrius Butkevicius 14937e7dd2 build: Fix proto builder on Windows 2016-11-03 22:06:51 +00:00
Jakob Borg 4e8c8d7e2c cmd/syncthing, lib/db, lib/model: Track more detailed file/dirs/links/deleted counts 2016-10-17 23:57:43 +02:00
Jakob Borg 5b37d0356c lib/model, gui: Correct completion percentages when there are lots of deletes (fixes #3496) 
We used to consider deleted files & directories 128 bytes large. After
the delta indexes change a bug slipped in where deleted files would be
weighted according to their old non-deleted size. Both ways are
incorrect (but the latest change made it worse), as if there are more
files deleted than remaining data in the repo the needSize can be
greater than the globalSize, resulting in a negative completion
percentage.

This change makes it so that deleted items are zero bytes large, which
makes more sense. Instead we expose the number of files that we need to
delete as a separate field in the Completion() result, and hack the
percentage down to 95% complete if it was 100% complete but we need to
delete files. This latter part is sort of ugly, but necessary to give
the user some sort of feedback.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3556
 2016-09-02 06:45:46 +00:00
Audrius Butkevicius af3b6f9c83 lib/model, lib/config: Support "live" device removal, folder unsharing and folder configuration changes 
Furthermore:
1. Cleans configs received, migrates them as we receive them.
2. Clears indexes of devices we no longer share the folder with

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3478
 2016-08-07 16:21:59 +00:00
Jakob Borg ea87bcefd6 lib/protocol, lib/model: Implement high precision time stamps (fixes #3305) 
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.

The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
 2016-08-06 13:05:59 +00:00
Jakob Borg 0655991a19 lib/db, lib/fs, lib/model: Introduce fs.MtimeFS, remove VirtualMtimeRepo 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3479
 2016-08-05 17:45:45 +00:00
Jakob Borg 1eb6db6ca8 cmd/syncthing, lib/...: Correctly handle ignores & invalid file names (fixes #3012, fixes #3457, fixes #3458) 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3464
 2016-08-05 07:13:52 +00:00
Jakob Borg 297240facf all: Rename LocalVersion to Sequence (fixes #3461) 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
 2016-07-29 19:54:24 +00:00
Jakob Borg 72026db599 lib/db, lib/model: Create temp sorting database in config dir (fixes #3449) 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3454
 2016-07-27 21:38:43 +00:00
Jakob Borg 694da60659 lib/db: Reinstate database update locking 
The previous commit loosened the locking around database updates.
Apparently that was not fine - what happens is that parallell updates
to the same file for different devices stomp on each others updates to
the global index, leaving it missing one of the two devices.
 2016-07-23 20:32:15 +02:00
Jakob Borg 47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438) 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
 2016-07-23 12:46:31 +00:00
Jakob Borg fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080) 
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
 2016-07-04 10:40:29 +00:00
Jakob Borg d57e6808cc lib/db: Fix alignment crash on 32 bit platforms 
Fixes #3347
Fixes #3348
Fixes #3349

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3350
 2016-06-26 13:40:51 +00:00
Jakob Borg b779e22205 lib/model: Don't set ignore bit when it's already set 
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
 2016-06-13 17:44:03 +00:00
Audrius Butkevicius 6f63909c65 lib/db,cmd/stindex: Expose VersionList and use it in stindex 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3231
 2016-05-31 19:29:26 +00:00
Audrius Butkevicius c2dc4a8e06 lib/db: Have prefix should be normalized 
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3206
 2016-05-28 04:18:31 +00:00
Jakob Borg 1a703efa78 lib/model: Fix accounting error in rescan with multiple subs (fixes #3028) 
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.

Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
 2016-05-09 12:56:21 +00:00
Jakob Borg ccfcdf7f48 cmd/syncthing: Don't compact database at startup 
This happens automatically in the background anyway, and it can take a
long time on low powered devices at an inconvenient time. We just want
to get up and running as quickly as possible.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3000
 2016-04-22 07:34:11 +00:00
Audrius Butkevicius 6130578d18 lib/db: Empty slice is not nil (fixes #2872) 2016-04-09 07:46:19 +00:00
Jakob Borg f5f0e46016 lib: Use bytes.Equal instead of bytes.Compare where possible 2016-03-31 15:12:46 +00:00
Jakob Borg aba2cc4db2 lib/model: Properly handle deleting multiple files when doing scans with subs (fixes #2851) 2016-03-18 12:16:33 +00:00
Jakob Borg c439c543d0 tests: messagediff argument order should be expected, actual 
So that the diff describes the changes that happened in actual as
compared to expected. The opposite is confusing.
 2016-03-17 08:03:29 +01:00
Lars K.W. Gohlke 9b1bebc9b2 Correct path to genxdr after the change to Go1.5+ vendoring 2016-03-09 12:43:16 +00:00
Audrius Butkevicius a8ffde6f21 Add deps 2016-03-06 20:32:10 +00:00
Laurent Etiemble c8b6e6fd9b Increase maximum allowed file size to 10 Mblocks 
Upgrade FileInfo up to 10000000 blocks. 1310 GB files can be shared.
Increase limit when unmarshaling XDR.
Increase the size of message.
 2016-03-04 16:24:54 +01:00
Jakob Borg e1ac740ac4 Use v2 of XDR package (actual changes) 2016-02-02 15:33:46 +01:00
Jakob Borg 4feeaf1641 Use v2 of XDR package (auto generated) 2016-02-02 12:44:33 +01:00
Jakob Borg 79680b1d5e Benchmark for single database update 2016-01-28 09:12:01 +01:00
Jakob Borg ac190b2e39 Change DB label format (index folders, devices) 2016-01-03 19:32:40 +01:00
Jakob Borg fa4b4dece1 Compact database on startup (ref #2400) 2015-11-24 13:17:30 +01:00
Jakob Borg 2f12d41d9d Don't dirty blockmap key between lookups (fixes #2455) 2015-11-13 15:44:30 +01:00
Mike Boone 342036408e Fix typos. 2015-11-11 21:20:34 -05:00
Jakob Borg 36f6a9347c Benchmark must use *db.Instance 2015-11-05 17:46:53 +00:00
Jakob Borg 2a4fc28318 We should pass around db.Instance instead of leveldb.DB 
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
 2015-10-31 12:35:30 +01:00
Jakob Borg 313485e406 Remove file that snuck in by mistake 2015-10-31 11:38:59 +01:00
Jakob Borg faf4267c73 Refactor: the various db key functions should be instance methods 2015-10-31 11:27:04 +01:00
Jakob Borg c4ad97136f Move leveldb instance and transactions into separate files 2015-10-29 08:07:51 +01:00
Jakob Borg fd7b8ec77e Neater transaction handling 2015-10-28 22:04:00 +01:00
Jakob Borg e28c991331 Create an instance type to tie database methods to 2015-10-28 21:03:05 +01:00
Jakob Borg 83131103cf Don't load block list in ...Truncated methods 
Speeds up and reduces allocations on those operations, at the price of
having a manually tweaked XDR decoder for FileInfoTruncated.

benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             1868198122     1880206886     +0.64%
BenchmarkUpdateOneChanged-8       231852         172695         -25.51%
BenchmarkUpdateOneUnchanged-8     230624         179341         -22.24%
BenchmarkNeedHalf-8               104601744      109461427      +4.65%
BenchmarkHave-8                   29102480       34105026       +17.19%
BenchmarkGlobal-8                 150547687      172778045      +14.77%
BenchmarkNeedHalfTruncated-8      102471355      76564986       -25.28%
BenchmarkHaveTruncated-8          28758368       14277481       -50.35%
BenchmarkGlobalTruncated-8        151192913      106070136      -29.84%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             555577         557554         +0.36%
BenchmarkUpdateOneChanged-8       1135           587            -48.28%
BenchmarkUpdateOneUnchanged-8     1135           587            -48.28%
BenchmarkNeedHalf-8               374780         374775         -0.00%
BenchmarkHave-8                   151992         152085         +0.06%
BenchmarkGlobal-8                 530033         530135         +0.02%
BenchmarkNeedHalfTruncated-8      374699         22160          -94.09%
BenchmarkHaveTruncated-8          151834         4904           -96.77%
BenchmarkGlobalTruncated-8        530037         30536          -94.24%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             1765116216     1765305376     +0.01%
BenchmarkUpdateOneChanged-8       135085         93043          -31.12%
BenchmarkUpdateOneUnchanged-8     134976         92928          -31.15%
BenchmarkNeedHalf-8               44758752       44751791       -0.02%
BenchmarkHave-8                   11845052       11967172       +1.03%
BenchmarkGlobal-8                 80431136       80431065       -0.00%
BenchmarkNeedHalfTruncated-8      46526459       18243543       -60.79%
BenchmarkHaveTruncated-8          11348357       418998         -96.31%
BenchmarkGlobalTruncated-8        80977672       43116991       -46.75%
 2015-10-21 23:49:10 +02:00
Jakob Borg c1591a5efd Only run benchmarks with -tags benchmark 
Avoids creating temp database and stuff on a normal test run
 2015-10-21 23:19:26 +02:00
Jakob Borg 918ef4dff8 Use batches in blockmap, speeds up and reduces memory usage on large Replace and Update ops 
benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             2880834572     1868198122     -35.15%
BenchmarkUpdateOneChanged-8       236596         231852         -2.01%
BenchmarkUpdateOneUnchanged-8     227326         230624         +1.45%
BenchmarkNeedHalf-8               105151538      104601744      -0.52%
BenchmarkHave-8                   28827492       29102480       +0.95%
BenchmarkGlobal-8                 150768724      150547687      -0.15%
BenchmarkNeedHalfTruncated-8      104434216      102471355      -1.88%
BenchmarkHaveTruncated-8          27860093       28758368       +3.22%
BenchmarkGlobalTruncated-8        149972888      151192913      +0.81%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             555451         555577         +0.02%
BenchmarkUpdateOneChanged-8       1135           1135           +0.00%
BenchmarkUpdateOneUnchanged-8     1135           1135           +0.00%
BenchmarkNeedHalf-8               374779         374780         +0.00%
BenchmarkHave-8                   151996         151992         -0.00%
BenchmarkGlobal-8                 530066         530033         -0.01%
BenchmarkNeedHalfTruncated-8      374702         374699         -0.00%
BenchmarkHaveTruncated-8          151834         151834         +0.00%
BenchmarkGlobalTruncated-8        530049         530037         -0.00%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             5018351912     1765116216     -64.83%
BenchmarkUpdateOneChanged-8       135085         135085         +0.00%
BenchmarkUpdateOneUnchanged-8     134976         134976         +0.00%
BenchmarkNeedHalf-8               44769400       44758752       -0.02%
BenchmarkHave-8                   11930612       11845052       -0.72%
BenchmarkGlobal-8                 81523668       80431136       -1.34%
BenchmarkNeedHalfTruncated-8      46692342       46526459       -0.36%
BenchmarkHaveTruncated-8          11348357       11348357       +0.00%
BenchmarkGlobalTruncated-8        81843956       80977672       -1.06%
 2015-10-21 23:05:23 +02:00
Jakob Borg 0d9a04c713 Reuse blockkey, speeds up large Update and Replace calls 
benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             2866418930     2880834572     +0.50%
BenchmarkUpdateOneChanged-8       226635         236596         +4.40%
BenchmarkUpdateOneUnchanged-8     229090         227326         -0.77%
BenchmarkNeedHalf-8               104483393      105151538      +0.64%
BenchmarkHave-8                   29288220       28827492       -1.57%
BenchmarkGlobal-8                 159269126      150768724      -5.34%
BenchmarkNeedHalfTruncated-8      108235000      104434216      -3.51%
BenchmarkHaveTruncated-8          28945489       27860093       -3.75%
BenchmarkGlobalTruncated-8        149355833      149972888      +0.41%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             1054944        555451         -47.35%
BenchmarkUpdateOneChanged-8       1135           1135           +0.00%
BenchmarkUpdateOneUnchanged-8     1135           1135           +0.00%
BenchmarkNeedHalf-8               374777         374779         +0.00%
BenchmarkHave-8                   151995         151996         +0.00%
BenchmarkGlobal-8                 530063         530066         +0.00%
BenchmarkNeedHalfTruncated-8      374699         374702         +0.00%
BenchmarkHaveTruncated-8          151834         151834         +0.00%
BenchmarkGlobalTruncated-8        530021         530049         +0.01%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             5074297112     5018351912     -1.10%
BenchmarkUpdateOneChanged-8       135097         135085         -0.01%
BenchmarkUpdateOneUnchanged-8     134976         134976         +0.00%
BenchmarkNeedHalf-8               44759436       44769400       +0.02%
BenchmarkHave-8                   11911138       11930612       +0.16%
BenchmarkGlobal-8                 81609867       81523668       -0.11%
BenchmarkNeedHalfTruncated-8      46588024       46692342       +0.22%
BenchmarkHaveTruncated-8          11348354       11348357       +0.00%
BenchmarkGlobalTruncated-8        79485168       81843956       +2.97%
 2015-10-21 23:05:23 +02:00
Jakob Borg 0c0c69f0cf The GC runs are legacy and slows things down quite a bit 
benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             2942370526     2866418930     -2.58%
BenchmarkUpdateOneChanged-8       7402489        226635         -96.94%
BenchmarkUpdateOneUnchanged-8     7298777        229090         -96.86%
BenchmarkNeedHalf-8               113608416      104483393      -8.03%
BenchmarkHave-8                   29834263       29288220       -1.83%
BenchmarkGlobal-8                 162773699      159269126      -2.15%
BenchmarkNeedHalfTruncated-8      111943400      108235000      -3.31%
BenchmarkHaveTruncated-8          29490369       28945489       -1.85%
BenchmarkGlobalTruncated-8        165841081      149355833      -9.94%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             1054942        1054944        +0.00%
BenchmarkUpdateOneChanged-8       1149           1135           -1.22%
BenchmarkUpdateOneUnchanged-8     1135           1135           +0.00%
BenchmarkNeedHalf-8               374774         374777         +0.00%
BenchmarkHave-8                   151995         151995         +0.00%
BenchmarkGlobal-8                 530042         530063         +0.00%
BenchmarkNeedHalfTruncated-8      374697         374699         +0.00%
BenchmarkHaveTruncated-8          151834         151834         +0.00%
BenchmarkGlobalTruncated-8        530050         530021         -0.01%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             5074294728     5074297112     +0.00%
BenchmarkUpdateOneChanged-8       141048         135097         -4.22%
BenchmarkUpdateOneUnchanged-8     134976         134976         +0.00%
BenchmarkNeedHalf-8               44734813       44759436       +0.06%
BenchmarkHave-8                   11911634       11911138       -0.00%
BenchmarkGlobal-8                 80436854       81609867       +1.46%
BenchmarkNeedHalfTruncated-8      46514673       46588024       +0.16%
BenchmarkHaveTruncated-8          11348357       11348354       -0.00%
BenchmarkGlobalTruncated-8        81730740       79485168       -2.75%
 2015-10-21 23:05:22 +02:00
Jakob Borg 943e80e26c Make benchmarks more realistic 2015-10-21 23:04:29 +02:00
Jakob Borg 1eca4170f7 Add test for LocalSize/GlobalSize results 2015-10-21 21:58:48 +02:00