Commit Graph

18 Commits

Author SHA1 Message Date
greatroar
82fbcb96f8
lib/db: Remove unused blockFinder global (#6534) 2020-04-14 17:09:59 +02:00
Simon Frei
0bec01b827 lib/db: Remove *instance by making everything *Lowlevel (#6204) 2019-12-02 08:18:04 +01:00
Jakob Borg
c71116ee94
Implement database abstraction, error checking (ref #5907) (#6107)
This PR does two things, because one lead to the other:

- Move the leveldb specific stuff into a small "backend" package that
defines a backend interface and the leveldb implementation. This allows,
potentially, in the future, switching the db implementation so another
KV store should we wish to do so.

- Add proper error handling all along the way. The db and backend
packages are now errcheck clean. However, I drew the line at modifying
the FileSet API in order to keep this manageable and not continue
refactoring all of the rest of Syncthing. As such, the FileSet methods
still panic on database errors, except for the "database is closed"
error which is instead handled by silently returning as quickly as
possible, with the assumption that we're anyway "on the way out".
2019-11-29 09:11:52 +01:00
Simon Frei
42bd42df5a lib/db: Do all update operations on a single item at once (#5441)
To do so the BlockMap struct has been removed. It behaves like any other prefixed
part of the database, but was not integrated in the recent keyer refactor. Now
the database is only flushed when files are in a consistent state.
2019-01-23 10:22:33 +01:00
Simon Frei
158559023e lib/db: Fix sequence updating for remote invalid items (#5420)
* lib/db: Fix sequence updating for remote invalid items

* fix for the unit test introduced in the previous commit

* lib/db: Polish blockmap
2019-01-04 20:19:10 +01:00
Jakob Borg
b50d57b7fd
lib/db: Refactor: use a Lowlevel type underneath Instance (ref #5198) (#5212)
This adds a thin type that holds the state associated with the
leveldb.DB, leaving the huge Instance type more or less stateless. Also
moves some keying stuff into the DB package so that other packages need
not know the keying specifics.

(This does not, yet, fix the cmd/stindex program, in order to keep the
diff size down. Hence the keying constants are still exported.)
2018-10-10 11:34:24 +02:00
Audrius Butkevicius
ef0dcea6a4 lib/model: Verify request content against weak (and possibly strong) hash (#4767) 2018-05-05 10:24:44 +02:00
Jakob Borg
e9f05d138f lib/db, lib/model: Always use reasonable sized batches (fixes #2250, fixes #4112)
Harmonize how we use batches in the model, using ProtoSize() to judge
the actual weight of the entire batch instead of estimating. Use smaller
batches in the block map - I think we might have though that batch.Len()
in the leveldb was the batch size in bytes, but it's actually number of
operations.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4114
2017-04-22 14:23:33 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Jakob Borg
47f22ff3e5 build: Enable gometalinter "unconvert" check 2016-12-21 14:53:45 +01:00
Jakob Borg
ac190b2e39 Change DB label format (index folders, devices) 2016-01-03 19:32:40 +01:00
Jakob Borg
2f12d41d9d Don't dirty blockmap key between lookups (fixes #2455) 2015-11-13 15:44:30 +01:00
Jakob Borg
2a4fc28318 We should pass around db.Instance instead of leveldb.DB
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
2015-10-31 12:35:30 +01:00
Jakob Borg
918ef4dff8 Use batches in blockmap, speeds up and reduces memory usage on large Replace and Update ops
benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             2880834572     1868198122     -35.15%
BenchmarkUpdateOneChanged-8       236596         231852         -2.01%
BenchmarkUpdateOneUnchanged-8     227326         230624         +1.45%
BenchmarkNeedHalf-8               105151538      104601744      -0.52%
BenchmarkHave-8                   28827492       29102480       +0.95%
BenchmarkGlobal-8                 150768724      150547687      -0.15%
BenchmarkNeedHalfTruncated-8      104434216      102471355      -1.88%
BenchmarkHaveTruncated-8          27860093       28758368       +3.22%
BenchmarkGlobalTruncated-8        149972888      151192913      +0.81%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             555451         555577         +0.02%
BenchmarkUpdateOneChanged-8       1135           1135           +0.00%
BenchmarkUpdateOneUnchanged-8     1135           1135           +0.00%
BenchmarkNeedHalf-8               374779         374780         +0.00%
BenchmarkHave-8                   151996         151992         -0.00%
BenchmarkGlobal-8                 530066         530033         -0.01%
BenchmarkNeedHalfTruncated-8      374702         374699         -0.00%
BenchmarkHaveTruncated-8          151834         151834         +0.00%
BenchmarkGlobalTruncated-8        530049         530037         -0.00%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             5018351912     1765116216     -64.83%
BenchmarkUpdateOneChanged-8       135085         135085         +0.00%
BenchmarkUpdateOneUnchanged-8     134976         134976         +0.00%
BenchmarkNeedHalf-8               44769400       44758752       -0.02%
BenchmarkHave-8                   11930612       11845052       -0.72%
BenchmarkGlobal-8                 81523668       80431136       -1.34%
BenchmarkNeedHalfTruncated-8      46692342       46526459       -0.36%
BenchmarkHaveTruncated-8          11348357       11348357       +0.00%
BenchmarkGlobalTruncated-8        81843956       80977672       -1.06%
2015-10-21 23:05:23 +02:00
Jakob Borg
0d9a04c713 Reuse blockkey, speeds up large Update and Replace calls
benchmark                         old ns/op      new ns/op      delta
BenchmarkReplaceAll-8             2866418930     2880834572     +0.50%
BenchmarkUpdateOneChanged-8       226635         236596         +4.40%
BenchmarkUpdateOneUnchanged-8     229090         227326         -0.77%
BenchmarkNeedHalf-8               104483393      105151538      +0.64%
BenchmarkHave-8                   29288220       28827492       -1.57%
BenchmarkGlobal-8                 159269126      150768724      -5.34%
BenchmarkNeedHalfTruncated-8      108235000      104434216      -3.51%
BenchmarkHaveTruncated-8          28945489       27860093       -3.75%
BenchmarkGlobalTruncated-8        149355833      149972888      +0.41%

benchmark                         old allocs     new allocs     delta
BenchmarkReplaceAll-8             1054944        555451         -47.35%
BenchmarkUpdateOneChanged-8       1135           1135           +0.00%
BenchmarkUpdateOneUnchanged-8     1135           1135           +0.00%
BenchmarkNeedHalf-8               374777         374779         +0.00%
BenchmarkHave-8                   151995         151996         +0.00%
BenchmarkGlobal-8                 530063         530066         +0.00%
BenchmarkNeedHalfTruncated-8      374699         374702         +0.00%
BenchmarkHaveTruncated-8          151834         151834         +0.00%
BenchmarkGlobalTruncated-8        530021         530049         +0.01%

benchmark                         old bytes      new bytes      delta
BenchmarkReplaceAll-8             5074297112     5018351912     -1.10%
BenchmarkUpdateOneChanged-8       135097         135085         -0.01%
BenchmarkUpdateOneUnchanged-8     134976         134976         +0.00%
BenchmarkNeedHalf-8               44759436       44769400       +0.02%
BenchmarkHave-8                   11911138       11930612       +0.16%
BenchmarkGlobal-8                 81609867       81523668       -0.11%
BenchmarkNeedHalfTruncated-8      46588024       46692342       +0.22%
BenchmarkHaveTruncated-8          11348354       11348357       +0.00%
BenchmarkGlobalTruncated-8        79485168       81843956       +2.97%
2015-10-21 23:05:23 +02:00
Jakob Borg
4581c57478 Fix import paths 2015-09-22 19:38:46 +02:00
Jakob Borg
4a9997e449 lib/db need not depend on lib/config 2015-09-04 12:01:00 +02:00
Jakob Borg
7705a6c1f1 mv internal lib 2015-08-09 09:35:26 +02:00