Commit Graph

633 Commits

Author SHA1 Message Date
Audrius Butkevicius
e9bb17307d
lib/fs: Unwrap mtimeFile, get fd the "correct" way (ref #6875) (#6877) 2020-08-07 07:47:48 +02:00
Audrius Butkevicius
96e197e502
lib/model: Don't close file early (fixes #6875) (#6876) 2020-08-03 21:54:42 +02:00
Simon Frei
a85bc1c3a6
lib/model: Check folder error before sync-waiting (fixes #6793) (#6847) 2020-07-31 19:26:09 +02:00
Simon Frei
850dd4cd25
lib/db: Don't count invalid items to global state (ref #6850) (#6863) 2020-07-30 13:49:14 +02:00
Simon Frei
e46c8ab9ee
lib/model: Detect deleted RO items scanning on non-RO (fixes #6864) (#6865) 2020-07-30 13:41:45 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Jakob Borg
aedc2d788f
lib/model, lib/versioner: Drive version cleanup from scanner (fixes #6313) (#6693)
This change adds a separate config for the cleanup interval, and runs that cleanup from the main folder loop.
2020-07-14 10:48:50 +02:00
Jakob Borg
0c61c66511
lib/api, lib/model: Improve folder completion API (fixes #6075) (#6808) 2020-07-03 08:48:37 +02:00
Simon Frei
9c45ac381c
lib/model: Consider weak hash on recheckFile (#6797) 2020-06-26 16:47:03 +02:00
Simon Frei
4f06708330
lib/api, lib/model: Fixes around event request tracking (#6774) 2020-06-25 21:48:48 +02:00
Simon Frei
0648fb0626
lib/model: Don't force rescan already changed items (#6798) 2020-06-25 20:23:59 +02:00
Simon Frei
90e248615f
lib/scanner: Test weak hash consistency (ref #5556) (#6794)
Relevant much earlier changes:
  9b1c592fb7
  bd1c29ee32

Make sure vanilla and rolling adler are consistent. And that they match
with scanner.Validate.
2020-06-25 14:47:35 +02:00
Audrius Butkevicius
689cf2a5ee
lib/ur: Normalise contract between syncthing and ursrv (#6770)
* Fix ui, hide report date

* Undo Goland madness

* UR now web scale

* Fix migration

* Fix marshaling, force tick on start

* Fix tests

* Darwin build

* Split "all" build target, add package name as a tag

* Remove pq and sql dep from syncthing, split build targets

* Empty line

* Revert "Empty line"

This reverts commit f74af2b067dadda8a343714123512bd545a643c3.

* Revert "Remove pq and sql dep from syncthing, split build targets"

This reverts commit 8fc295ad007c5bb7886c557f492dacf51be307ad.

* Revert "Split "all" build target, add package name as a tag"

This reverts commit f4dc88995106d2b06042f30bea781a0feb08e55f.

* Normalise contract types

* Fix build add more logging
2020-06-23 09:47:15 +01:00
Simon Frei
85794933d3
lib/model: Remove double error handling in performFinish (#6780) 2020-06-23 07:30:16 +02:00
Audrius Butkevicius
5fb3992275
lib/model: More connection closing (#6778)
* lib/model: More connection closing

* Revert "lib/model: More connection closing"

This reverts commit 5397c3a55c0de25fab93d4ec622530818f1b2a46.

* Add tests, fix one broken-ness

* Update lib/model/model.go

Co-authored-by: Simon Frei <freisim93@gmail.com>

* Update model.go

* Update model.go

Co-authored-by: Simon Frei <freisim93@gmail.com>
2020-06-22 21:26:26 +01:00
Simon Frei
f66e57947b
lib/model: Use right db snap when scanning recvonly folder (#6769) 2020-06-21 09:28:29 +02:00
Simon Frei
2716898cb9
lib/model: Don't ignore stat failure in performFinish (#6766) 2020-06-19 23:47:29 +02:00
Simon Frei
22f0077262
lib/model: Don't stay scanning forever on fail (#6761) 2020-06-18 14:13:46 +01:00
Jakob Borg
946170f3fc
gui, lib/ignore: Handle editing ignores with error (fixes #5425) (#6757)
This changes the error handling in loading ignores slightly:

- There is a new ParseError type that is returned as the error
  (somewhere in the chain) when the problem was not an I/O error loading
  the file, but some issue with the contents.

- If the file was read successfully but not parsed successfully we still
  return the lines read (in addition to nil patterns and a ParseError).

- In the API, if the error IsParseError then we return a successful
  HTTP response with the lines and the actual error included in the JSON
  object.

- In the GUI, as long as the HTTP call to load the ignores was
  successful we can edit the ignores. If there was an error we show this
  as a validation error on the dialog.

Also some cleanup on the Javascript side as it for some reason used
jQuery instead of Angular for this editor...
2020-06-18 11:04:00 +02:00
Simon Frei
8cf9d91ed4
lib: Print nicely rounded durations (#6756) 2020-06-18 10:55:41 +02:00
Audrius Butkevicius
4812fd3ec1
all: Add copy-on-write filesystem support (fixes #4271) (#6746) 2020-06-18 08:15:47 +02:00
Simon Frei
6976219d6d
lib/model: Check dir before deletion when pulling (#6741) 2020-06-16 15:20:08 +02:00
Simon Frei
a47546a1f1
lib/scanner, lib/model: Improve error handling when scanning (#6736) 2020-06-16 09:25:41 +02:00
Jakob Borg
5be13f62a2 lib/model: Fix minor flakiness in TestModTimeWindow 2020-06-16 06:32:37 +02:00
Simon Frei
1f8e6c55f6
lib/db: Refactor to use global list by version (fixes #6372) (#6638)
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-05-30 09:50:23 +02:00
Simon Frei
ed6bfc5417
lib/model: Don't increase pull pause on triggered pull (#6690) 2020-05-29 09:52:28 +02:00
Simon Frei
b784f5b9e3
lib/config, lib/model: Commit auto-accepted folders all at once (#6684) 2020-05-27 08:05:26 +02:00
Simon Frei
c3b5eba205
lib/model: Fix checking children when trying to delete a dir (fixes #6646) (#6647) 2020-05-25 08:46:24 +02:00
Simon Frei
cf75329067
lib/model: Handle concurrently removed device in ClusterConfig (#6666) 2020-05-20 11:13:55 +02:00
Jakob Borg
5ffa012410
lib/model: Don't crash when taking rename shortcut (fixes #6654) (#6657)
If we fail to take the rename shortcut we may crash on a later loop,
because we do trickiness with the indexes but the original buckets[key]
in "range buckets[key]" isn't re-evaluated so i exceeds the max index.
2020-05-17 08:48:35 +01:00
Jakob Borg
651ee2ce74
lib/model: Partial revert of rename fix (fixes #6653) (#6656)
We can't just drop the snap because it's in use elsewhere. This should
be equally functional.
2020-05-16 23:05:33 +02:00
Audrius Butkevicius
a1c5b44c74
lib/model: Fix rename handling (ref #6650) (#6652) 2020-05-16 14:39:27 +02:00
Simon Frei
e5cc55ce09
lib/model: Close conns when devices are removed (fixes #6564) (#6641) 2020-05-14 07:50:53 +02:00
Simon Frei
299b9d8883
lib/model: Adjust remote-rename-test to timer-based versions (fixes #6625) (#6644) 2020-05-14 00:31:05 +02:00
Simon Frei
974551375e
lib/db: Dont add symlinks to blocks map (fixes #6637) (#6639) 2020-05-13 20:38:21 +02:00
Jakob Borg
531ceb2b0f
Add indirection for large version vectors. (#6376)
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.

To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
2020-05-13 14:28:42 +02:00
Audrius Butkevicius
6201eebc98
lib/model: Add support for different puller block ordering (#6587)
* WIP

* Tests

* Header and format

* WIP

* Fix tests

* Goland you disappoint me

* Remove CC storage

* Update blockpullreorderer.go
2020-05-11 22:44:04 +01:00
Audrius Butkevicius
decb967969
all: Reorder sequences for better rename detection (#6574) 2020-05-11 20:15:11 +02:00
Simon Frei
a94951becd
lib/db, lib/model: Keep need stats in metadata (ref #5899) (#6413) 2020-05-11 15:07:06 +02:00
Simon Frei
57ea8a1bf5
lib/model: Prevent panic on test failure (ref #6618) (#6620) 2020-05-08 17:56:49 +02:00
Simon Frei
2cdeb1bf70
lib/model: Spurious tmp file (ref #6607) (#6609) 2020-05-07 08:49:59 +02:00
Jakob Borg
876609a0f0
lib/model: Fix test after version vector changes (#6607) 2020-05-06 21:19:33 +02:00
Jakob Borg
744ef0d8ac
lib/protocol: Avoid data loss on database wipe by higher version numbers (fixes #3876) (#6605)
This makes version vector values clock based instead of just incremented
from zero. The effect is that a vector that is created from scratch
(after database reset) will have a higher value for the local device
than what it could have been previously, causing a conflict. That is, if
we are A and we had

    {A: 42, B: 12}

in the old scheme, a reset and rescan would give us

    {A: 1}

which is a strict ancestor of the older file (this might be wrong). With
the new scheme we would instead have

    {A: someClockTime, b: otherClockTime}

and the new version after reset would become

    {A: someClockTime+delta}

which is in conflict with the previous entry (better).
In case the clocks are wrong (current time is less than the value in the
vector) we fall back to just simple increment like today.

This scheme is ineffective if we suffer a database reset while at the
same time setting the clock back far into the past. It's however no
worse than what we already do.

This loses the ability to emit the "added" event, as we can't look for
the magic 1 entry any more. That event was however already broken
(#5541).

Another place where we infer meaning from the vector itself is in
receive only folders, but there the only criteria is that the vector is
one item long and includes just ourselves, which remains the case with
this change.

* wip
2020-05-06 08:47:02 +02:00
Simon Frei
b5fc332782
lib/model: Merge add and start folder funcs and related refactor (#6594) 2020-05-06 08:34:54 +02:00
Simon Frei
914eb77ca4
lib/model: Don't include iolimiter wait into sync duration (#6593) 2020-05-04 08:43:35 +02:00
Simon Frei
2e3975e956
lib/model: Improve errors when deleting dirs (fixes #6575) (#6586) 2020-05-01 11:11:38 +02:00
Audrius Butkevicius
bd0c2bf237
lib/model: Do file recheck in folder loop (fixes #6583) (#6585) 2020-05-01 11:08:59 +02:00
Audrius Butkevicius
f86deedd9c
lib/model: Progress emitter network operations dont need to be blocking (#6589)
* lib/model: Progress emitter network operations dont need to be blocking

* Do sends outside of the lock
2020-05-01 08:54:15 +01:00
Audrius Butkevicius
782bd08aad
lib/model: Add option to disable fsync (#6588)
* lib/model: Add option to disable fsync

* Fix test

* Dont open stuff for no reason
2020-05-01 08:36:46 +01:00
Jakob Borg
6c73617974
lib/model: Use semaphore to limit concurrent folder writes (fixes #6541) (#6573) 2020-04-27 00:13:18 +02:00