Commit Graph

110 Commits

Author SHA1 Message Date
Simon Frei
ce29d3a574 all: Various debug logging improvements
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4529
2017-11-22 08:05:27 +00:00
Audrius Butkevicius
44a542391e lib/connections: Trust the model to tell us if we are connected
This should address issue as described in https://forum.syncthing.net/t/stun-nig-party-with-paused-devices/10942/13
Essentially the model and the connection service goes out of sync in terms of thinking if we are connected or not.
Resort to model as being the ultimate source of truth.

I can't immediately pin down how this happens, yet some ideas.

ConfigSaved happens in separate routine, so it's possbile that we have some sort of device removed yet connection comes in parallel kind of thing.
However, in this case the connection exists in the model, and does not exist in the connection service and the only way for the connection to be removed
in the connection service is device removal from the config.

Given the subject, this might also be related to the device being paused.

Also, adds more info to the logs

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4533
2017-11-21 07:25:38 +00:00
Jakob Borg
0901350087 lib/model: Trigger a pull when ignore patterns change
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4532
2017-11-20 16:29:36 +00:00
Simon Frei
6cf01c1d30 lib/model: Don't update ignore hash when pull fails
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4522
2017-11-17 12:42:41 +00:00
Jakob Borg
5f4ed66aa1 lib/model: Properly schedule pull on reconnect (fixes #4504)
We need to reset prevSeq so that we force a full check when someone
reconnects - the sequence number may not have changed due to the
reconnect. (This is a regression; we did this before f6ea2a7.)

Also add an optimization: we schedule a pull after scanning, but there
is no need to do so if no changes were detected. This matters now
because the scheduled pull actually traverses the database which is
expensive.

This, however, makes the pull not happen on initial scan if there were
no changes during the initial scan. Compensate by always scheduling a
pull after initial scan in the rwfolder itself.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4508
LGTM: imsodin, AudriusButkevicius
2017-11-17 12:11:45 +00:00
Simon Frei
fa12a18190 lib/model: Handle type changes when pulling (ref #4505 #4506 #4507)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4509
LGTM: AudriusButkevicius, calmh
2017-11-13 15:16:27 +00:00
Jakob Borg
2b65e1062e lib/model: Fix rescan detection (fixes #4505, fixes #4506)
Diff is large due to comment reformatting and indentation but all it
does is wrap the file mtime/size/permissions check in an "if
stat.IsRegular()".

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4507
2017-11-13 06:57:07 +00:00
Simon Frei
c080f677cb all: Add invalid/ignored files to global list, announce to peers (fixes #623)
This lets us determine accurate completion status for remote peers when they
have ignored files.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4460
2017-11-11 19:18:17 +00:00
Audrius Butkevicius
386cb274bd lib/model: Incremental block stats usage reporting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4498
LGTM: calmh
2017-11-09 21:16:29 +00:00
Jakob Borg
a9d422e008 Revert "lib/model: Use the new file name when requesting rescan"
This has consequences that I haven't thought through for case mismatches
(i.e. we have foo.txt in the database and queue a rescan for FOO.txt).
2017-11-09 13:55:45 +01:00
Jakob Borg
75f64733f9 lib/model: Use the new file name when requesting rescan
If the file is not already in the database, the "curfile" will be empty
and the rescan request cover the entire folder.
2017-11-09 13:48:39 +01:00
Simon Frei
f6ea2a7f8e lib/model: Trigger pulls instead of pulling periodically
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4340
2017-11-07 06:59:35 +00:00
Simon Frei
4b69d0e093 lib/model: Add initial noop watch cancel func (fixes #4464)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4465
2017-10-26 11:49:06 +00:00
Simon Frei
dc42db444b lib/model, lib/config: Refactor folder health/error handling (fixes #4445, fixes #4451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4455
LGTM: AudriusButkevicius, calmh
2017-10-24 07:58:55 +00:00
Audrius Butkevicius
622b614f31 all: Ignore Sync errors on directories (fixes #4432)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4449
2017-10-21 22:00:46 +00:00
Michael Ploujnikov
f98c21b68e all: Add filesystem notification support
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3986
2017-10-20 14:52:55 +00:00
Audrius Butkevicius
74c8d34805 lib/model: Centralize error reporting, modified files are errors (fixes #4392) 2017-10-12 08:23:33 +02:00
Audrius Butkevicius
2760d032ca cmd/syncthing: Add more stats to usage reports (ref #3628)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4347
2017-10-12 06:16:46 +00:00
Simon Frei
1088eb12ea lib/model: Fix logging inconsistencies (fixes #4375)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4376
2017-09-18 11:56:19 +00:00
Simon Frei
f7d2c58783 lib/model: Deduplicate folder loops
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4352
LGTM: AudriusButkevicius, calmh
2017-09-07 06:17:47 +00:00
Simon Frei
55e0ac3e24 lib/model: Make puller retrying and logging less aggressive
Currently all errors during pulling and the first of these errors again on
finishing are logged to info. Besides that the errors logged when finishing
are stored in f.errors. This PR moves all logging during pulling to the debug
channel (they might still be relevant in some obscure debugging case) and
uses the stored errors to log the main error per fail when all pulling
iterations are done and failed.

Additional instead of trying 11 times it now only tries 3 times.

This is the first part of what is discussed here:
https://forum.syncthing.net/t/reduce-verboseness-of-puller/10261

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4338
2017-09-02 06:05:55 +00:00
Audrius Butkevicius
ab132ff6fe lib: Folder marker is now a folder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4341
LGTM: calmh
2017-09-02 05:52:38 +00:00
Jakob Borg
a8145e187f lib/model: Fixup because I suck and shouldn't commit directly to master 2017-08-31 10:49:17 +02:00
Jakob Borg
e33fa10115 lib/model: Use same batch size constants in db updater as for protocol 2017-08-31 10:47:39 +02:00
Simon Frei
ddf6d64faa lib/model: Create folders via newFolder
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4329
2017-08-25 19:47:01 +00:00
Audrius Butkevicius
b8c249cddc lib/model: Move stale scan check info finisher (ref #4305, fix #3742)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4317
2017-08-22 06:42:09 +00:00
Audrius Butkevicius
3d8b4a42b7 all: Convert folders to use filesystem abstraction
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4228
2017-08-19 14:36:56 +00:00
Jakob Borg
f1f21bf220 lib/model, lib/versioner: Prevent symlink attack via versioning (fixes #4286)
Prior to this, the following is possible:

- Create a symlink "foo -> /somewhere", it gets synced
- Delete "foo", it gets versioned
- Create "foo/bar", it gets synced
- Delete "foo/bar", it gets versioned in "/somewhere/bar"

With this change, versioners should never version symlinks.
2017-08-07 07:57:10 +02:00
nrm21
1235cead35 lib/model: Add name of latest modifying device to conflict file (fixes #3524)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4161
LGTM: AudriusButkevicius, calmh
2017-05-25 10:26:41 +00:00
Jakob Borg
d6fbfc3545 lib/fs, lib/model, lib/scanner: Make scans cancellable (fixes #3965)
The folder already knew how to stop properly, but the fs.Walk() didn't
and can potentially take a very long time. This adds context support to
Walk and the underlying scanning stuff, and passes in an appropriate
context from above. The stop channel in model.folder is replaced with a
context for this purpose.

To test I added an infiniteFS that represents a large amount of data
(not actually infinite, but close) and verify that walking it is
properly stopped. For that to be implemented smoothly I moved out the
Walk function to it's own type, as typically the implementer of a new
filesystem type might not need or want to reimplement Walk.

It's somewhat tricky to test that this actually works properly on the
actual sendReceiveFolder and so on, as those are started from inside the
model and the filesystem isn't easily pluggable etc. Instead I've tested
that part manually by adding a huge folder and verifying that pause,
resume and reconfig do the right things by looking at debug output.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4117
2017-04-26 00:15:23 +00:00
Simon Frei
6b188ebcf3 lib/model: Mark initial scan as finished even if failed and refactor (fixes #4103)
The mechanism to disallow manual scans before the initial scan completed
(#3996) , had the side effect, that if the initial scan failed, no further
scans are allowed. So this marks the initial scan as finished regardless of
whether it succeeded or not.

There was also redundant code in rofolder and a pointless check for folder
health in scanSubsIfHealthy (happens in internalScanFolderSubdirs as well).
This also moves logging from folder.go to ro/rw-folder.go to include the
information about whether it is send-only or send-receive

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4104
2017-04-20 00:20:34 +00:00
Jakob Borg
da34f27546 gui, lib/config, lib/model: Allow absolute values for minimum disk free space (fixes #3307)
This deprecates the current minDiskFreePct setting and introduces
minDiskFree. The latter is, in it's serialized form, a string with a
unit. We accept percentages ("2.35%") and absolute values ("250 k", "12.5
Gi"). Common suffixes are understood. The config editor lets the user
enter the string, and validates it.

We still default to "1 %", but the user can change that to an absolute
value at will.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4087
LGTM: AudriusButkevicius, imsodin
2017-04-12 09:01:19 +00:00
Simon Frei
25b314f5f1 lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3996
LGTM: calmh, AudriusButkevicius
2017-04-01 09:58:06 +00:00
Jakob Borg
4253f22680 lib/scanner: Use fs.Filesystem for all operations
One more step on the path of the great refactoring. Touches rwfolder a
little bit since it uses the Lstat from fs as well, but mostly this is
just on the scanner as rwfolder is scheduled for a later refactor.

There are a couple of usages of fs.DefaultFilesystem that will in the
end become a filesystem injected from the top, but that comes later.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4070
LGTM: AudriusButkevicius, imsodin
2017-04-01 09:04:11 +00:00
Jakob Borg
5c27796471 lib/model, lib/scanner: Properly ignore symlinks on Windows (fixes #4035)
Adds a unit test to ensure we don't scan symlinks on Windows. For the
rwfolder, trusts that the logic in the invalid check is correct and that
the check is actually called from the need loop.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4042
2017-03-18 00:25:47 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Audrius Butkevicius
72de47df00 lib/model: Increase in-flight buffer from 2mb to 8mb, download 2 files at a time
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3972
2017-02-08 20:00:55 +00:00
Jakob Borg
c4ba580cbb all: Remove symlink support on Windows, SymlinksEnabled config
After this change,

- Symlinks on Windows are always unsupported. Sorry.

- Symlinks are always enabled on other platforms. They are just a small
  file like anything else. There is no need to special case them. If you
  don't want to sync some symlinks, ignore them.

- The protocol doesn't differentiate between different "types" of
  symlinks. If that distinction ever does become relevant the individual
  devices can figure it out by looking at the destination when they
  create the link.

It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
2017-02-07 08:34:24 +00:00
Audrius Butkevicius
67acef1794 lib/weakhash, lib/model, cmd/syncthing: Decide if to use weakhash on startup (fixes #3938)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3949
2017-02-06 10:27:11 +00:00
Audrius Butkevicius
dd78177ae0 scanner: Allow disabling weak hash in scanning (fixes #3891)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3905
2017-01-23 13:50:32 +00:00
Simon Frei
dbb3a34887 lib/ignore: Centralize handling of temporary filenames (fixes #3899)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3901
LGTM: calmh, AudriusButkevicius
2017-01-17 07:33:48 +00:00
Audrius Butkevicius
1a1e35d998 lib/osutil: Replace IsDir with TraversesSymlink (fixes #3839)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3883
LGTM: calmh
2017-01-10 07:09:31 +00:00
Audrius Butkevicius
29d010ec0e lib/model, lib/weakhash: Hash using adler32, add heuristic in puller
Adler32 is much faster, and the heuristic avoid the obvious cases where it
will not help.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3872
2017-01-04 21:04:13 +00:00
Jakob Borg
f5a310ad64 Revert "lib/model: Handle filename conflicts on Windows."
This reverts commit 01e50eb3fa.
2016-12-23 11:10:58 +01:00
Unrud
01e50eb3fa lib/model: Handle filename conflicts on Windows.
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3810
LGTM: calmh
2016-12-22 23:04:53 +00:00
Jakob Borg
744c2e82b5 build: Enable gometalinter "staticcheck" check 2016-12-21 14:53:45 +01:00
Jakob Borg
5c067661f4 lib/model: Consistently show folder description in startup messages
Since we anyway need the folderConfig for this I'm skipping the copying
of all it's attributes that rwfolder did and just keeping the original
around instead.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3825
2016-12-21 11:23:20 +00:00
Heiko Zuerker
398c356f22 lib/model: Clarify master terminology (fixes #2679)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3793
2016-12-16 22:23:35 +00:00
Audrius Butkevicius
0582836820 lib/model, lib/scanner: Efficient inserts/deletes in the middle of the file
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3527
2016-12-14 23:30:29 +00:00
Jakob Borg
3582783972 lib/model, lib/osutil: Verify target directory before pulling / requesting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3798
2016-12-13 10:24:10 +00:00