Commit Graph

728 Commits

Author SHA1 Message Date
Jakob Borg
68c1a0b9b4 lib/ignore, lib/model: Use an interface to detect file changes, improving tests
This solves the erratic test failures on model.TestIgnores by ensuring
that the ignore patterns are reloaded even in the face of unchanged
timestamps.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4208
2017-06-11 10:27:12 +00:00
Audrius Butkevicius
9667a0a618 lib/connections: Fix race (fixes #4177) 2017-06-07 10:17:01 +02:00
Simon Frei
719c313b23 cmd/syncthing, lib/logger: Set debug flags on SetDebug for tests
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4187
2017-06-02 07:04:06 +00:00
nrm21
b49bbe82dd gui, lib/config, lib/model: Add ability to ignore folders offered by other nodes (fixes #3993)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4179
LGTM: AudriusButkevicius, calmh
2017-05-31 18:04:00 +00:00
Jakob Borg
3959eb26fb lib/model: Events should have "folder" key, not "folderID" 2017-05-30 08:57:18 +02:00
nrm21
1235cead35 lib/model: Add name of latest modifying device to conflict file (fixes #3524)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4161
LGTM: AudriusButkevicius, calmh
2017-05-25 10:26:41 +00:00
Audrius Butkevicius
91d37f35bc lib/model: Use up to date device name, do not provide name to unknown devices (fixes #4164)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4168
2017-05-22 19:58:33 +00:00
Audrius Butkevicius
51518490c6 lib/connections: Fix KCP from locking up the service (fixes #4072) 2017-05-21 22:16:21 +02:00
Simon Frei
0b854dff9d lib/ignore: Don't match root (".")
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4122
2017-05-01 16:58:08 +00:00
Jakob Borg
5045842f4f lib/config: Don't XML serialize deprecated minDiskFreePct 2017-04-27 14:46:19 +09:00
Jakob Borg
e0405de5bf lib/protocol: More descriptive errors on device ID parse failures 2017-04-27 14:45:35 +09:00
Jakob Borg
d6fbfc3545 lib/fs, lib/model, lib/scanner: Make scans cancellable (fixes #3965)
The folder already knew how to stop properly, but the fs.Walk() didn't
and can potentially take a very long time. This adds context support to
Walk and the underlying scanning stuff, and passes in an appropriate
context from above. The stop channel in model.folder is replaced with a
context for this purpose.

To test I added an infiniteFS that represents a large amount of data
(not actually infinite, but close) and verify that walking it is
properly stopped. For that to be implemented smoothly I moved out the
Walk function to it's own type, as typically the implementer of a new
filesystem type might not need or want to reimplement Walk.

It's somewhat tricky to test that this actually works properly on the
actual sendReceiveFolder and so on, as those are started from inside the
model and the filesystem isn't easily pluggable etc. Instead I've tested
that part manually by adding a huge folder and verifying that pause,
resume and reconfig do the right things by looking at debug output.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4117
2017-04-26 00:15:23 +00:00
Jakob Borg
488444354b lib/db: Don't panic
So, when first implementing the database layer I added panics on every
unexpected error condition mostly to be sure to flush out bugs and
inconsistencies. Then it became sort of standard, and we don't seem to
have many bugs here any more so the panics are usually caused by things
like checksum errors on read. But it's not an optimal user experience to
crash all the time.

Here I've weeded out most of the panics, while retaining a few "can't
happen" ones like errors on marshalling and write that we really can't
recover from.

For the rest, I'm mostly treating any read error as "entry didn't
exist". This should mean we'll rescan the file and correct the info (if
scanning) or treat it as a new file and do conflict handling (when
pulling). In some cases things like our global stats may be slightly
incorrect until a restart, if a database entry goes suddenly missing
during runtime.

All in all, I think this makes us a bit more robust and friendly without
introducing too many risks for the user. If the database is truly toast,
probably many other things on the system will be toast as well...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4118
2017-04-25 22:52:37 +00:00
Simon Frei
914b09fd1f lib/model: Create root directory for paused folders (fixes #4094)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4095
2017-04-23 23:50:56 +00:00
Jakob Borg
e9f05d138f lib/db, lib/model: Always use reasonable sized batches (fixes #2250, fixes #4112)
Harmonize how we use batches in the model, using ProtoSize() to judge
the actual weight of the entire batch instead of estimating. Use smaller
batches in the block map - I think we might have though that batch.Len()
in the leveldb was the batch size in bytes, but it's actually number of
operations.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4114
2017-04-22 14:23:33 +00:00
Simon Frei
6b188ebcf3 lib/model: Mark initial scan as finished even if failed and refactor (fixes #4103)
The mechanism to disallow manual scans before the initial scan completed
(#3996) , had the side effect, that if the initial scan failed, no further
scans are allowed. So this marks the initial scan as finished regardless of
whether it succeeded or not.

There was also redundant code in rofolder and a pointless check for folder
health in scanSubsIfHealthy (happens in internalScanFolderSubdirs as well).
This also moves logging from folder.go to ro/rw-folder.go to include the
information about whether it is send-only or send-receive

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4104
2017-04-20 00:20:34 +00:00
Jakob Borg
dd1f7a5ab7 lib/connections: Allow negative ACL entries on devices (fixes #4096)
Prefix an entry with "!" to make it a negative entry. First match wins.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4097
2017-04-13 17:43:29 +00:00
Jakob Borg
d48e46a29c cmd/syncthing: Allow custom event subscriptions (fixes #1879)
This adds a parameter "events" to the /rest/events endpoint. It should
be a comma separated list of the events the consumer is interested in.
When not given it defaults to the current set of events, so it's
backwards compatible.

The API service then manages subscriptions, creating them as required
for each requested event mask. Old subscriptions are not "garbage
collected" - it's assumed that in normal usage the set of event
subscriptions will be small enough. Possibly lower than before, as we
will not set up the disk event subscription unless it's actually used.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4092
2017-04-13 17:14:34 +00:00
Jakob Borg
e3e028c988 lib/protocol: Accept invalid files without blocks (fixes #4093) 2017-04-12 11:28:12 +02:00
Jakob Borg
da34f27546 gui, lib/config, lib/model: Allow absolute values for minimum disk free space (fixes #3307)
This deprecates the current minDiskFreePct setting and introduces
minDiskFree. The latter is, in it's serialized form, a string with a
unit. We accept percentages ("2.35%") and absolute values ("250 k", "12.5
Gi"). Common suffixes are understood. The config editor lets the user
enter the string, and validates it.

We still default to "1 %", but the user can change that to an absolute
value at will.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4087
LGTM: AudriusButkevicius, imsodin
2017-04-12 09:01:19 +00:00
Simon Frei
5521759b23 lib/model: TestIgnores: Add ms sleep on all platforms (ref #3986 #3996)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4083
2017-04-06 09:55:54 +00:00
Jakob Borg
da35820fd5 lib/model: Work around Mac filesystem precision in TestIgnores (ref #3996) 2017-04-03 15:47:23 +02:00
Jakob Borg
2ff08e6c84 lib/sync: Make some tests not depend on real clock
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4065
2017-04-01 11:03:11 +00:00
Simon Frei
25b314f5f1 lib/model, gui: Allow creating and editing ignores of paused folders (fixes #3608)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3996
LGTM: calmh, AudriusButkevicius
2017-04-01 09:58:06 +00:00
Jakob Borg
c5e0c47989 lib/connections, lib/model, gui: Specify allowed networks per device (fixes #219)
This adds a new config AllowedNetworks per device, which when set should
contain a list of network prefixes (192.168.0.0/126 etc) that are
allowed for the given device. The connection service will not attempt
connections to addresses outside of the given networks and incoming
connections will be rejected as well.

I've added the config to the normal device editor and shown it (when
set) in the device summary on the main screen.

There's a unit test for the IsAllowedNetwork method, I've done some
manual sanity testing on top of that.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4073
2017-04-01 09:52:31 +00:00
Jakob Borg
4253f22680 lib/scanner: Use fs.Filesystem for all operations
One more step on the path of the great refactoring. Touches rwfolder a
little bit since it uses the Lstat from fs as well, but mostly this is
just on the scanner as rwfolder is scheduled for a later refactor.

There are a couple of usages of fs.DefaultFilesystem that will in the
end become a filesystem injected from the top, but that comes later.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4070
LGTM: AudriusButkevicius, imsodin
2017-04-01 09:04:11 +00:00
Jakob Borg
bdb56d91b9 lib/model: Honor umask when creating folder directories on Unix (fixes #2519)
Doesn't change the behavior on Windows.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4076
2017-03-31 07:51:23 +00:00
Jakob Borg
b75b4190c8 lib/protocol: Add some consistency checks on incoming index updates (fixes #4053)
With this change we will throw a protocol error on some kinds of
malformed index entries.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4064
2017-03-27 07:21:08 +00:00
kwhite17
1caa683ec1 lib/scanner: Stopped outputting rescan debug message if file doesn't exist locally (fixes #1350)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4028
2017-03-18 00:36:54 +00:00
Jakob Borg
5c27796471 lib/model, lib/scanner: Properly ignore symlinks on Windows (fixes #4035)
Adds a unit test to ensure we don't scan symlinks on Windows. For the
rwfolder, trusts that the logic in the invalid check is correct and that
the check is actually called from the need loop.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4042
2017-03-18 00:25:47 +00:00
Audrius Butkevicius
ceea5ebeb3 lib/connections, vendor: Change KCP mux to SMUX
Closes #4032
2017-03-09 14:03:09 +01:00
Jakob Borg
f35e1ac0c5 lib/config: Oops, update tests 2017-03-08 14:23:48 +01:00
Jakob Borg
0e76f9d93b lib/config: Update default update interval for KCP 2017-03-08 14:18:39 +01:00
Jakob Borg
b3e2665a79 vendor: Update github.com/xtaci/kcp 2017-03-07 14:29:21 +01:00
Jakob Borg
81af29e3e2 lib/config, lib/connections: Configurables for KCP, disable by default
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4030
2017-03-07 12:55:50 +00:00
Audrius Butkevicius
0da0774ce4 lib/connections: Add KCP support (fixes #804)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3489
2017-03-07 12:44:16 +00:00
Simon Frei
1c29a93013 lib/events, cmd/syncthing: Correct GlobalID in debug and mark "since" parameter optional
Skip-check: metalint

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4025
2017-03-07 05:44:47 +00:00
Simon Frei
416811a2a9 cmd/syncthing, lib/config: Pause/resume all devices whithout argument
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3989
LGTM: AudriusButkevicius, calmh
2017-03-04 07:54:13 +00:00
Simon Frei
c20d612736 cmd/syncthing, lib/model: Handle rel/abs paths for config/protected paths (fixes #3183)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3995
2017-03-04 07:49:48 +00:00
HairyFotr
c56c48a777 all: Correct various typos
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4005
2017-02-25 08:12:13 +00:00
Jakob Borg
89f8be40c6 lib/model: Use ignore patterns in remote completion calculation
Basically, if we don't care about the sync status of the file we should
not tag someone else out of sync because they don't have the latest
version. This solves *my* "Syncing - 100%" scenario at least.

The reason this happens seems to be like this, in my situation. I have
three devices, connected in a "line": A-B-C. A is a Mac and litters
.DS_Store files everywhere. I've ignored these, but some escaped into
the folders before I did so. I've also ignored them on B and C but at
different stages. B was flagging C as out of sync, because at the point
the ignores were introduced C had a lower version of .DS_Store than A.
Now none of them are sending updates about it any more since it's
ignored...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3981
2017-02-10 11:22:09 +00:00
Simon Frei
22903df2c1 lib/model: Meaningful error messages for paused folders
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3977
LGTM: AudriusButkevicius, calmh
2017-02-09 20:29:56 +00:00
Jakob Borg
f7fc0c1d3e all: Update license url to https (ref #3976) 2017-02-09 08:04:16 +01:00
Audrius Butkevicius
120e6eab2c lib/sync: Fix a race in unlocker logging (fixes #3884)
Other routines use atomics, hence even if we are under a lock, we should
too.

We might atomically store with
Not sure how it happens, but it's between lines

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3974
2017-02-08 22:31:19 +00:00
Audrius Butkevicius
72de47df00 lib/model: Increase in-flight buffer from 2mb to 8mb, download 2 files at a time
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3972
2017-02-08 20:00:55 +00:00
Jakob Borg
73f9c7d174 lib/versioner: Convert Staggered to a suture.Service (fixes #3820) 2017-02-07 14:33:33 +01:00
Jakob Borg
c4ba580cbb all: Remove symlink support on Windows, SymlinksEnabled config
After this change,

- Symlinks on Windows are always unsupported. Sorry.

- Symlinks are always enabled on other platforms. They are just a small
  file like anything else. There is no need to special case them. If you
  don't want to sync some symlinks, ignore them.

- The protocol doesn't differentiate between different "types" of
  symlinks. If that distinction ever does become relevant the individual
  devices can figure it out by looking at the destination when they
  create the link.

It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
2017-02-07 08:34:24 +00:00
Jakob Borg
9fda9642d3 lib/events: Make test even less timing dependent 2017-02-07 08:57:39 +01:00
Jakob Borg
dfd2c464b6 lib/events: Overflow test should calculate average log time 2017-02-07 08:47:53 +01:00
Jakob Borg
fdcbd54cd7 lib/events: Be more resilient against dropping events (fixes #3952)
Instead of just immediately dropping the event if the subscription isn't
ready to receive it, give it 15 ms to catch up. The value 15 ms is
grabbed out of thin air - it just seems reasonable to me.

The timer juggling makes the event send pretty much exactly twice as
slow as it was before, but we're still under a microsecond. I think it's
negligible compared to whatever event that just happened that we're
interested in logging (usually a file operation of some kind).

	benchmark                  old ns/op     new ns/op     delta
	BenchmarkBufferedSub-8     475           950           +100.00%

	benchmark                  old allocs     new allocs     delta
	BenchmarkBufferedSub-8     4              4              +0.00%

	benchmark                  old bytes     new bytes     delta
	BenchmarkBufferedSub-8     104           117           +12.50%

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3960
2017-02-07 07:25:09 +00:00