Commit Graph

840 Commits

Author SHA1 Message Date
Jakob Borg 992bb0a98c lib/config, lib/discover: Support new discovery cluster (ref #4618)
This adds one new feature, that discovery servers can have ?nolookup to
be used only for announces. The default set of discovery servers is
changed to:

- discovery.s.n used for lookups. This is dual stack load balanced over
  all discovery servers, and returns both IPv4 and IPV6 results when they
  exist.

- discovery-v4.s.n used for announces. This has IPv4 addresses only and
  the discovery servers will update the unspecified address with the IPv4
  source address, as usual.

- discovery-v6.s.n which is exactly the same for IPv6.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4647
2018-01-05 14:18:32 +00:00
Audrius Butkevicius 53f4dfe83c lib/model: Fix panic when auto-accepting existing paused folder (fixes #4636)
This no longer pokes at model internals, and only touches the config.
As a result, model handles this in CommitConfiguration, which restarts
the folders if things change, which repopulate m.folderDevice, m.deviceFolder
and other interal mappings.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4639
2018-01-03 07:42:25 +00:00
Audrius Butkevicius b0e2050cdb cmd/syncthing: UI for version restoration (fixes #2599) (#4602)
cmd/syncthing: Add UI for version restoration (fixes #2599)
2018-01-01 15:39:23 +01:00
Audrius Butkevicius c7f136c2b8 lib/upnp: Each service is it's own NAT device
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4625
2017-12-30 19:16:08 +00:00
Jakob Borg a9f0659f2f lib/fs: Handle deduplicated files on NTFS (fixes #1845)
These files always have the symlink bit set, because they are reparse
points. Nonetheless they are not symlinks, and Lstat reports a size for
them. We use this fact to disambiguate, and hope fervently that nothing
else matches this description so it comes back to bite us...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4622
2017-12-29 21:23:06 +00:00
Lars K.W. Gohlke 9988044bbe lib/model: Cleanup conditions
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4621
2017-12-29 13:14:39 +00:00
Audrius Butkevicius 72172d853c vendor: Move back to upstream KCP (fixes #4407)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4614
2017-12-27 11:33:12 +00:00
Simon Frei 841205dbfe lib/model: Check for invalid filenames after ignore patterns
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4607
LGTM: calmh, AudriusButkevicius
2017-12-25 17:54:34 +00:00
Audrius Butkevicius c58b383b6d gui: Add debug tab to settings (ref #2644)
Just because there are a ton of people struggling to set env vars.
Perhaps this should live in advanced settings, and perhaps we should have a button to view the log.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4604
LGTM: calmh, imsodin
2017-12-24 22:26:05 +00:00
Audrius Butkevicius 2547a29dd7 lib/connections: Don't close nil connections (fixes #4605) 2017-12-18 14:40:51 +00:00
Simon Frei 8fa2b7765a gui, lib/model: Display list of files needed by remote (fixes #4369)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4559
LGTM: AudriusButkevicius, calmh
2017-12-15 20:01:56 +00:00
Jakob Borg c7522063b3 lib/model: Don't leak fd when truncate fails (fixes #4593)
Also attempt to handle this nicer by ignoring the truncate failure when
it doesn't matter, and recover by deleting the temp file when it does.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4594
2017-12-14 10:42:40 +00:00
Jakob Borg d1d967f0cf lib/db: Keep folder meta data persistently in db (fixes #4400)
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).

The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.

A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
2017-12-14 09:51:17 +00:00
Jakob Borg 8c91ced784 cmd/syncthing: Clean up deadlock envvars
So STDEADLOCK seems to do the same thing as STDEADLOCKTIMEOUT, except in
the other package. Consolidate?

STDEADLOCKTHRESHOLD is actually called STLOCKTHRESHOLD, correct the help
text.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4598
2017-12-13 19:40:12 +00:00
Jakob Borg 230fb083fc lib/model: Remove dead code (*sharedPullerState).sourceFile 2017-12-12 11:30:47 +01:00
Pawel Palenica ece1defb2f lib/dialer: Register dialer for socks URL scheme (fixes #4515)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4579
2017-12-08 12:04:43 +00:00
Simon Frei cce634f340 lib/model: Improve scan scheduling and dir del during pull (fixes #4475 #4476)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4493
2017-12-07 08:42:03 +00:00
Jakob Borg 47429d01e8 lib/config, lib/model: Tweaks to the auto accept feature
Fix the folder restart behavior (ignore Label), improve the API for that
(imho).

Also removes the tab switch animation in the settings modal, because
annoying.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4577
2017-12-07 08:33:32 +00:00
Audrius Butkevicius 445c4edeca gui, lib/config, lib/model: Support auto-accepting folders (fixes #2299)
Also introduces a new Waiter interface for config changes and segments the
configuration GUI.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4551
2017-12-07 07:08:24 +00:00
Simon Frei c005b8dcb0 lib/fs: Prolong test timeout on darwin, hopefully fixing flakyness
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4576
2017-12-06 22:07:08 +00:00
Audrius Butkevicius 95a65bf0d0 lib/config: Support symlinked root (fixes #4542, fixes #4353)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4545
LGTM: imsodin, calmh
2017-11-26 07:51:22 +00:00
Jakob Borg 429b3a0429 lib/osutil, lib/scanner: Run symlink test on Windows when possible
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4548
2017-11-25 21:49:53 +00:00
Thomas Hipp b2af8f135b lib/events: Fix unmarshaling of EventType
skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4540
2017-11-22 23:25:55 +00:00
Simon Frei ce29d3a574 all: Various debug logging improvements
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4529
2017-11-22 08:05:27 +00:00
Jakob Borg 6daa766fde lib/connections: Actually fix LAN detection, for real (ref #4534) 2017-11-22 09:01:21 +01:00
Audrius Butkevicius 4922b46fbd lib/connections: Fix local address priority
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4534
LGTM: imsodin, calmh
2017-11-22 07:05:49 +00:00
xjtdy888 a17d953334 lib/connections: Actually make connection attempts for lower priority addresses as well
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4535
2017-11-21 14:58:18 +00:00
Audrius Butkevicius 44a542391e lib/connections: Trust the model to tell us if we are connected
This should address issue as described in https://forum.syncthing.net/t/stun-nig-party-with-paused-devices/10942/13
Essentially the model and the connection service goes out of sync in terms of thinking if we are connected or not.
Resort to model as being the ultimate source of truth.

I can't immediately pin down how this happens, yet some ideas.

ConfigSaved happens in separate routine, so it's possbile that we have some sort of device removed yet connection comes in parallel kind of thing.
However, in this case the connection exists in the model, and does not exist in the connection service and the only way for the connection to be removed
in the connection service is device removal from the config.

Given the subject, this might also be related to the device being paused.

Also, adds more info to the logs

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4533
2017-11-21 07:25:38 +00:00
Jakob Borg 0901350087 lib/model: Trigger a pull when ignore patterns change
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4532
2017-11-20 16:29:36 +00:00
Jakob Borg 72d645865e lib/connections: Fix race condition in parallel dial, minor cleanups (fixes #4526)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4527
2017-11-19 17:38:13 +00:00
Dmitry Saveliev 9471b9f6af lib/versioner: Purge the empty directories in .stversions (fixes #4406)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4514
LGTM: AudriusButkevicius, imsodin
2017-11-18 15:56:53 +00:00
Audrius Butkevicius 0518a92cdb lib/connections: Only announce punchable nats (fixes #4519)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4523
2017-11-17 14:46:45 +00:00
Simon Frei 6cf01c1d30 lib/model: Don't update ignore hash when pull fails
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4522
2017-11-17 12:42:41 +00:00
Jakob Borg 5f4ed66aa1 lib/model: Properly schedule pull on reconnect (fixes #4504)
We need to reset prevSeq so that we force a full check when someone
reconnects - the sequence number may not have changed due to the
reconnect. (This is a regression; we did this before f6ea2a7.)

Also add an optimization: we schedule a pull after scanning, but there
is no need to do so if no changes were detected. This matters now
because the scheduled pull actually traverses the database which is
expensive.

This, however, makes the pull not happen on initial scan if there were
no changes during the initial scan. Compensate by always scheduling a
pull after initial scan in the rwfolder itself.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4508
LGTM: imsodin, AudriusButkevicius
2017-11-17 12:11:45 +00:00
Jakob Borg ee5d0dd43f lib/fs: Add case insensitivity to MtimeFS
This is step one of a hundred fifty on the path to case insensitivity.
It brings in the basic case folding mechanism and adds it to the
mtimefs, as this is something outside the fileset that touches stuff in
the database based on name. No effort to convert or handle existing
entries when the insensitivity is changed, I don't think we need it...

Useless by itself but includes tests and will reduce the review load
along the way.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4521
2017-11-17 12:10:16 +00:00
Jakob Borg 7ebf58f1bc Fix discovery in the absence of listen addresses (fixes #4418)
This makes it OK to not have any listeners working. Specifically,

- We don't complain about an empty listener address
- We don't complain about not having anything to announce to global
  discovery servers
- We don't send local discovery packets when there is nothing to
  announce.

The last point also fixes a thing where the list of addresses for local
discovery was set at startup time and never refreshed.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4517
2017-11-17 09:12:35 +00:00
AudriusButkevicius aecd7c64ce lib/connections: Parallel dials in the same priority (fixes #4456)
Well Tested(TM)

Introduces a potential issue where we always pick some connectable but dodgy connection that breaks
soon after the TLS handshake.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4489
2017-11-15 09:36:33 +00:00
Simon Frei 4efff736b3 lib/connections: Consistent log levels & polish (fixes #4510)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4511
2017-11-14 21:49:36 +00:00
Simon Frei fa12a18190 lib/model: Handle type changes when pulling (ref #4505 #4506 #4507)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4509
LGTM: AudriusButkevicius, calmh
2017-11-13 15:16:27 +00:00
Jakob Borg 2b65e1062e lib/model: Fix rescan detection (fixes #4505, fixes #4506)
Diff is large due to comment reformatting and indentation but all it
does is wrap the file mtime/size/permissions check in an "if
stat.IsRegular()".

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4507
2017-11-13 06:57:07 +00:00
Jakob Borg 6e148e20cd test: Mend tests for latest event changes etc 2017-11-13 00:25:07 +01:00
Jakob Borg d7d45d8092 lib/db: Refactor away the large genericReplace thing
This removes a significant, complex chunk of database code. The
"replace" operation walked both the old and new in lockstep and made the
relevant changes to make the new situation correct. But since delta
indexes we pretty much never need this - we just used replace to drop
the existing data and start over.

This makes that explicit and removes the complexity.

(This is one of those things that would be annoying to make case
insensitive, while the actual "drop and then insert" that we do is
easier.)

This is fairly well unit tested...

The one change to the tests is to cover the fact that previously replace
with something identical didn't bump the sequence number, while
obviously removing everything and re-inserting does. This is not
behavior we depend on anywhere.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4500
LGTM: imsodin, AudriusButkevicius
2017-11-12 20:20:34 +00:00
Simon Frei c080f677cb all: Add invalid/ignored files to global list, announce to peers (fixes #623)
This lets us determine accurate completion status for remote peers when they
have ignored files.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4460
2017-11-11 19:18:17 +00:00
Simon Frei ec4c3bae0d lib/watchaggregator: Don't care about timings during testing on darwin
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4502
2017-11-10 17:05:31 +00:00
Audrius Butkevicius 386cb274bd lib/model: Incremental block stats usage reporting
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4498
LGTM: calmh
2017-11-09 21:16:29 +00:00
Jakob Borg a9d422e008 Revert "lib/model: Use the new file name when requesting rescan"
This has consequences that I haven't thought through for case mismatches
(i.e. we have foo.txt in the database and queue a rescan for FOO.txt).
2017-11-09 13:55:45 +01:00
Jakob Borg 75f64733f9 lib/model: Use the new file name when requesting rescan
If the file is not already in the database, the "curfile" will be empty
and the rescan request cover the entire folder.
2017-11-09 13:48:39 +01:00
AudriusButkevicius 0ee1146e1c lib/connections: Indicate stack in transport (fixes #4463)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4486
2017-11-07 07:25:05 +00:00
Simon Frei f6ea2a7f8e lib/model: Trigger pulls instead of pulling periodically
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4340
2017-11-07 06:59:35 +00:00
AudriusButkevicius 62a4106a79 lib/connections: Fix lan detection (fixes #4421)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4487
2017-11-06 14:05:29 +00:00