Commit Graph

161 Commits

Author SHA1 Message Date
Jakob Borg
46a143e80e lib/model: Handle deleted-then-ignored files (fixes #3502)
When files that were previously marked as deleted became ignored, we
used to do nothing at all. This changes that behavior to set the Invalid
bit (that we should rename to Ignored). This then becomes an update to
other devices that they should not trust our knowledge about the file in
question.

Read this diff without whitespace...

Tested by
- creating a bunch of files on s1
- letting them sync to s2
- shutting down s2
- deleting the files on s1 and rescanning
- adding the files to .stignore on s1 and rescanning
- starting up s2 and letting it sync
- observing the files are not deleted on s2, and it considers itself up
  to date.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3557
2016-09-02 13:23:24 +00:00
Jakob Borg
69b7f26e4c lib/model, cmd/syncthing: Also account for deleted files in folder summary events (ref #3496)
This should probably be reflected in the GUI somewhere as well...
2016-09-02 10:45:39 +02:00
Jakob Borg
5b37d0356c lib/model, gui: Correct completion percentages when there are lots of deletes (fixes #3496)
We used to consider deleted files & directories 128 bytes large. After
the delta indexes change a bug slipped in where deleted files would be
weighted according to their old non-deleted size. Both ways are
incorrect (but the latest change made it worse), as if there are more
files deleted than remaining data in the repo the needSize can be
greater than the globalSize, resulting in a negative completion
percentage.

This change makes it so that deleted items are zero bytes large, which
makes more sense. Instead we expose the number of files that we need to
delete as a separate field in the Completion() result, and hack the
percentage down to 95% complete if it was 100% complete but we need to
delete files. This latter part is sort of ugly, but necessary to give
the user some sort of feedback.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3556
2016-09-02 06:45:46 +00:00
Jakob Borg
0b1475169f lib/model: Correct virtual mtime handling (fixes #3516)
We previously set the mtime on the temp file, and then renamed it to the
real path. Unfortunately that means we'd save the real timestamp under
the under the temp name ".syncthing.foo.tmp" when the actual file that
we will look up on the next scan is "foo". This moves the Chtimes later,
ensuring that it gets recorded correctly under the right name.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3519
2016-08-16 18:22:19 +00:00
Audrius Butkevicius
6ec4fbc82b lib/model: Add minumum interval for progress emitter (fixes #3517)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3518
2016-08-16 18:22:01 +00:00
Jakob Borg
18cc7a663b lib: Remove osutil.Remove & osutil.RemoveAll (fixes #3513)
These are no longer required with Go 1.7. Change made by removing the
functions, doing a global s/osutil.Remove/os.Remove/.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3514
2016-08-16 10:01:58 +00:00
Jakob Borg
cf5febad47 build, cmd, lib: Minimum supported compiler version is Go 1.5 2016-08-15 08:37:32 +02:00
Jakob Borg
7776839c82 cmd/syncthing, gui: Improve completion calculation (fixes #3492)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3493
2016-08-12 06:41:43 +00:00
Jakob Borg
e52be3d83e lib/connections, lib/model: Refactor connection close handling (fixes #3466)
So there were some issues here. The main problem was that
model.Close(deviceID) was overloaded to mean "the connection was closed
by the protocol layer" and "i want to close this connection". That meant
it could get called twice - once *to* close the connection and then once
more when the connection *was* closed.

After this refactor there is instead a Closed(conn) method that is the
callback. I didn't need to change the parameter in the end, but I think
it's clearer what it means when it takes the connection that was closed
instead of a device ID. To close a connection, the new close(deviceID)
method is used instead, which only closes the underlying connection and
leaves the cleanup to the Closed() callback.

I also changed how we do connection switching. Instead of the connection
service calling close and then adding the connection, it just adds the
new connection. The model knows that it already has a connection and
makes sure to close and clean out that one before adding the new
connection.

To make sure to sequence this properly I added a new map of channels
that get created on connection add and closed by Closed(), so that
AddConnection() can do the close and wait for the cleanup to happen
before proceeding.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3490
2016-08-10 09:37:32 +00:00
Audrius Butkevicius
af3b6f9c83 lib/model, lib/config: Support "live" device removal, folder unsharing and folder configuration changes
Furthermore:
1. Cleans configs received, migrates them as we receive them.
2. Clears indexes of devices we no longer share the folder with

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3478
2016-08-07 16:21:59 +00:00
Jakob Borg
ea87bcefd6 lib/protocol, lib/model: Implement high precision time stamps (fixes #3305)
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.

The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
2016-08-06 13:05:59 +00:00
Jakob Borg
0655991a19 lib/db, lib/fs, lib/model: Introduce fs.MtimeFS, remove VirtualMtimeRepo
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3479
2016-08-05 17:45:45 +00:00
Jakob Borg
f368d2278f lib/config, lib/connections: Refactor handling of ignored devices (fixes #3470)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3471
2016-08-05 09:29:49 +00:00
Jakob Borg
1eb6db6ca8 cmd/syncthing, lib/...: Correctly handle ignores & invalid file names (fixes #3012, fixes #3457, fixes #3458)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3464
2016-08-05 07:13:52 +00:00
Audrius Butkevicius
08b5a7908f gui: Add one-off notifications that need to be acked
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3254
2016-08-02 08:07:30 +00:00
Jakob Borg
297240facf all: Rename LocalVersion to Sequence (fixes #3461)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
2016-07-29 19:54:24 +00:00
Jakob Borg
72026db599 lib/db, lib/model: Create temp sorting database in config dir (fixes #3449)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3454
2016-07-27 21:38:43 +00:00
Jakob Borg
aafc96f58f lib/model, lib/protocol: Sequence ClusterConfig properly (fixes #3448)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3452
2016-07-27 21:36:25 +00:00
Jakob Borg
7c7e8648ff lib/model: Trigger a puller iteration on connection (fixes #3451)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3453
2016-07-27 21:35:41 +00:00
Jakob Borg
7aaa1dd8a3 lib/scanner: Recheck file size and modification time after hashing (ref #3440)
To catch the case where the file changed. Also make sure we never let a
size-vs-blocklist mismatch slip through.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3443
2016-07-26 08:51:39 +00:00
Jakob Borg
6ed22d0885 lib/model: Stricter temporary file permissions
We could have a file to sync with permissions rw------- but we'd create
the temp file with rw-rw-rw- minus umask, usually rw-r--r--. This
potentially exposes private data while the file is being synced.

Similarly, when ignorePerms was set and we were reusing a temp files we
would set the permissions to rw-r--r-- explicitly, potentially
overriding a strict umask that would otherwise have had the file be
rw-------.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3437
2016-07-25 10:18:05 +00:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
8ab6b60778 lib/model: Sort outgoing index updates by LocalVersion
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3411
2016-07-21 17:21:15 +00:00
Jakob Borg
3176629410 cmd, lib: Fix ineffectual assignments (ineffasign) and comment spelling
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3405
2016-07-15 14:23:20 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00
Jakob Borg
6d357211b2 lib/config: Remove "Invalid" attribute (fixes #2471)
This contains the following behavioral changes:

 - Duplicate folder IDs is now fatal during startup
 - Invalid folder flags in the ClusterConfig is fatal for the connection
   (this will go away soon with the proto changes, as we won't have any
   unknown flags any more then)
 - Empty path is a folder error reported at runtime

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3370
2016-07-02 19:38:39 +00:00
Nicolas Braud-Santoni
8e39e2889d lib: Fix typos in connections/service.go and model/model.go
Skip-check: authors

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3366
LGTM: aviau
2016-06-30 13:42:53 +00:00
Lars K.W. Gohlke
af0bc95de5 lib/model: Refactor encapsulation of the folder scanning
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3017
2016-06-29 06:37:34 +00:00
Jakob Borg
a165838cbd lib/model: Decrease max temp filename length (fixes #3338, fixes #3355)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3356
2016-06-27 11:47:40 +00:00
Jakob Borg
b0d03d1f1c lib/config: Retain slash at end of path after expanding ~ (fixes #2782)
The various path cleaning operations done in in cleanedPath() removes
it, so we make sure it's added again at the end. This makes adding the
slash in prepare() unnecessary, but keep it anyway for display purposes
(people looking at the config).

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3342
2016-06-26 10:17:20 +00:00
Jakob Borg
9323f0faf8 lib/model: Refactor CheckFolderHealth into separate methods
While attempting to fix #2782 I thought the problem was the
CheckFolderHealth method, so I cleaned it up. That turned out not to be
the case, but I think this is better anyhow.

It also moves the "create folder and marker if the folder was empty in
the index" code to StartFolder where I think it makes better sense.

This is covered by a number of existing tests.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3343
2016-06-26 10:07:27 +00:00
Jakob Borg
f343c8ba36 lib/model, lib/scanner: Silence vet warnings
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3333
2016-06-20 21:00:39 +00:00
Jakob Borg
b779e22205 lib/model: Don't set ignore bit when it's already set
This adds a metric for "committed items" to the database instance that I
use in the test code, and a couple of tests that ensure that scans that
don't change anything also don't commit anything.

There was a case in the scanner where we set the invalid bit on files
that are ignored, even though they were already ignored and had the
invalid bit set. I had assumed this would result in an extra database
commit, but it was in fact filtered out by the Set... Anyway, I think we
can save some work on not pushing that change to the Set at all.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3298
2016-06-13 17:44:03 +00:00
Jakob Borg
d507126101 lib/protocol: Understand older/newer Hello messages (fixes #3287)
This is in preparation for future changes, but also improves the
handling when talking to pre-v0.13 clients. It breaks out the Hello
message and magic from the rest of the protocol implementation, with the
intention that this small part of the protocol will survive future
changes.

To enable this, and future testing, the new ExchangeHello function takes
an interface that can be implemented by future Hello versions and
returns a version indendent result type. It correctly detects pre-v0.13
protocols and returns a "too old" error message which gets logged to the
user at warning level:

   [I6KAH] 09:21:36 WARNING: Connecting to [...]:
     the remote device speaks an older version of the protocol (v0.12) not
     compatible with this version

Conversely, something entirely unknown will generate:

   [I6KAH] 09:40:27 WARNING: Connecting to [...]:
     the remote device speaks an unknown (newer?) version of the protocol

The intention is that in future iterations the Hello exchange will
succeed on at least one side and ExchangeHello will return the actual
data from the Hello together with ErrTooOld and an even more precise
message can be generated.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3289
2016-06-09 10:50:14 +00:00
Majed Abdulaziz
48245effdf lib/model, lib/stats: Keep track of folder's last scan time (ref #3143)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3250
LGTM: calmh, AudriusButkevicius
2016-06-02 19:26:52 +00:00
Audrius Butkevicius
92a23da3ec lib/model: Make the (?d) prefix actually work
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3205
2016-05-28 04:17:34 +00:00
Jakob Borg
9bb5988b4e lib/model: Don't deadlock when returning temp index block counts
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3194
2016-05-26 09:16:08 +00:00
Audrius Butkevicius
fc173bf679 lib/model: Fix wild completion percentages
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3188
2016-05-26 06:53:27 +00:00
Audrius Butkevicius
915e1ac7de lib/model: Handle (?d) deletes of directories (fixes #3164)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3170
2016-05-23 23:32:08 +00:00
Audrius Butkevicius
00be2bf18d lib/model: Track puller creation times (fixes #3145)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3150
2016-05-22 10:16:09 +00:00
Jakob Borg
44290a66b7 lib/model: Leave temp file in place when final rename fails (fixes #3146)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3148
2016-05-22 09:06:07 +00:00
Jakob Borg
a0ec4467fd cmd/syncthing: Emit new RemoteDownloadProgress event to track remote download progress
Without this the summary service doesn't know to recalculate completion
percentage for remote devices when DownloadProgress messages come in.
That means that completion percentage isn't updated in the GUI while
transfers of large files are ongoing. With this change, it updates
correctly.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3144
2016-05-22 07:52:08 +00:00
Jakob Borg
1dbc310c9b cmd/syncthing: Rename event LocalDiskUpdated -> LocalChangeDetected
I think this better reflects what it means. Also tweaks the verbose
format to be more like our other things and lightly refactors the code
to not have the boolean and include the folder in the event.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3121
2016-05-19 07:01:43 +00:00
Nate Morrison
86ca58e2a9 lib/model: Emit LocalDiskUpdated events on detecting local changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3055
2016-05-19 00:19:26 +00:00
Jakob Borg
2c1323ece6 lib/connections: Un-deprecate relaysEnabled (fixes #3074)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3098
2016-05-17 00:05:38 +00:00
Jakob Borg
6f743f3138 Revert "lib/model: Emit LocalDiskUpdated events on detecting local changes"
This reverts commit 5a7fad0bcd.
2016-05-14 10:55:24 +02:00
Nate Morrison
5a7fad0bcd lib/model: Emit LocalDiskUpdated events on detecting local changes
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3055
2016-05-14 08:37:07 +00:00
Jakob Borg
b993b41847 lib/config: Minor attribute updates
As discussed in
https://github.com/syncthing/docs/pull/169

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3082
2016-05-12 08:23:18 +00:00
Jakob Borg
21e116aa45 lib/scanner: Refactor scanner.Walk API
The old usage pattern was to create a Walker with a bunch of attributes,
then call Walk() on it and nothing else. This extracts the attributes
into a Config struct and exposes a Walk(cfg Config) method instead, as
there was no reason to expose the state-holding walker type.

Also creates a few no-op implementations of the necessary interfaces
so that we can skip nil checks and simiplify things here and there.

Definitely look at this diff without whitespace.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3060
2016-05-09 18:25:39 +00:00
Jakob Borg
1a703efa78 lib/model: Fix accounting error in rescan with multiple subs (fixes #3028)
When doing prefix scans in the database, "foo" should not be considered
a prefix of "foo2". Instead, it should match "foo" exactly and also
strings with the prefix "foo/". This is more restrictive than what the
standard leveldb prefix scan does so we add some code to enforce it.

Also exposes the initialScanCompleted on the rwfolder for testing, and
change it to be a channel (so we can wait for it from another
goroutine). Otherwise we can't be sure when the initial scan has
completed, and we need to wait for that or it might pick up changes
we're doing at an unexpected time.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3067
2016-05-09 12:56:21 +00:00