Commit Graph

23 Commits

Author SHA1 Message Date
Jakob Borg
b1b68ceedb
Add LocalFlags to FileInfo (#4952)
We have the invalid bit to indicate that a file isn't good. That's enough for remote devices. For ourselves, it would be good to know sometimes why the file isn't good - because it's an unsupported type, because it matches an ignore pattern, or because we detected the data is bad and we need to rescan it.

Or, and this is the main future reason for the PR, because it's a change detected on a receive only device. We will want something like the invalid flag for those changes, but marking them as invalid today means the scanner will rehash them. Hence something more fine grained is required.

This introduces a LocalFlags fields to the FileInfo where we can stash things that we care about locally. For example,

    FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
    FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
    FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully

The LocalFlags fields isn't sent over the wire; instead the Invalid attribute is calculated based on the flags at index sending time. It's on the FileInfo anyway because that's what we serialize to database etc.

The actual Invalid flag should after this just be considered when building the global state and figuring out availability for remote devices. It is not used for local file index entries.
2018-06-24 09:50:18 +02:00
Audrius Butkevicius
ef0dcea6a4 lib/model: Verify request content against weak (and possibly strong) hash (#4767) 2018-05-05 10:24:44 +02:00
Jakob Borg
19c7cd99f5 all: Implement variable sized blocks (fixes #4807) 2018-04-16 19:08:50 +01:00
Jakob Borg
d1d967f0cf lib/db: Keep folder meta data persistently in db (fixes #4400)
This keeps the data we need about sequence numbers and object counts
persistently in the database. The sizeTracker is expanded into a
metadataTracker than handled multiple folders, and the Counts struct is
made protobuf serializable. It gains a Sequence field to assist in
tracking that as well, and a collection of Counts become a CountsSet
(for serialization purposes).

The initial database scan is also a consistency check of the global
entries. This shouldn't strictly be necessary. Nonetheless I added a
created timestamp to the metadata and set a variable to compare against
that. When the time since the metadata creation is old enough, we drop
the metadata and rebuild from scratch like we used to, while also
consistency checking.

A new environment variable STCHECKDBEVERY can override this interval,
and for example be set to zero to force the check immediately.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4547
LGTM: imsodin
2017-12-14 09:51:17 +00:00
Simon Frei
c080f677cb all: Add invalid/ignored files to global list, announce to peers (fixes #623)
This lets us determine accurate completion status for remote peers when they
have ignored files.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4460
2017-11-11 19:18:17 +00:00
Jakob Borg
c4ba580cbb all: Remove symlink support on Windows, SymlinksEnabled config
After this change,

- Symlinks on Windows are always unsupported. Sorry.

- Symlinks are always enabled on other platforms. They are just a small
  file like anything else. There is no need to special case them. If you
  don't want to sync some symlinks, ignore them.

- The protocol doesn't differentiate between different "types" of
  symlinks. If that distinction ever does become relevant the individual
  devices can figure it out by looking at the destination when they
  create the link.

It's backwards compatible in that all the old symlink types are still
understood to be symlinks, and the new SYMLINK type is equivalent to the
old SYMLINK_UNKNOWN which was always a valid way to do it.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3962
LGTM: AudriusButkevicius
2017-02-07 08:34:24 +00:00
Jakob Borg
987718baf8 vendor: Update github.com/gogo/protobuf
Also tweaks the proto definitions:

 - [packed=false] on the block_indexes field to retain compat with
   v0.14.16 and earlier.

 - Uses the vendored protobuf package in include paths.

And, "build.go setup" will install the vendored protoc-gen-gogofast.
This should ensure that a proto rebuild isn't so dependent on whatever
version of the compiler and package the developer has installed...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3864
2017-01-03 00:16:21 +00:00
Jakob Borg
6809d38cde lib/protocol: Revert protobuf encoder changes in v0.14.17 (fixes #3855)
The protobuf encoder now produces packed arrays for things like []int32,
which is actually correct according to the proto3 spec. However
Syncthing v0.14.16 and earlier doesn't support this. This reverts the
encoding change, but keeps the updated decoder so that we are both more
compatible with other proto3 implementations and can move to the updated
encoder in the future.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3856
2017-01-01 17:19:00 +00:00
Audrius Butkevicius
bab7c8ebbf all: Add folder pause, make pauses permanent (fixes #3407, fixes #215, fixes #3001)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3520
2016-12-21 18:41:25 +00:00
Nathan Morrison
0725e3af38 all: Add a global change list
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3694
2016-12-21 16:35:20 +00:00
Audrius Butkevicius
0582836820 lib/model, lib/scanner: Efficient inserts/deletes in the middle of the file
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3527
2016-12-14 23:30:29 +00:00
Jakob Borg
7b07ed6580 lib/model, lib/protocol, lib/scanner: Include symlink target in index, pull symlinks synchronously
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3792
2016-12-09 18:02:18 +00:00
Audrius Butkevicius
a1a91d5ef4 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-13 09:29:33 +01:00
Jakob Borg
59f3d1445f Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit 0b88cf1d03.
2016-11-12 08:38:29 +01:00
Audrius Butkevicius
0b88cf1d03 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-11 15:54:25 +01:00
Jakob Borg
3f9b75b7b3 Revert "lib/model: Introducer can remove stuff it introduced (fixes #1015)"
This reverts commit ec2b097313.
2016-11-08 14:27:32 +08:00
Audrius Butkevicius
ec2b097313 lib/model: Introducer can remove stuff it introduced (fixes #1015)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3522
2016-11-08 00:40:48 +08:00
Audrius Butkevicius
14937e7dd2 build: Fix proto builder on Windows 2016-11-03 22:06:51 +00:00
Jakob Borg
0296c23685 lib/protocol: Use DeviceID in protocol messages, with custom marshalling
This makes the device ID a real type that can be used in the protobuf
schema. That avoids the juggling back and forth from []byte in a bunch
of places and simplifies the code.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3695
2016-10-29 21:56:24 +00:00
Jakob Borg
ea87bcefd6 lib/protocol, lib/model: Implement high precision time stamps (fixes #3305)
This adds a new nanoseconds field to the FileInfo, populates it during
scans and sets the non-truncated time in Chtimes calls.

The actual file modification time is defined as modified_s seconds +
modified_ns nanoseconds. It's expected that the modified_ns field is <=
1e9 (that is, all whole seconds should go in the modified_s field) but
not really enforced. Given that it's an int32 the timestamp can be
adjusted += ~2.9 seconds by the modified_ns field...

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3431
2016-08-06 13:05:59 +00:00
Jakob Borg
297240facf all: Rename LocalVersion to Sequence (fixes #3461)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3462
2016-07-29 19:54:24 +00:00
Jakob Borg
47fa4b0a2c cmd/syncthing, lib/db, lib/model, lib/protocol: Implement delta indexes (fixes #438)
GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3427
2016-07-23 12:46:31 +00:00
Jakob Borg
fa0101bd60 lib/protocol, lib/discover, lib/db: Use protocol buffer serialization (fixes #3080)
This changes the BEP protocol to use protocol buffer serialization
instead of XDR, and therefore also the database format. The local
discovery protocol is also updated to be protocol buffer format.

GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/3276
LGTM: AudriusButkevicius
2016-07-04 10:40:29 +00:00