Commit Graph

2109 Commits

Author SHA1 Message Date
Jakob Borg
c1ec9a8826
lib/fs: Reduce memory usage in xattrs handling (#9251)
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)

With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:

	Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
	Dropped 1435 nodes (cum <= 7.26MB)
	Showing top 10 nodes out of 54
	      flat  flat%   sum%        cum   cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New

After the change, it's this:

	Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
	Dropped 1400 nodes (cum <= 2.62MB)
	Showing top 10 nodes out of 91
	      flat  flat%   sum%        cum   cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
	   15.28MB  2.91% 87.57%    15.28MB  2.91%  strings.(*Builder).grow

(The usage for xattrs is reduced from 976 MB to 68 MB)
2023-12-04 12:48:17 +01:00
Jakob Borg
1625b44892
lib/model: Improve LastSeen handling (#9256)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
2023-12-04 09:24:10 +01:00
Jakob Borg
d51760f410
lib/scanner: Record inode change time for directories and symlinks (#9250) 2023-12-04 07:48:24 +01:00
Jakob Borg
7b1932d64e
lib/api: Improve ignore loading error handling (fixes #9253) (#9254) 2023-12-04 07:11:35 +01:00
Jakob Borg
4cba99fcd4 lib/fs: Better equality comparison in mtimefs 2023-12-03 16:01:46 +01:00
Simon Frei
958ff67ccc
lib/model: Acquire fmut lock in ensureIndexHandler (fixes #9234) (#9235)
Towards the end of the function f.folderCfgs and f.folderRunners are
read without the lock.
2023-11-20 08:12:25 +01:00
Jakob Borg
439c6c5b7c
lib/api: Add cache busting for basic auth (ref #9208) (#9215)
This adds our short device ID to the basic auth realm. This has at least
two consequences:

- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.

- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.

I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...

Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
2023-11-14 11:57:39 +01:00
Jakob Borg
8f1b0df74b
lib/api: Improve cookie handling (fixes #9208) (#9214) 2023-11-13 20:37:29 +01:00
Jakob Borg
bae6d5f375
lib/location: Fix regression of timestamp handling (ref #9180) (#9185) 2023-10-26 07:41:02 +02:00
Jakob Borg
b5082f6af8
lib/locations: Change default config/data location to new XDG recommendation (fixes #9178, fixes #9179) (#9180)
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.

Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).

### Documentation

Needs update to the config docs about our default locations.
2023-10-25 11:16:24 +02:00
tomasz1986
16ae1fbe5e
lib/fs: Ignore inode change time on Android (#9177)
lib/fs: Fix conflicts on Android due to fluctuating inode change time

[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.

[1] 6cac308bcd

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2023-10-21 08:24:29 +02:00
Emil Lundberg
14569f12d3
Hide log out button when auth is not enabled (#9158)
This was an oversight in #8757: the new "Log out" button is always shown
in the "Actions" menu, even when authentication is not enabled.
2023-10-15 14:10:41 +02:00
Eric P
9553365d31
lib/fs: Properly handle Windows deduplicated files (fixes #9120) (#9168)
### Purpose

Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.

### Testing

Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
2023-10-11 14:40:55 +02:00
Emil Lundberg
ea1ea366d2 lib/api: Check basic auth (and set session cookie) before noauth exceptions (#9159)
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554

The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.

This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.

`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).

And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.

I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d7034302b729ada6d91cff6e2b29678da fail
before the change in 0e47d37e738d4c15736c496e01cd949afb372e71 is
applied.
2023-10-10 07:48:55 +02:00
Jakob Borg
3d0da5ac60
lib/api: Better handle %s templates in LDAP strings (fixes #9072) (#9155)
Also add some escaping for good measure.
2023-10-07 02:29:53 +00:00
Jakob Borg
a64ae36bcc
lib/model: Verify versioning on configuration reload (fixes #9106) (#9154) 2023-10-07 04:09:51 +02:00
Emil Lundberg
8294870ffc
Add HTML login form (fixes #4137) (#8757) 2023-10-06 13:00:58 +02:00
Jakob Borg
296db314f5
lib/config: Improve parsing of gui-address overrides (#9144)
improve parsing of gui-address overrides

make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".

the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.

the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.

simplify the `useTLS()` function's codepath for overrides.

Co-authored-by: digital <didev@dinid.net>
2023-10-02 08:40:03 +02:00
Jakob Borg
b91d7711aa
Update dependencies (#9129)
And some QUIC API changes, of course.
2023-09-25 21:45:57 +02:00
Jakob Borg
6ed9c0c34c
lib/config: Accept pre-hashed password (fixes #9123) (#9124) 2023-09-24 19:23:49 +02:00
bt90
cf46bf0297
lib/connections: Fix transport type detection for QUIC (fixes #8274) (#9114)
Check remote address
2023-09-20 11:23:48 +02:00
Jakob Borg
051cbdc713
lib/fs, lib/model: Be careful about potentially negative durations (fixes #9112) (#9113)
I don't really understand under what circumstances, but sometimes these
calls panic with a "panic: counter cannot decrease in value" because the
value passed to Add() was negative.
2023-09-20 09:04:47 +02:00
Jakob Borg
f47de83914
lib/protocol: Ensure starting & closing a connection are exclusive (fixes #9102) (#9103)
In principle a connection can close while it's in progress with
starting, and then it's undefined if we wait for goroutines to exit etc.
With this change, we will wait for start to complete before starting to
stop everything.
2023-09-12 14:48:15 +02:00
bt90
e860d3b974
lib/connections: Make assumptions about isLAN when interface address listing fails (#9093) 2023-09-12 12:34:30 +00:00
bt90
ed66fba42b
lib/beacon, lib/discover: Send IPv4 limited broadcast when address listing fails (fixes #1628) (#9087) 2023-09-12 14:28:17 +02:00
Jakob Borg
4812600098
lib/versioner: Don't complain when folder is stopping (#9097) 2023-09-11 23:10:18 +02:00
Jakob Borg
7c0223bd06 lib/build: Next version is the Gold Grasshopper 2023-09-06 13:13:39 +02:00
Jakob Borg
c6334e61aa
all: Support multiple device connections (fixes #141) (#8918)
This adds the ability to have multiple concurrent connections to a single device. This is primarily useful when the network has multiple physical links for aggregated bandwidth. A single connection will never see a higher rate than a single link can give, but multiple connections are load-balanced over multiple links.

It is also incidentally useful for older multi-core CPUs, where bandwidth could be limited by the TLS performance of a single CPU core -- using multiple connections achieves concurrency in the required crypto calculations...

Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: tomasz1986 <twilczynski@naver.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
2023-09-06 12:52:01 +02:00
Maximilian
c42c0e7ceb
lib/connections: Fix WANAddresses returning only unspecified IPs (ref #9010) (#9073)
Avoids taking the address of the same variable twice.
2023-09-03 15:03:27 +00:00
Jakob Borg
5118538179
lib/model: Refactor folderRunners to use a serviceMap (#9071)
Instead of separately tracking the token.

Also changes serviceMap to have a channel version of RemoveAndWait, so
that it's possible to do the removal under a lock but wait outside of
the lock. And changed where we do that in connection close, reversing
the change that happened when I added the serviceMap in 40b3b9ad1.
2023-09-02 16:42:46 +02:00
Jakob Borg
3130af3773
lib/upgrade: Enable HTTP/2 for upgrade checks (#9060) 2023-08-30 21:58:34 +02:00
Jakob Borg
abd89f15f7
lib/discover: Enable HTTP/2 for global discovery requests (#9059)
By creating the http.Transport and tls.Configuration ourselves we
override some default behavior and end up with a client that speaks only
HTTP/1.1.

This adds a call to http.ConfigureTransport to do the relevant magic to
enable HTTP/2.

Also tweaks the keepalive settings to be a little kinder to the
server(s).
2023-08-30 21:58:05 +02:00
bt90
467522d04d
lib/connections: Allow IPv6 ULA in discovery announcements (fixes #7456) (#9048)
The allowed IPv4 ranges are the same as before. But we now also accept IPv6 addresses in the ULA range FC00::/7. These addresses don't require an interface identifier and are roughly equivalent to the IPv4 private ranges.

Typical usecases:

VPN interface IPs: Wireguard, OpenVPN, Tailscale, ...
fixed IPv6 LAN addressing while the provider assigns a dynamic prefix. e.g used by pihole
https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/net/ip.go;l=146
2023-08-23 12:28:48 +02:00
bt90
3147285c60
lib/beacon: Check FlagRunning (#9051) 2023-08-22 11:27:43 +02:00
Jakob Borg
acd767b30b
all: Remove lib/util package (#9049)
Grab-bag packages are nasty, this cleans it up a little by splitting it
into topical packages sempahore, netutil, stringutil, structutil.
2023-08-21 19:44:33 +02:00
Jakob Borg
40b3b9ad15
lib/model: Clean up index handler life cycle (fixes #9021) (#9038)
Co-authored-by: Simon Frei <freisim93@gmail.com>
2023-08-21 18:39:13 +02:00
bt90
c2c6133aa5
lib/osutil, lib/upnp: Check FlagRunning (fixes #8767) (#9047) 2023-08-21 14:49:28 +00:00
Jakob Borg
cbf0e31f69
all: Use Go 1.21, new QUIC API (#9040) 2023-08-21 15:25:52 +02:00
Jakob Borg
462389934b cmd/stupgrades: Serve friendlier URLs for upgrade assets (fixes #9033) 2023-08-09 21:01:15 +02:00
André Colomb
a8cacdca94
lib/versioner: Minor fixes in comments and error message (#9031)
* lib/versioner: Factor out DefaultPath constant.

Replace several instances where .stversions is named literally to all
use the same definition in the versioner package.  Exceptions are the
packages where a cyclic dependency on versioner is impossible, or some
tests which combine the versions base path with other components.

* lib/versioner: Fix comment about trash can in simple versioner.

* lib/versioner: Fix wrong versioning type string in error message.

The error message shows the folder type instead of the versioning
type, although the correct field is used in the comparison.
2023-08-09 07:10:06 +00:00
Jakob Borg
8b87cd5229
lib/model: Reinstate setting folder idle state (#9029) 2023-08-08 07:24:02 +02:00
Jakob Borg
b9c08d3814
all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) (#9003) 2023-08-04 19:57:30 +02:00
Keith Harrison
eed12f3ec5
lib/config: Allow sharing already encrypted folder with untrusted devices (fixes #8965) (#9012)
Safety check added in v1.23.6 introduced bug. Bug unshares folders with untrusted devices if folder does not have an encryption password set, regardless of whether the folder is shared with the untrusted device as encrypted or not. Prevents sharing with untrusted devices in some cases where sharing would be encrypted.

Patch preserves safety check but permits sharing folders with untrusted devices if they are shared as encrypted.

Signed-off-by: kewiha <keithh@protonmail.com>
2023-08-02 07:14:53 +00:00
Jakob Borg
9d21b91124
all: Refactor the protocol/model interface a bit (ref #8981) (#9007) 2023-07-29 10:24:44 +02:00
Chih-Hsuan Yen
b806026990
lib/connections: Fix building with -tags noquic (#9009) 2023-07-28 10:08:50 +00:00
Emil Lundberg
b08b99e284
lib/api: Fix data race in TestCSRFRequired (#9006) 2023-07-26 21:33:45 +00:00
Jakob Borg
855c6dc67b
lib/api: Allow Bearer authentication style with API key (#9002)
Currently, historically, we look for the `X-API-Key` header to
authenticate with an API key. There's nothing wrong with this, but in
some scenarios it's easier to produce an `Authorization` header with a
`Bearer $token` content, which is nowadays more common. This change adds
support for both, so that we will accept an API key either in our custom
header or as a bearer token.
2023-07-26 13:13:06 +02:00
tomasz1986
f42f041f53
lib/ur: Don't report uptime if start time is in the past (fixes #7698) (#8996)
Currently, because of devices with unset RTC clock, the 100% percentile
for Uptime on [1] is calculated since the Unix epoch which is useless as
far as usage statistics are concerned. Thus, if the Syncthing start time
is set to a past date, assume that the clock is wrong and do not even
try to report the uptime.

[1] https://data.syncthing.net

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: Jakob Borg <jakob@kastelo.net>
2023-07-22 21:25:03 +00:00
Christian Kujau
6b6b2c6194
lib/model: use WARN for "Unexpected folder" messages (#8998) 2023-07-22 21:17:32 +00:00
tomasz1986
d70eb569f2
lib/osutil: Skip setLowPriority in Windows if already lower (fixes #6597) (#8993) 2023-07-21 04:38:15 +00:00