Commit Graph

1419 Commits

Author SHA1 Message Date
Jakob Borg
1e212a8dc6 cmd/syncthing: Wording on error return in "cli debug file" 2021-06-05 11:45:29 +02:00
Jakob Borg
0e9d2a13af cmd/syncthing: Improve "cli debug file" handling
Proper URL encoding, and return a sensible error when it's not found.
2021-06-05 11:19:02 +02:00
Simon Frei
5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
Audrius Butkevicius
58e81fdffb
cmd/syncthing/cli: Update recli, fix stdin handling (fixes #7685, fixes #7673) (#7694) 2021-05-18 20:09:48 +01:00
tomasz1986
0619a27872
cmd/strelaypoolsrv: Fix minor grammar, use https in links (#7695)
* cmd/strelaypoolsrv: Fix minor grammar, use https in links

Add a few minor grammatical/stylistic fixes. Use `https` instead of
`http` in the MaxMind link in the footer.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>

* wip

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-05-18 19:07:09 +01:00
Audrius Butkevicius
96dae7bfec
cmd/uraggregate: Optimise queries (#7679)
* cmd/uraggregate: Optimise queries

* Update main.go
2021-05-16 12:34:46 +01:00
Audrius Butkevicius
4ace451013
Update main.go (#7667) 2021-05-12 08:01:18 +01:00
Audrius Butkevicius
a8e2fc6f61
cmd/uraggregate: Handle malformed data, dont abort on error (fixes #7639) (#7659) 2021-05-11 08:02:19 +02:00
Simon Frei
713527facf
all: Refactor relay invitations (#7646) 2021-05-10 22:25:43 +02:00
Audrius Butkevicius
87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei
ef4b8a2cf8
cmd/syncthing: Don't fail early on api setup error (fixes 7558) (#7591)
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)

* switch to factory pattern

* refactor config command to show help on nothing

* wip

* wip

* already abort in before
2021-04-25 20:48:17 +01:00
Simon Frei
54e27f551d
cmd/syncthing: Add debug commands to cli (#7503) 2021-04-22 11:14:25 +02:00
Jakob Borg
a514b65d81
cmd/syncthing: Rename --conf back to --config (fixes #7549) (#7550) 2021-04-06 10:40:52 +02:00
Audrius Butkevicius
f7929229c8
cmd/strelaysrv: Increase default strelaysrv network buffer size (fixes #7514) (#7517)
Increases throughput at the cost of more memory per connection
2021-03-22 21:47:51 +01:00
Simon Frei
bc08a951f1
lib/model: Encrypted fileinfo trailer needs to be in wire format (#7505) 2021-03-21 10:34:08 +01:00
Jakob Borg
8ef504f745
all: Simplify some method calls (#7499)
strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)

(Go 1.12) and who knows what was up with that dialQueue.Sort() thing.
2021-03-17 23:12:26 +01:00
Jakob Borg
960e850a78
cmd/uraggregate: Ignore errors harder (#7498)
We were already ignoring them, it just didn't look like it.
2021-03-17 22:55:43 +01:00
Jakob Borg
ea701a4e9e
cmd/syncthing: Error handling on upgrade (#7497)
dat innocuous little colon
2021-03-17 22:52:43 +01:00
Jakob Borg
f4372710bf
all: Remove crypto/md5 (#7493)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
2021-03-17 22:22:49 +01:00
deepsource-autofix[bot]
0930bccf88
cmd/ursrv, lib/scanner: Remove unnecessary slicing of slices (#7491)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 21:04:36 +01:00
Jakob Borg
e321bd3941
lib/*/auto: Add noassets files (#7490)
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
2021-03-17 21:03:35 +01:00
Simon Frei
39c2d1bc1a
cmd/syncthing: Check os.Args length (fixes #7486) 2021-03-17 15:51:14 +01:00
Simon Frei
cd21b8dfa5
cmd/syncthing/cli: Remove db reset and adjust others (#7484) 2021-03-17 09:06:37 +01:00
Simon Frei
40fbdc87ce
cmd/syncthing: Refactor cli subcommand to allow flags (#7454) 2021-03-17 09:04:50 +01:00
Jakob Borg
4d979a1ce9
all: Truncate some timestamps (fixes #7457) (#7459)
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.

I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...

I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
2021-03-12 10:35:10 +01:00
Jakob Borg
cdef503db6
all: Make config.Wrapper an actual suture.Service (fixes #7451) (#7452) 2021-03-11 14:51:00 +01:00
Simon Frei
9a001051d6
cmd/ursrv, lib/ur: Collect and present encryption usage (#7448) 2021-03-10 22:26:56 +00:00
Simon Frei
727df34aa1
cmd/syncthing: Correct name of HiddenConsole flag (fixes #7446) (#7447) 2021-03-10 07:31:31 +01:00
Simon Frei
587c89d979
cmd/syncthing, etc: Consistent failure restart backoff (#7439) 2021-03-07 15:25:55 +01:00
Simon Frei
4735575e8d
cmd/syncthing: Set flag defaults through kong vars (#7431) 2021-03-05 08:06:54 +01:00
Jakob Borg
55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
otbutz
c18dc6a629
cmd/strelaysrv: Load map tiles over HTTPS (#7382) 2021-02-19 22:07:05 +01:00
Simon Frei
0970aed596
cmd/stcrashreceiver: Add tag for report type (crash/failure) (#7374) 2021-02-18 13:16:32 +00:00
Audrius Butkevicius
fb078068b4
cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719) (#7364)
* cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719)

* Hijack help

* Add comment

* Revert go.mod/go.sum
2021-02-15 18:50:53 +01:00
Jakob Borg
55afa625fc
cmd/syncthing: Add decrypt subcommand (#7332)
This adds the `syncthing decrypt` subcommand that is used to
(offline-)decrypt or just verify the contents of an encrypted folder.
2021-02-12 08:38:43 +01:00
Simon Frei
6697b8fde3
cmd/syncthing: Fix hyphenated flag names (ref #7330) (#7352) 2021-02-11 11:37:48 +01:00
Jakob Borg
4f20c900d0
cmd/syncthing: Refactor command line parsing (#7330) 2021-02-10 20:35:37 +01:00
Simon Frei
27a34609a1
all: Failure reporting fixes (#7331) 2021-02-05 11:21:14 +01:00
Jakob Borg
ffcb57580f
cmd/syncthing: Provide early startup for config service (ref #7188) (#7285) 2021-01-16 12:58:02 +01:00
Simon Frei
f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
André Colomb
1555a4da7f
cmd/stevents: Add command line argument for event type filtering. (#7226)
Add a -types flag which can be set to a comma-separated list of event
types.  The contents will be included verbatim in the URL parameter
"events".
2020-12-22 22:10:26 +01:00
Simon Frei
a20a5f61f0
lib/ur: Send unreported failures on shutdown (#7164) 2020-12-22 20:17:14 +01:00
Simon Frei
78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Roberto Santalla
b5de49917c
cmd/relaypoolsrv: Allow validation of relay join requests by certificate (fixes #7196) (#7217) 2020-12-21 11:55:16 +01:00
André Colomb
7502997e7e
all: Store pending devices and folders in database (fixes #7178) (#6443) 2020-12-17 19:54:31 +01:00
Simon Frei
cccbb0bd5e
lib/ur: Reset timer when there's nothing to report (#7169) 2020-11-28 20:09:22 +01:00
Simon Frei
24af89c8e2
all: Refactor preparing configuration (#7127) 2020-11-20 14:21:54 +01:00
Simon Frei
9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei
31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Simon Frei
d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling (#7029) 2020-10-08 17:37:45 +02:00
Simon Frei
9d09fd6af3
all: Add failure reporting (fixes #7028) (#6925) 2020-10-07 10:05:13 +02:00
Jakob Borg
67761d8795 cmd/ursrv: Properly sort versions (fixes #6991) 2020-09-18 08:11:48 +02:00
Jakob Borg
9f8b01b1b9 cmd/ursrv: Recognize new-style dev versions 2020-09-18 08:11:44 +02:00
Simon Frei
0e3e0a7c9e
cmd/stindex, lib/db: Use same condition to check for needed files (#6954) 2020-09-04 13:59:04 +02:00
Jakob Borg
71bfad0bc6 cmd/stindex: Add dumping folder meta 2020-09-04 11:39:21 +02:00
Jakob Borg
b7986801cd cmd/stindex: Print more hashes and versions 2020-09-04 09:10:33 +02:00
Jaroslav Lichtblau
e3e46ec1de
cmd/ursrv: BUILDHOST name changed for Arch (#6948)
Recognize both old and new.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-09-02 08:20:25 +02:00
greatroar
0941ce76b7
cmd/strelaypoolsrv: Fix relay shuffling (fixes #6936) (#6935)
When cap(permanentRelays) >= len(permanentRelays) + len(knownRelays),

	append(permanentRelays, knownRelays...)

returns a slice of the array underlying permanentRelays. The subsequent
rand.Shuffle then mixes the permanent and known relays. Sequential
requests may cause strelaypoolsrv to forget its permanent relays. Worse,
concurrent requests may cause shuffling of the same slice on multiple
processors concurrently.

Co-authored-by: greatroar <@>
2020-08-27 15:51:58 +02:00
Audrius Butkevicius
d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Jakob Borg
2286a6ebef cmd/stcrashreceiver: Don't crash on nil err 2020-08-18 12:54:20 +02:00
Audrius Butkevicius
bf9ff17267
all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
Simon Frei
932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Jakob Borg
78d294f78c cmd/ursrv: Skip the duplicate complaints with new index name 2020-06-23 14:18:52 +02:00
Audrius Butkevicius
689cf2a5ee
lib/ur: Normalise contract between syncthing and ursrv (#6770)
* Fix ui, hide report date

* Undo Goland madness

* UR now web scale

* Fix migration

* Fix marshaling, force tick on start

* Fix tests

* Darwin build

* Split "all" build target, add package name as a tag

* Remove pq and sql dep from syncthing, split build targets

* Empty line

* Revert "Empty line"

This reverts commit f74af2b067dadda8a343714123512bd545a643c3.

* Revert "Remove pq and sql dep from syncthing, split build targets"

This reverts commit 8fc295ad007c5bb7886c557f492dacf51be307ad.

* Revert "Split "all" build target, add package name as a tag"

This reverts commit f4dc88995106d2b06042f30bea781a0feb08e55f.

* Normalise contract types

* Fix build add more logging
2020-06-23 09:47:15 +01:00
Audrius Butkevicius
f310bbaaac
cmd/stdiscosrv: Don't abuse wrong header (#6749) 2020-06-16 10:58:25 +02:00
greatroar
df83b84aa1
all: Make all error implementations pointer types (#6726)
This matches the convention of the stdlib and avoids ambiguity: when
customErr{} and &customErr{} both implement error, client code needs to
check for both.

Memory use should remain the same, since storing a non-pointer type in
an interface value still copies the value to the heap.
2020-06-16 09:27:34 +02:00
Audrius Butkevicius
f619a7f4cc
lib/connections: Try TCP punchthrough (fixes #4259) (#5753) 2020-06-16 09:17:07 +02:00
Jakob Borg
ec718e729e various: Master is now main
Also cleans out various old strings from the translation strings.
2020-06-16 07:08:09 +02:00
Audrius Butkevicius
36e6f8b082
cmd/ursrv: Attempt to fix js failure to load (#6747)
Following what was said in:

https://support.google.com/maps/thread/44057943?hl=en
2020-06-15 22:08:56 +01:00
Gleb Sinyavskiy
091cb5e611
cmd/strelaysrv: Fix JSON unmarshalling when joining a relay pool (fixes #6733) (#6729) 2020-06-10 11:26:09 +02:00
André Colomb
46536509d7
lib/protocol: Avoid panic in DeviceIDFromBytes (#6714) 2020-06-07 10:31:12 +02:00
Jakob Borg
1950efb790 cmd/strelaysrv: Add note about relay operators 2020-06-06 08:35:01 +02:00
Jakob Borg
1b77ab2b52
cmd/stcrashreceiver: Pick up extra build tags and send to Sentry (#6710)
This extracts the extra tags from any `[foo]` stuff at the end of the
version and sends them to Sentry for indexing.

If I need to modify that regexp again I'll probably write a from scratch
tokenizer and parser for our version string instead...
2020-06-03 15:00:46 +02:00
Jakob Borg
bfc9478965
cmd/syncthing: Correct auto upgrade criteria (fixes #6701) (#6702) 2020-06-02 11:38:39 +02:00
Simon Frei
1f8e6c55f6
lib/db: Refactor to use global list by version (fixes #6372) (#6638)
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-05-30 09:50:23 +02:00
Jakob Borg
1eea076f5c cmd/stindex: Detect and open Badger databases 2020-05-30 09:47:11 +02:00
greatroar
baa38eea7a
lib/assets: Allow assets to remain uncompressed (#6661) 2020-05-25 08:51:27 +02:00
André Colomb
668979605b
cmd/stindex: Add missing KeyType values in stindex dump code (#6659)
* cmd/stindex: Unify access to key from cached variable.

Avoid calling the Key() method from the iterator each time the value
is needed.  Just reuse the cache variable already assigned before the
switch block.

* cmd/stindex: Display the prefix byte value for unknown key types.

Make it easier to diagnose corrupt / unknown key type entries by
showing their decimal value, correlating with the definitions in
keyer.go.

* cmd/stindex: Add missing KeyType values in stindex dump code.

Recently added DB key prefixes KeyTypeBlockListMap and KeyTypeVersion
were unknown to the stindex dumping tool.  Add basic parsing to dump
their key structure.
2020-05-17 11:06:14 +02:00
Jakob Borg
5c54d879a1
cmd/syncthing: Don't crash when failing to create default config (fixes #6655) (#6658)
This is not an ignorable error, because it can happen if we fail to
allocate a free port for the GUI or sync port on first startup.
2020-05-17 07:56:24 +02:00
Jakob Borg
531ceb2b0f
Add indirection for large version vectors. (#6376)
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.

To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
2020-05-13 14:28:42 +02:00
greatroar
06365e5635
cmd/strelaypoolsrv, lib/api: Factor out static asset serving (#6624) 2020-05-10 11:44:34 +02:00
greatroar
e2febf246e
all: Store assets as strings (#6611)
Storing assets as []byte requires every compiled-in asset to be copied
into writable memory at program startup. That currently takes up 1.6MB
per syncthing process. Strings stay in the RODATA section and should be
shared between processes running the same binary.
2020-05-07 11:47:23 +02:00
Jakob Borg
7fa699e159
build, lib/build: Build faster (#6538)
This changes the build script to build all the things in one go
invocation, instead of one invocation per cmd. This is a lot faster
because it means more things get compiled concurrently. It's especially
a lot faster when things *don't* need to be rebuilt, possibly because it
only needs to build the dependency map and such once instead of once per
binary.

In order for this to work we need to be able to pass the same ldflags to
all the binaries. This means we can't set the program name with an
ldflag.

When it needs to rebuild everything (go clean -cache):

    ( ./old-build -gocmd go1.14.2 build all 2> /dev/null; )  65.82s user 11.28s system 574% cpu 13.409 total
    ( ./new-build -gocmd go1.14.2 build all 2> /dev/null; )  63.26s user 7.12s system 1220% cpu 5.766 total

On a subsequent run (nothing to build, just link the binaries):

    ( ./old-build -gocmd go1.14.2 build all 2> /dev/null; )  26.58s user 7.53s system 582% cpu 5.853 total
    ( ./new-build -gocmd go1.14.2 build all 2> /dev/null; )  18.66s user 2.45s system 1090% cpu 1.935 total
2020-04-16 10:09:33 +02:00
Jakob Borg
5373e38ac8 cmd/ursrv: Filter out ancient versions from chart 2020-04-16 09:13:01 +02:00
Simon Frei
ab92f8520c
cmd/syncthing, lib/db: Store upgrade info to throttle queries (fixes #6513) (#6514) 2020-04-13 10:21:07 +02:00
Jakob Borg
4b17c511f9
cmd/stcrashreceiver: Enable (rough) affected users count (#6511)
Seeing thousands of reports is no use when we don't know if they
represent one poor user or thousands.
2020-04-07 13:19:49 +02:00
greatroar
674a99e9ae
cmd/strelaypoolsrv: Simplify LRU usage (#6507) 2020-04-06 12:43:56 +02:00
greatroar
b7ba401c0b
cmd/strelaypoolsrv: Fix race condition in caching (#6496)
Successful LRU cache lookups modify the cache's recency list, so
RWMutex.RLock isn't enough protection.

Secondarily, multiple concurrent lookups with the same key should not
create separate rate limiters, so release the lock only when presence
of the key in the cache has been ascertained.

Co-authored-by: greatroar <@>
2020-04-04 20:20:25 +01:00
Kevin Bushiri
e1324a0e23
cmd/strelaypoolsrv: Use OpenStreetMap (fixes #6150) (#6459) 2020-04-04 13:48:20 +02:00
Jakob Borg
b7b9476e5a cmd/strelaysrv: Harmonize and improve log output (ref #6492) 2020-04-04 13:31:42 +02:00
Jakob Borg
d1db7e3dd2 cmd/strelaypoolsrv: Configurable request processors & queue len 2020-04-04 13:31:42 +02:00
Jakob Borg
362da59396 cmd/strelaypoolsrv: Expose check error to client, fix incorrect response code handling 2020-04-04 13:31:42 +02:00
Jakob Borg
66262392c3 cmd/strelaypoolsrv: Correctly account status codes, tweak status codes 2020-04-04 13:31:42 +02:00
Jakob Borg
2c2e6cd0d5 cmd/ursrv: Minor heatmap tweaks 2020-03-26 15:19:05 +01:00
Kevin Bushiri
963e9a4071
cmd/ursrv: Use OpenStreetMap and Leaflet for heat map (ref #6150) (#6454) 2020-03-26 12:32:14 +01:00
Jakob Borg
b28899ac07 cmd/ursrv: Provide cached locations.json 2020-03-25 14:19:35 +01:00
Simon Frei
7f23de4f03
all: Pass db intervals as args not env vars (#6448) 2020-03-24 13:53:20 +01:00
Jakob Borg
1ea98a16b1
cmd/syncthing: Don't open browser on upgrade restarts (fixes #6437) (#6442)
We set the STRESTART environment when starting the inner process after
the first time, but this didn't persist when restarting the monitor
process. Now it does.
2020-03-22 11:39:09 +01:00
Jakob Borg
e2f3500df9
cmd/syncthing: Properly handle STNORESTART=1 (fixes #6440) (#6441)
Makes it truly equivalent to -no-restart, and also updates the option
descriptions to be more truthful.
2020-03-22 11:38:53 +01:00
Kevin Bushiri
5975772ed8
cmd/stdiscosrv: Only generate keypair if it doesn't exist (fixes #5809) (#6419) 2020-03-19 14:50:24 +01:00