Commit Graph

1420 Commits

Author SHA1 Message Date
Simon Frei 1ae5ac7d0b
cmd/stcrashreceiver: Sanitize failure report fingerprints (#7840) 2021-07-22 11:16:24 +02:00
Jakob Borg 1e212a8dc6 cmd/syncthing: Wording on error return in "cli debug file" 2021-06-05 11:45:29 +02:00
Jakob Borg 0e9d2a13af cmd/syncthing: Improve "cli debug file" handling
Proper URL encoding, and return a sensible error when it's not found.
2021-06-05 11:19:02 +02:00
Simon Frei 5bdb6798a9
all: Regenerate proto (#7696) 2021-05-19 13:30:20 +02:00
Audrius Butkevicius 58e81fdffb
cmd/syncthing/cli: Update recli, fix stdin handling (fixes #7685, fixes #7673) (#7694) 2021-05-18 20:09:48 +01:00
tomasz1986 0619a27872
cmd/strelaypoolsrv: Fix minor grammar, use https in links (#7695)
* cmd/strelaypoolsrv: Fix minor grammar, use https in links

Add a few minor grammatical/stylistic fixes. Use `https` instead of
`http` in the MaxMind link in the footer.

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>

* wip

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2021-05-18 19:07:09 +01:00
Audrius Butkevicius 96dae7bfec
cmd/uraggregate: Optimise queries (#7679)
* cmd/uraggregate: Optimise queries

* Update main.go
2021-05-16 12:34:46 +01:00
Audrius Butkevicius 4ace451013
Update main.go (#7667) 2021-05-12 08:01:18 +01:00
Audrius Butkevicius a8e2fc6f61
cmd/uraggregate: Handle malformed data, dont abort on error (fixes #7639) (#7659) 2021-05-11 08:02:19 +02:00
Simon Frei 713527facf
all: Refactor relay invitations (#7646) 2021-05-10 22:25:43 +02:00
Audrius Butkevicius 87a0eecc31
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file

* Fix wrong error returned by CLI

* Gofmt

* Better names

* Review comments

* Review comments
2021-05-03 11:28:25 +01:00
Simon Frei ef4b8a2cf8
cmd/syncthing: Don't fail early on api setup error (fixes 7558) (#7591)
* cmd/syncthing: Don't fail early on api setup error (fixes 7558)

* switch to factory pattern

* refactor config command to show help on nothing

* wip

* wip

* already abort in before
2021-04-25 20:48:17 +01:00
Simon Frei 54e27f551d
cmd/syncthing: Add debug commands to cli (#7503) 2021-04-22 11:14:25 +02:00
Jakob Borg a514b65d81
cmd/syncthing: Rename --conf back to --config (fixes #7549) (#7550) 2021-04-06 10:40:52 +02:00
Audrius Butkevicius f7929229c8
cmd/strelaysrv: Increase default strelaysrv network buffer size (fixes #7514) (#7517)
Increases throughput at the cost of more memory per connection
2021-03-22 21:47:51 +01:00
Simon Frei bc08a951f1
lib/model: Encrypted fileinfo trailer needs to be in wire format (#7505) 2021-03-21 10:34:08 +01:00
Jakob Borg 8ef504f745
all: Simplify some method calls (#7499)
strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)

(Go 1.12) and who knows what was up with that dialQueue.Sort() thing.
2021-03-17 23:12:26 +01:00
Jakob Borg 960e850a78
cmd/uraggregate: Ignore errors harder (#7498)
We were already ignoring them, it just didn't look like it.
2021-03-17 22:55:43 +01:00
Jakob Borg ea701a4e9e
cmd/syncthing: Error handling on upgrade (#7497)
dat innocuous little colon
2021-03-17 22:52:43 +01:00
Jakob Borg f4372710bf
all: Remove crypto/md5 (#7493)
This is a mostly pointless change to make security scanners and static
analysis tools happy, as they all hate seeing md5. None of our md5 uses
were security relevant, but still. Only visible effect of this change is
that our temp file names for very long file names become slightly longer
than they were previously...
2021-03-17 22:22:49 +01:00
deepsource-autofix[bot] 0930bccf88
cmd/ursrv, lib/scanner: Remove unnecessary slicing of slices (#7491)
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
2021-03-17 21:04:36 +01:00
Jakob Borg e321bd3941
lib/*/auto: Add noassets files (#7490)
This adds a couple of dummy asset files protected by the "noassets"
build tag. The purpose is that it should be possible for, for example,
CI tools and static analysis things to compile and analyze the source
tree without our custom asset generation step. Also makes `go test -tags
noassets ./...` work without building assets first.
2021-03-17 21:03:35 +01:00
Simon Frei 39c2d1bc1a
cmd/syncthing: Check os.Args length (fixes #7486) 2021-03-17 15:51:14 +01:00
Simon Frei cd21b8dfa5
cmd/syncthing/cli: Remove db reset and adjust others (#7484) 2021-03-17 09:06:37 +01:00
Simon Frei 40fbdc87ce
cmd/syncthing: Refactor cli subcommand to allow flags (#7454) 2021-03-17 09:04:50 +01:00
Jakob Borg 4d979a1ce9
all: Truncate some timestamps (fixes #7457) (#7459)
This truncates times meant for API consumption to second precision,
where fractions won't typically matter or add any value. Exception to
this is timestamps on logs and events, and of course I'm not touching
things like file metadata.

I'm not 100% certain this is an exhaustive change, but it's the things I
found by grepping and following the breadcrumbs from lib/api...

I also considered general-but-ugly solutions, like having the API
serializer itself do reflection magic or even regexps on returned
objects, but decided against it because aurgh...
2021-03-12 10:35:10 +01:00
Jakob Borg cdef503db6
all: Make config.Wrapper an actual suture.Service (fixes #7451) (#7452) 2021-03-11 14:51:00 +01:00
Simon Frei 9a001051d6
cmd/ursrv, lib/ur: Collect and present encryption usage (#7448) 2021-03-10 22:26:56 +00:00
Simon Frei 727df34aa1
cmd/syncthing: Correct name of HiddenConsole flag (fixes #7446) (#7447) 2021-03-10 07:31:31 +01:00
Simon Frei 587c89d979
cmd/syncthing, etc: Consistent failure restart backoff (#7439) 2021-03-07 15:25:55 +01:00
Simon Frei 4735575e8d
cmd/syncthing: Set flag defaults through kong vars (#7431) 2021-03-05 08:06:54 +01:00
Jakob Borg 55d5e03639
lib/db: Remove Badger experiment (#7413) 2021-03-01 09:16:08 +01:00
otbutz c18dc6a629
cmd/strelaysrv: Load map tiles over HTTPS (#7382) 2021-02-19 22:07:05 +01:00
Simon Frei 0970aed596
cmd/stcrashreceiver: Add tag for report type (crash/failure) (#7374) 2021-02-18 13:16:32 +00:00
Audrius Butkevicius fb078068b4
cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719) (#7364)
* cmd/syncthing: Add cli as a subcommand (fixes #6566, fixes #4719)

* Hijack help

* Add comment

* Revert go.mod/go.sum
2021-02-15 18:50:53 +01:00
Jakob Borg 55afa625fc
cmd/syncthing: Add decrypt subcommand (#7332)
This adds the `syncthing decrypt` subcommand that is used to
(offline-)decrypt or just verify the contents of an encrypted folder.
2021-02-12 08:38:43 +01:00
Simon Frei 6697b8fde3
cmd/syncthing: Fix hyphenated flag names (ref #7330) (#7352) 2021-02-11 11:37:48 +01:00
Jakob Borg 4f20c900d0
cmd/syncthing: Refactor command line parsing (#7330) 2021-02-10 20:35:37 +01:00
Simon Frei 27a34609a1
all: Failure reporting fixes (#7331) 2021-02-05 11:21:14 +01:00
Jakob Borg ffcb57580f
cmd/syncthing: Provide early startup for config service (ref #7188) (#7285) 2021-01-16 12:58:02 +01:00
Simon Frei f63cdbfcfa
lib: Apply config changes sequentially (ref #5298) (#7188) 2021-01-15 15:43:34 +01:00
André Colomb 1555a4da7f
cmd/stevents: Add command line argument for event type filtering. (#7226)
Add a -types flag which can be set to a comma-separated list of event
types.  The contents will be included verbatim in the URL parameter
"events".
2020-12-22 22:10:26 +01:00
Simon Frei a20a5f61f0
lib/ur: Send unreported failures on shutdown (#7164) 2020-12-22 20:17:14 +01:00
Simon Frei 78bd0341a8
all: Handle errors opening db/creating file-set (ref #5907) (#7150) 2020-12-21 12:59:22 +01:00
Roberto Santalla b5de49917c
cmd/relaypoolsrv: Allow validation of relay join requests by certificate (fixes #7196) (#7217) 2020-12-21 11:55:16 +01:00
André Colomb 7502997e7e
all: Store pending devices and folders in database (fixes #7178) (#6443) 2020-12-17 19:54:31 +01:00
Simon Frei cccbb0bd5e
lib/ur: Reset timer when there's nothing to report (#7169) 2020-11-28 20:09:22 +01:00
Simon Frei 24af89c8e2
all: Refactor preparing configuration (#7127) 2020-11-20 14:21:54 +01:00
Simon Frei 9524b51708
all: Implement suture v4-api (#6947) 2020-11-17 13:19:04 +01:00
Simon Frei 31559e908b
all: Add untrusted folders behind feature flag (ref #62) (#7055) 2020-11-09 15:33:32 +01:00
Simon Frei d828adb648
cmd/stcrashreceiver, lib/db: Improve panic message handling (#7029) 2020-10-08 17:37:45 +02:00
Simon Frei 9d09fd6af3
all: Add failure reporting (fixes #7028) (#6925) 2020-10-07 10:05:13 +02:00
Jakob Borg 67761d8795 cmd/ursrv: Properly sort versions (fixes #6991) 2020-09-18 08:11:48 +02:00
Jakob Borg 9f8b01b1b9 cmd/ursrv: Recognize new-style dev versions 2020-09-18 08:11:44 +02:00
Simon Frei 0e3e0a7c9e
cmd/stindex, lib/db: Use same condition to check for needed files (#6954) 2020-09-04 13:59:04 +02:00
Jakob Borg 71bfad0bc6 cmd/stindex: Add dumping folder meta 2020-09-04 11:39:21 +02:00
Jakob Borg b7986801cd cmd/stindex: Print more hashes and versions 2020-09-04 09:10:33 +02:00
Jaroslav Lichtblau e3e46ec1de
cmd/ursrv: BUILDHOST name changed for Arch (#6948)
Recognize both old and new.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-09-02 08:20:25 +02:00
greatroar 0941ce76b7
cmd/strelaypoolsrv: Fix relay shuffling (fixes #6936) (#6935)
When cap(permanentRelays) >= len(permanentRelays) + len(knownRelays),

	append(permanentRelays, knownRelays...)

returns a slice of the array underlying permanentRelays. The subsequent
rand.Shuffle then mixes the permanent and known relays. Sequential
requests may cause strelaypoolsrv to forget its permanent relays. Worse,
concurrent requests may cause shuffling of the same slice on multiple
processors concurrently.

Co-authored-by: greatroar <@>
2020-08-27 15:51:58 +02:00
Audrius Butkevicius d507d932b8
all: Use protobuf to generate config structs (fixes #6734) (#6900) 2020-08-25 08:11:14 +02:00
Jakob Borg 2286a6ebef cmd/stcrashreceiver: Don't crash on nil err 2020-08-18 12:54:20 +02:00
Audrius Butkevicius bf9ff17267
all: Remove need to restart syncthing (#6883) 2020-08-18 09:26:33 +02:00
Simon Frei 932d8c69de lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of
insensitive filesystems. This means we correctly pick up case-only renames
and throw a case conflict error when there would be multiple files differing
only in case.

This safety check has a small performance hit (about 20% more filesystem
operations when scanning for changes). The new advanced folder option
`caseSensitiveFS` can be used to disable the safety checks, retaining the
previous behavior on systems known to be fully case sensitive.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-07-28 11:15:11 +02:00
Jakob Borg 78d294f78c cmd/ursrv: Skip the duplicate complaints with new index name 2020-06-23 14:18:52 +02:00
Audrius Butkevicius 689cf2a5ee
lib/ur: Normalise contract between syncthing and ursrv (#6770)
* Fix ui, hide report date

* Undo Goland madness

* UR now web scale

* Fix migration

* Fix marshaling, force tick on start

* Fix tests

* Darwin build

* Split "all" build target, add package name as a tag

* Remove pq and sql dep from syncthing, split build targets

* Empty line

* Revert "Empty line"

This reverts commit f74af2b067dadda8a343714123512bd545a643c3.

* Revert "Remove pq and sql dep from syncthing, split build targets"

This reverts commit 8fc295ad007c5bb7886c557f492dacf51be307ad.

* Revert "Split "all" build target, add package name as a tag"

This reverts commit f4dc88995106d2b06042f30bea781a0feb08e55f.

* Normalise contract types

* Fix build add more logging
2020-06-23 09:47:15 +01:00
Audrius Butkevicius f310bbaaac
cmd/stdiscosrv: Don't abuse wrong header (#6749) 2020-06-16 10:58:25 +02:00
greatroar df83b84aa1
all: Make all error implementations pointer types (#6726)
This matches the convention of the stdlib and avoids ambiguity: when
customErr{} and &customErr{} both implement error, client code needs to
check for both.

Memory use should remain the same, since storing a non-pointer type in
an interface value still copies the value to the heap.
2020-06-16 09:27:34 +02:00
Audrius Butkevicius f619a7f4cc
lib/connections: Try TCP punchthrough (fixes #4259) (#5753) 2020-06-16 09:17:07 +02:00
Jakob Borg ec718e729e various: Master is now main
Also cleans out various old strings from the translation strings.
2020-06-16 07:08:09 +02:00
Audrius Butkevicius 36e6f8b082
cmd/ursrv: Attempt to fix js failure to load (#6747)
Following what was said in:

https://support.google.com/maps/thread/44057943?hl=en
2020-06-15 22:08:56 +01:00
Gleb Sinyavskiy 091cb5e611
cmd/strelaysrv: Fix JSON unmarshalling when joining a relay pool (fixes #6733) (#6729) 2020-06-10 11:26:09 +02:00
André Colomb 46536509d7
lib/protocol: Avoid panic in DeviceIDFromBytes (#6714) 2020-06-07 10:31:12 +02:00
Jakob Borg 1950efb790 cmd/strelaysrv: Add note about relay operators 2020-06-06 08:35:01 +02:00
Jakob Borg 1b77ab2b52
cmd/stcrashreceiver: Pick up extra build tags and send to Sentry (#6710)
This extracts the extra tags from any `[foo]` stuff at the end of the
version and sends them to Sentry for indexing.

If I need to modify that regexp again I'll probably write a from scratch
tokenizer and parser for our version string instead...
2020-06-03 15:00:46 +02:00
Jakob Borg bfc9478965
cmd/syncthing: Correct auto upgrade criteria (fixes #6701) (#6702) 2020-06-02 11:38:39 +02:00
Simon Frei 1f8e6c55f6
lib/db: Refactor to use global list by version (fixes #6372) (#6638)
Group the global list of files by version, instead of having one flat list for all devices. This removes lots of duplicate protocol.Vectors.

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2020-05-30 09:50:23 +02:00
Jakob Borg 1eea076f5c cmd/stindex: Detect and open Badger databases 2020-05-30 09:47:11 +02:00
greatroar baa38eea7a
lib/assets: Allow assets to remain uncompressed (#6661) 2020-05-25 08:51:27 +02:00
André Colomb 668979605b
cmd/stindex: Add missing KeyType values in stindex dump code (#6659)
* cmd/stindex: Unify access to key from cached variable.

Avoid calling the Key() method from the iterator each time the value
is needed.  Just reuse the cache variable already assigned before the
switch block.

* cmd/stindex: Display the prefix byte value for unknown key types.

Make it easier to diagnose corrupt / unknown key type entries by
showing their decimal value, correlating with the definitions in
keyer.go.

* cmd/stindex: Add missing KeyType values in stindex dump code.

Recently added DB key prefixes KeyTypeBlockListMap and KeyTypeVersion
were unknown to the stindex dumping tool.  Add basic parsing to dump
their key structure.
2020-05-17 11:06:14 +02:00
Jakob Borg 5c54d879a1
cmd/syncthing: Don't crash when failing to create default config (fixes #6655) (#6658)
This is not an ignorable error, because it can happen if we fail to
allocate a free port for the GUI or sync port on first startup.
2020-05-17 07:56:24 +02:00
Jakob Borg 531ceb2b0f
Add indirection for large version vectors. (#6376)
This adds indirection of large version vectors in the same manner as we
already to block lists. The effect is the same: less duplicated data in
some situations.

To mitigate the impact for when this indirection
wouldn't be needed I've added an indirection cutoff for both blocks and
the new version vector stuff: we don't do the indirection at all for
small block lists or small version vectors, instead storing it directly
like we used to do. This is faster for small files and small setups.
2020-05-13 14:28:42 +02:00
greatroar 06365e5635
cmd/strelaypoolsrv, lib/api: Factor out static asset serving (#6624) 2020-05-10 11:44:34 +02:00
greatroar e2febf246e
all: Store assets as strings (#6611)
Storing assets as []byte requires every compiled-in asset to be copied
into writable memory at program startup. That currently takes up 1.6MB
per syncthing process. Strings stay in the RODATA section and should be
shared between processes running the same binary.
2020-05-07 11:47:23 +02:00
Jakob Borg 7fa699e159
build, lib/build: Build faster (#6538)
This changes the build script to build all the things in one go
invocation, instead of one invocation per cmd. This is a lot faster
because it means more things get compiled concurrently. It's especially
a lot faster when things *don't* need to be rebuilt, possibly because it
only needs to build the dependency map and such once instead of once per
binary.

In order for this to work we need to be able to pass the same ldflags to
all the binaries. This means we can't set the program name with an
ldflag.

When it needs to rebuild everything (go clean -cache):

    ( ./old-build -gocmd go1.14.2 build all 2> /dev/null; )  65.82s user 11.28s system 574% cpu 13.409 total
    ( ./new-build -gocmd go1.14.2 build all 2> /dev/null; )  63.26s user 7.12s system 1220% cpu 5.766 total

On a subsequent run (nothing to build, just link the binaries):

    ( ./old-build -gocmd go1.14.2 build all 2> /dev/null; )  26.58s user 7.53s system 582% cpu 5.853 total
    ( ./new-build -gocmd go1.14.2 build all 2> /dev/null; )  18.66s user 2.45s system 1090% cpu 1.935 total
2020-04-16 10:09:33 +02:00
Jakob Borg 5373e38ac8 cmd/ursrv: Filter out ancient versions from chart 2020-04-16 09:13:01 +02:00
Simon Frei ab92f8520c
cmd/syncthing, lib/db: Store upgrade info to throttle queries (fixes #6513) (#6514) 2020-04-13 10:21:07 +02:00
Jakob Borg 4b17c511f9
cmd/stcrashreceiver: Enable (rough) affected users count (#6511)
Seeing thousands of reports is no use when we don't know if they
represent one poor user or thousands.
2020-04-07 13:19:49 +02:00
greatroar 674a99e9ae
cmd/strelaypoolsrv: Simplify LRU usage (#6507) 2020-04-06 12:43:56 +02:00
greatroar b7ba401c0b
cmd/strelaypoolsrv: Fix race condition in caching (#6496)
Successful LRU cache lookups modify the cache's recency list, so
RWMutex.RLock isn't enough protection.

Secondarily, multiple concurrent lookups with the same key should not
create separate rate limiters, so release the lock only when presence
of the key in the cache has been ascertained.

Co-authored-by: greatroar <@>
2020-04-04 20:20:25 +01:00
Kevin Bushiri e1324a0e23
cmd/strelaypoolsrv: Use OpenStreetMap (fixes #6150) (#6459) 2020-04-04 13:48:20 +02:00
Jakob Borg b7b9476e5a cmd/strelaysrv: Harmonize and improve log output (ref #6492) 2020-04-04 13:31:42 +02:00
Jakob Borg d1db7e3dd2 cmd/strelaypoolsrv: Configurable request processors & queue len 2020-04-04 13:31:42 +02:00
Jakob Borg 362da59396 cmd/strelaypoolsrv: Expose check error to client, fix incorrect response code handling 2020-04-04 13:31:42 +02:00
Jakob Borg 66262392c3 cmd/strelaypoolsrv: Correctly account status codes, tweak status codes 2020-04-04 13:31:42 +02:00
Jakob Borg 2c2e6cd0d5 cmd/ursrv: Minor heatmap tweaks 2020-03-26 15:19:05 +01:00
Kevin Bushiri 963e9a4071
cmd/ursrv: Use OpenStreetMap and Leaflet for heat map (ref #6150) (#6454) 2020-03-26 12:32:14 +01:00
Jakob Borg b28899ac07 cmd/ursrv: Provide cached locations.json 2020-03-25 14:19:35 +01:00
Simon Frei 7f23de4f03
all: Pass db intervals as args not env vars (#6448) 2020-03-24 13:53:20 +01:00
Jakob Borg 1ea98a16b1
cmd/syncthing: Don't open browser on upgrade restarts (fixes #6437) (#6442)
We set the STRESTART environment when starting the inner process after
the first time, but this didn't persist when restarting the monitor
process. Now it does.
2020-03-22 11:39:09 +01:00
Jakob Borg e2f3500df9
cmd/syncthing: Properly handle STNORESTART=1 (fixes #6440) (#6441)
Makes it truly equivalent to -no-restart, and also updates the option
descriptions to be more truthful.
2020-03-22 11:38:53 +01:00