This is a new revision of the discovery server. Relevant changes and non-changes: - Protocol towards clients is unchanged. - Recommended large scale design is still to be deployed nehind nginx (I tested, and it's still a lot faster at terminating TLS). - Database backend is leveldb again, only. It scales enough, is easy to setup, and we don't need any backend to take care of. - Server supports replication. This is a simple TCP channel - protect it with a firewall when deploying over the internet. (We deploy this within the same datacenter, and with firewall.) Any incoming client announces are sent over the replication channel(s) to other peer discosrvs. Incoming replication changes are applied to the database as if they came from clients, but without the TLS/certificate overhead. - Metrics are exposed using the prometheus library, when enabled. - The database values and replication protocol is protobuf, because JSON was quite CPU intensive when I tried that and benchmarked it. - The "Retry-After" value for failed lookups gets slowly increased from a default of 120 seconds, by 5 seconds for each failed lookup, independently by each discosrv. This lowers the query load over time for clients that are never seen. The Retry-After maxes out at 3600 after a couple of weeks of this increase. The number of failed lookups is stored in the database, now and then (avoiding making each lookup a database put). All in all this means clients can be pointed towards a cluster using just multiple A / AAAA records to gain both load sharing and redundancy (if one is down, clients will talk to the remaining ones). GitHub-Pull-Request: https://github.com/syncthing/syncthing/pull/4648
259 lines
10 KiB
Plaintext
259 lines
10 KiB
Plaintext
GoGoProtobuf http://github.com/gogo/protobuf extends
|
|
GoProtobuf http://github.com/golang/protobuf
|
|
|
|
# Go support for Protocol Buffers
|
|
|
|
Google's data interchange format.
|
|
Copyright 2010 The Go Authors.
|
|
https://github.com/golang/protobuf
|
|
|
|
This package and the code it generates requires at least Go 1.4.
|
|
|
|
This software implements Go bindings for protocol buffers. For
|
|
information about protocol buffers themselves, see
|
|
https://developers.google.com/protocol-buffers/
|
|
|
|
## Installation ##
|
|
|
|
To use this software, you must:
|
|
- Install the standard C++ implementation of protocol buffers from
|
|
https://developers.google.com/protocol-buffers/
|
|
- Of course, install the Go compiler and tools from
|
|
https://golang.org/
|
|
See
|
|
https://golang.org/doc/install
|
|
for details or, if you are using gccgo, follow the instructions at
|
|
https://golang.org/doc/install/gccgo
|
|
- Grab the code from the repository and install the proto package.
|
|
The simplest way is to run `go get -u github.com/golang/protobuf/protoc-gen-go`.
|
|
The compiler plugin, protoc-gen-go, will be installed in $GOBIN,
|
|
defaulting to $GOPATH/bin. It must be in your $PATH for the protocol
|
|
compiler, protoc, to find it.
|
|
|
|
This software has two parts: a 'protocol compiler plugin' that
|
|
generates Go source files that, once compiled, can access and manage
|
|
protocol buffers; and a library that implements run-time support for
|
|
encoding (marshaling), decoding (unmarshaling), and accessing protocol
|
|
buffers.
|
|
|
|
There is support for gRPC in Go using protocol buffers.
|
|
See the note at the bottom of this file for details.
|
|
|
|
There are no insertion points in the plugin.
|
|
|
|
GoGoProtobuf provides extensions for protocol buffers and GoProtobuf
|
|
see http://github.com/gogo/protobuf/gogoproto/doc.go
|
|
|
|
## Using protocol buffers with Go ##
|
|
|
|
Once the software is installed, there are two steps to using it.
|
|
First you must compile the protocol buffer definitions and then import
|
|
them, with the support library, into your program.
|
|
|
|
To compile the protocol buffer definition, run protoc with the --gogo_out
|
|
parameter set to the directory you want to output the Go code to.
|
|
|
|
protoc --gogo_out=. *.proto
|
|
|
|
The generated files will be suffixed .pb.go. See the Test code below
|
|
for an example using such a file.
|
|
|
|
The package comment for the proto library contains text describing
|
|
the interface provided in Go for protocol buffers. Here is an edited
|
|
version.
|
|
|
|
If you are using any gogo.proto extensions you will need to specify the
|
|
proto_path to include the descriptor.proto and gogo.proto.
|
|
gogo.proto is located in github.com/gogo/protobuf/gogoproto
|
|
This should be fine, since your import is the same.
|
|
descriptor.proto is located in either github.com/gogo/protobuf/protobuf
|
|
or code.google.com/p/protobuf/trunk/src/
|
|
Its import is google/protobuf/descriptor.proto so it might need some help.
|
|
|
|
protoc --gogo_out=. -I=.:github.com/gogo/protobuf/protobuf *.proto
|
|
|
|
==========
|
|
|
|
The proto package converts data structures to and from the
|
|
wire format of protocol buffers. It works in concert with the
|
|
Go source code generated for .proto files by the protocol compiler.
|
|
|
|
A summary of the properties of the protocol buffer interface
|
|
for a protocol buffer variable v:
|
|
|
|
- Names are turned from camel_case to CamelCase for export.
|
|
- There are no methods on v to set fields; just treat
|
|
them as structure fields.
|
|
- There are getters that return a field's value if set,
|
|
and return the field's default value if unset.
|
|
The getters work even if the receiver is a nil message.
|
|
- The zero value for a struct is its correct initialization state.
|
|
All desired fields must be set before marshaling.
|
|
- A Reset() method will restore a protobuf struct to its zero state.
|
|
- Non-repeated fields are pointers to the values; nil means unset.
|
|
That is, optional or required field int32 f becomes F *int32.
|
|
- Repeated fields are slices.
|
|
- Helper functions are available to aid the setting of fields.
|
|
Helpers for getting values are superseded by the
|
|
GetFoo methods and their use is deprecated.
|
|
msg.Foo = proto.String("hello") // set field
|
|
- Constants are defined to hold the default values of all fields that
|
|
have them. They have the form Default_StructName_FieldName.
|
|
Because the getter methods handle defaulted values,
|
|
direct use of these constants should be rare.
|
|
- Enums are given type names and maps from names to values.
|
|
Enum values are prefixed with the enum's type name. Enum types have
|
|
a String method, and a Enum method to assist in message construction.
|
|
- Nested groups and enums have type names prefixed with the name of
|
|
the surrounding message type.
|
|
- Extensions are given descriptor names that start with E_,
|
|
followed by an underscore-delimited list of the nested messages
|
|
that contain it (if any) followed by the CamelCased name of the
|
|
extension field itself. HasExtension, ClearExtension, GetExtension
|
|
and SetExtension are functions for manipulating extensions.
|
|
- Oneof field sets are given a single field in their message,
|
|
with distinguished wrapper types for each possible field value.
|
|
- Marshal and Unmarshal are functions to encode and decode the wire format.
|
|
|
|
When the .proto file specifies `syntax="proto3"`, there are some differences:
|
|
|
|
- Non-repeated fields of non-message type are values instead of pointers.
|
|
- Enum types do not get an Enum method.
|
|
|
|
Consider file test.proto, containing
|
|
|
|
```proto
|
|
syntax = "proto2";
|
|
package example;
|
|
|
|
enum FOO { X = 17; };
|
|
|
|
message Test {
|
|
required string label = 1;
|
|
optional int32 type = 2 [default=77];
|
|
repeated int64 reps = 3;
|
|
optional group OptionalGroup = 4 {
|
|
required string RequiredField = 5;
|
|
}
|
|
}
|
|
```
|
|
|
|
To create and play with a Test object from the example package,
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"log"
|
|
|
|
"github.com/gogo/protobuf/proto"
|
|
"path/to/example"
|
|
)
|
|
|
|
func main() {
|
|
test := &example.Test {
|
|
Label: proto.String("hello"),
|
|
Type: proto.Int32(17),
|
|
Reps: []int64{1, 2, 3},
|
|
Optionalgroup: &example.Test_OptionalGroup {
|
|
RequiredField: proto.String("good bye"),
|
|
},
|
|
}
|
|
data, err := proto.Marshal(test)
|
|
if err != nil {
|
|
log.Fatal("marshaling error: ", err)
|
|
}
|
|
newTest := &example.Test{}
|
|
err = proto.Unmarshal(data, newTest)
|
|
if err != nil {
|
|
log.Fatal("unmarshaling error: ", err)
|
|
}
|
|
// Now test and newTest contain the same data.
|
|
if test.GetLabel() != newTest.GetLabel() {
|
|
log.Fatalf("data mismatch %q != %q", test.GetLabel(), newTest.GetLabel())
|
|
}
|
|
// etc.
|
|
}
|
|
```
|
|
|
|
|
|
## Parameters ##
|
|
|
|
To pass extra parameters to the plugin, use a comma-separated
|
|
parameter list separated from the output directory by a colon:
|
|
|
|
|
|
protoc --gogo_out=plugins=grpc,import_path=mypackage:. *.proto
|
|
|
|
|
|
- `import_prefix=xxx` - a prefix that is added onto the beginning of
|
|
all imports. Useful for things like generating protos in a
|
|
subdirectory, or regenerating vendored protobufs in-place.
|
|
- `import_path=foo/bar` - used as the package if no input files
|
|
declare `go_package`. If it contains slashes, everything up to the
|
|
rightmost slash is ignored.
|
|
- `plugins=plugin1+plugin2` - specifies the list of sub-plugins to
|
|
load. The only plugin in this repo is `grpc`.
|
|
- `Mfoo/bar.proto=quux/shme` - declares that foo/bar.proto is
|
|
associated with Go package quux/shme. This is subject to the
|
|
import_prefix parameter.
|
|
|
|
## gRPC Support ##
|
|
|
|
If a proto file specifies RPC services, protoc-gen-go can be instructed to
|
|
generate code compatible with gRPC (http://www.grpc.io/). To do this, pass
|
|
the `plugins` parameter to protoc-gen-go; the usual way is to insert it into
|
|
the --go_out argument to protoc:
|
|
|
|
protoc --gogo_out=plugins=grpc:. *.proto
|
|
|
|
## Compatibility ##
|
|
|
|
The library and the generated code are expected to be stable over time.
|
|
However, we reserve the right to make breaking changes without notice for the
|
|
following reasons:
|
|
|
|
- Security. A security issue in the specification or implementation may come to
|
|
light whose resolution requires breaking compatibility. We reserve the right
|
|
to address such security issues.
|
|
- Unspecified behavior. There are some aspects of the Protocol Buffers
|
|
specification that are undefined. Programs that depend on such unspecified
|
|
behavior may break in future releases.
|
|
- Specification errors or changes. If it becomes necessary to address an
|
|
inconsistency, incompleteness, or change in the Protocol Buffers
|
|
specification, resolving the issue could affect the meaning or legality of
|
|
existing programs. We reserve the right to address such issues, including
|
|
updating the implementations.
|
|
- Bugs. If the library has a bug that violates the specification, a program
|
|
that depends on the buggy behavior may break if the bug is fixed. We reserve
|
|
the right to fix such bugs.
|
|
- Adding methods or fields to generated structs. These may conflict with field
|
|
names that already exist in a schema, causing applications to break. When the
|
|
code generator encounters a field in the schema that would collide with a
|
|
generated field or method name, the code generator will append an underscore
|
|
to the generated field or method name.
|
|
- Adding, removing, or changing methods or fields in generated structs that
|
|
start with `XXX`. These parts of the generated code are exported out of
|
|
necessity, but should not be considered part of the public API.
|
|
- Adding, removing, or changing unexported symbols in generated code.
|
|
|
|
Any breaking changes outside of these will be announced 6 months in advance to
|
|
protobuf@googlegroups.com.
|
|
|
|
You should, whenever possible, use generated code created by the `protoc-gen-go`
|
|
tool built at the same commit as the `proto` package. The `proto` package
|
|
declares package-level constants in the form `ProtoPackageIsVersionX`.
|
|
Application code and generated code may depend on one of these constants to
|
|
ensure that compilation will fail if the available version of the proto library
|
|
is too old. Whenever we make a change to the generated code that requires newer
|
|
library support, in the same commit we will increment the version number of the
|
|
generated code and declare a new package-level constant whose name incorporates
|
|
the latest version number. Removing a compatibility constant is considered a
|
|
breaking change and would be subject to the announcement policy stated above.
|
|
|
|
## Plugins ##
|
|
|
|
The `protoc-gen-go/generator` package exposes a plugin interface,
|
|
which is used by the gRPC code generation. This interface is not
|
|
supported and is subject to incompatible changes without notice.
|