Add generic gRPC stream forwarding #306

moroten · 2025-12-09T14:30:42Z

To be able to pass Bazel's build event stream (BES) to the same DNS name, without having to add an extra L7 router in front of the bb-storage frontend, add a configuration to forward specific gRPC methods to other backends. No authorization is possible on the passed through messages because Buildbarn has no knowledge about the semantics of the forwarded messages.

The gRPC reflection service has also been extended to forward requests that cannot be resolved locally.

moroten · 2025-12-09T14:31:06Z

I'm working on the tests, but that is not ready yet.

aspect-workflows · 2025-12-09T14:31:30Z

Test

26 test targets passed

Targets

//pkg/auth:auth_test [k8-fastbuild]                                                           884ms
//pkg/blobstore/buffer:buffer_test [k8-fastbuild]                                             47ms
//pkg/blobstore/completenesschecking:completenesschecking_test [k8-fastbuild]                 78ms
//pkg/blobstore/grpcclients:grpcclients_test [k8-fastbuild]                                   76ms
//pkg/blobstore/grpcservers:grpcservers_test [k8-fastbuild]                                   155ms
//pkg/blobstore/local:local_test [k8-fastbuild]                                               315ms
//pkg/blobstore/mirrored:mirrored_test [k8-fastbuild]                                         196ms
//pkg/blobstore/readcaching:readcaching_test [k8-fastbuild]                                   92ms
//pkg/blobstore/readfallback:readfallback_test [k8-fastbuild]                                 127ms
//pkg/blobstore/replication:replication_test [k8-fastbuild]                                   90ms
//pkg/blobstore/sharding/integration:integration_test [k8-fastbuild]                          167ms
//pkg/blobstore/sharding/legacy:legacy_test [k8-fastbuild]                                    863ms
//pkg/blobstore/sharding:sharding_test [k8-fastbuild]                                         1s
//pkg/blobstore:blobstore_test [k8-fastbuild]                                                 149ms
//pkg/builder:builder_test [k8-fastbuild]                                                     100ms
//pkg/capabilities:capabilities_test [k8-fastbuild]                                           126ms
//pkg/digest:digest_test [k8-fastbuild]                                                       101ms
//pkg/filesystem/path:path_test [k8-fastbuild]                                                283ms
//pkg/grpc:grpc_test [k8-fastbuild]                                                           44ms
//pkg/http/server:server_test [k8-fastbuild]                                                  77ms
//pkg/jmespath:jmespath_test [k8-fastbuild]                                                   422ms
//pkg/jwt:jwt_test [k8-fastbuild]                                                             117ms
//pkg/otel:otel_test [k8-fastbuild]                                                           77ms
//pkg/prometheus:prometheus_test [k8-fastbuild]                                               110ms
//pkg/util:util_test [k8-fastbuild]                                                           47ms
//pkg/x509:x509_test [k8-fastbuild]                                                           133ms

Total test execution time was 6s. 4 tests (13.3%) were fully cached saving 359ms.

To be able to pass Bazel's build event stream (BES) to the same DNS name, without having to add an extra L7 router in front of the bb-storage frontend, add a configuration to forward specific gRPC methods to other backends. No authorization is possible on the passed through messages because Buildbarn has no knowledge about the semantics of the forwarded messages. The gRPC reflection service has also been extended to forward requests that cannot be resolved locally.

pkg/grpc/routing_stream_forwarder.go

pkg/grpc/routing_stream_forwarder_test.go

pkg/grpc/server.go

pkg/grpc/simple_stream_forwarder.go

EdSchouten · 2025-12-10T12:19:05Z

pkg/grpc/simple_stream_forwarder.go

+	group.Go(func() error {
+		for {
+			msg := &emptypb.Empty{}
+			if err := outgoingStream.RecvMsg(msg); err != nil {


As far as I know, the only way to properly cleaning up the resources associated with a stream is to call RecvMsg() until a non-nil value is returned. I don't think that this implementation guarantees that, as we return immediately if incomingStream.SendMsg() fails.

Maybe instead of using errgroup.Group we should give in and use Buildbarn's own program.Group? Then we can do something like this:

https://github.com/buildbarn/bonanza/blob/7e085b3671950094bdfc76b4ad2814585f4f4a8e/pkg/storage/dag/grpc/uploader.go#L75-L84

According to https://pkg.go.dev/google.golang.org/grpc#ClientConn.NewStream, cancelling the context is enough. program.Group does give that guarantee, but as incomingStream.RecvMsg can block until the whole method returns, a go func is needed. That go func needs to be able to cancel the context, but may try to cancel after the whole method has returned. program.Group does not support that, so I stick with errgroup.

pkg/proto/configuration/grpc/grpc.proto

pkg/grpc/forwarding_stream_handler_test.go

moroten · 2025-12-10T20:08:11Z

I've tested the docker-compose deployment, adding the following to frontend.jsonnet:

    relays: [
      {
        endpoint: {address: "scheduler:8984"},
        services: ["buildbarn.buildqueuestate.BuildQueueState"],
      },
    ],

Then I successfully ran:

$ grpcurl -plaintext localhost:8980 list
build.bazel.remote.execution.v2.ActionCache
build.bazel.remote.execution.v2.Capabilities
build.bazel.remote.execution.v2.ContentAddressableStorage
build.bazel.remote.execution.v2.Execution
buildbarn.buildqueuestate.BuildQueueState
google.bytestream.ByteStream
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
grpc.reflection.v1alpha.ServerReflection
$ grpcurl -plaintext -d '{}' localhost:8980  buildbarn.buildqueuestate.BuildQueueState/ListPlatformQueues
{
  "platformQueues": [
...
  ]
}

pkg/grpc/server_transport_stream_context.go

pkg/grpc/forwarding_stream_handler_test.go

EdSchouten · 2025-12-15T09:25:55Z

pkg/grpc/reflection_relay.go

+	"github.com/jhump/protoreflect/v2/grpcreflect"
+	"github.com/jhump/protoreflect/v2/protoresolve"
+	v1reflectiongrpc "google.golang.org/grpc/reflection/grpc_reflection_v1"
+	v1alphareflectiongrpc "google.golang.org/grpc/reflection/grpc_reflection_v1alpha"


Is it common practice to still use the v1alpha protocol? If not, what are your thoughts on omitting it?

grpc-java added support for v1 as late as 2024. v1alpha proto file has been set as deprecated since three years. Still, grpc-go reflection.Register() uses both APIs. grpcurl added support for v1 in 2022, so I'm find with removing v1alpha support.

Three years for upgrading grpcurl should be enough. I'll remove v1alpha.

pkg/grpc/server.go

EdSchouten · 2025-12-15T09:28:46Z

pkg/grpc/reflection_relay.go

+		resolver := grpcreflect.NewClientAuto(backendCtx, grpcClient).AsResolver()
+		reflectionBackends = append(reflectionBackends, resolver)
+	}
+	combinedRemoteResolver := protoresolve.Combine(reflectionBackends...)


Question: What happens if one of the backends is down? Does this mean reflection stops functioning altogether?

Yes, for services bind that backend in the list. The combined resolver calls the reflection backends in order until something else than NotFound is returned. Is that acceptable?

moroten · 2025-12-15T10:10:53Z

Thank you for review. Busy today, will reply tonight.

moroten · 2025-12-31T07:55:55Z

@EdSchouten are we ready to merge this? (No hurry, is the holiday season.)

moroten · 2026-01-13T07:23:17Z

@EdSchouten Friendly ping.

pkg/grpc/routing_stream_handler.go

pkg/grpc/server.go

pkg/grpc/reflection_relay.go

pkg/grpc/server.go

moroten · 2026-01-19T14:21:00Z

@EdSchouten Friendly ping.

moroten force-pushed the forward-generic-grpc-streams branch from dbaf6e8 to 75d27e9 Compare December 9, 2025 15:14

Add tests and minor adjustments

e1e9af3

EdSchouten reviewed Dec 10, 2025

View reviewed changes

moroten added 2 commits December 10, 2025 16:21

Fix review comments apart from the tests

1011bb3

Fix tests with gomock

b8f9eff

Minor test adjustments

68d8e54

EdSchouten reviewed Dec 15, 2025

View reviewed changes

Review feedback

e8598d6

EdSchouten reviewed Jan 13, 2026

View reviewed changes

pkg/grpc/routing_stream_handler.go Outdated Show resolved Hide resolved

pkg/grpc/server.go Outdated Show resolved Hide resolved

pkg/grpc/reflection_relay.go Outdated Show resolved Hide resolved

pkg/grpc/reflection_relay.go Outdated Show resolved Hide resolved

Review feedback 2

080d955

moroten force-pushed the forward-generic-grpc-streams branch from 0e006a9 to 080d955 Compare January 13, 2026 15:22

EdSchouten reviewed Jan 13, 2026

View reviewed changes

pkg/grpc/reflection_relay.go Outdated Show resolved Hide resolved

pkg/grpc/server.go Outdated Show resolved Hide resolved

moroten added 2 commits January 14, 2026 00:07

Review feedback 3

9c87967

Minor adjustments

c10dc4b

EdSchouten approved these changes Jan 19, 2026

View reviewed changes

EdSchouten merged commit 5b5db75 into buildbarn:main Jan 19, 2026
3 checks passed

moroten deleted the forward-generic-grpc-streams branch January 19, 2026 14:39

Add generic gRPC stream forwarding #306

Add generic gRPC stream forwarding #306

Uh oh!

Conversation

moroten commented Dec 9, 2025

Uh oh!

moroten commented Dec 9, 2025

Uh oh!

aspect-workflows bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EdSchouten Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

moroten Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

moroten commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

EdSchouten Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

moroten Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

EdSchouten Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

moroten Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

moroten commented Dec 15, 2025

Uh oh!

moroten commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moroten commented Jan 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

moroten commented Jan 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aspect-workflows bot commented Dec 9, 2025 •

edited

Loading

moroten commented Dec 31, 2025 •

edited

Loading