kv/kvserver: TestAdminScatterWithDrainingNodes failed [drpc hanging BatchStream send]

[cockroach-teamcity]

kv/kvserver.TestAdminScatterWithDrainingNodes failed on release-25.4.0-rc @ 4f4179243a351bc48c00dd9243dc56b8af98e210:

goroutine 60028 [select, 74 minutes]:
storj.io/drpc/drpcmanager.(*Manager).manageStream(0xc0152d6960, {0xdac7538, 0xc0130373b0}, 0xc007ce4f08)
	external/io_storj_drpc/drpcmanager/manager.go:334 +0x14e
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc0152d6960)
	external/io_storj_drpc/drpcmanager/manager.go:323 +0xd8
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 60006
	external/io_storj_drpc/drpcmanager/manager.go:123 +0x946

goroutine 61035 [IO wait]:
internal/poll.runtime_pollWait(0x7f352e7cd5b8, 0x72)
	GOROOT/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc012403420, 0x72, 0x0)
	GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
	GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc012403400, {0xc00c675400, 0xc00, 0xc00})
	GOROOT/src/internal/poll/fd_unix.go:165 +0x466
net.(*netFD).Read(0xc012403400, {0xc00c675400, 0xc00, 0xc00})
	GOROOT/src/net/fd_posix.go:55 +0x4b
net.(*conn).Read(0xc00560d8d8, {0xc00c675400, 0xc00, 0xc00})
	GOROOT/src/net/net.go:189 +0xad
crypto/tls.(*atLeastReader).Read(0xc001b47980, {0xc00c675400, 0xc00, 0xc00})
	GOROOT/src/crypto/tls/conn.go:809 +0x89
bytes.(*Buffer).ReadFrom(0xc00d252d38, {0xda6de00, 0xc001b47980})
	GOROOT/src/bytes/buffer.go:211 +0x113
crypto/tls.(*Conn).readFromUntil(0xc00d252a88, {0x7f352d7e4bd0, 0xc012306240}, 0x5)
	GOROOT/src/crypto/tls/conn.go:831 +0x1da
crypto/tls.(*Conn).readRecordOrCCS(0xc00d252a88, 0x0)
	GOROOT/src/crypto/tls/conn.go:629 +0x365
crypto/tls.(*Conn).readRecord(...)
	GOROOT/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc00d252a88, {0xc015eae000, 0x1000, 0xc015eae004?})
	GOROOT/src/crypto/tls/conn.go:1385 +0x2dd
storj.io/drpc/drpcwire.(*Reader).read(0xc0042c8770, {0xc015eae000, 0x1000, 0x1000})
	external/io_storj_drpc/drpcwire/reader.go:68 +0xaa
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc0042c8770, {0xc00ee34300, 0x0?, 0x20})
	external/io_storj_drpc/drpcwire/reader.go:113 +0x8ab
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc0075e0d20)
	external/io_storj_drpc/drpcmanager/manager.go:230 +0x199
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 60467
	external/io_storj_drpc/drpcmanager/manager.go:122 +0x8dd

goroutine 57220 [select, 74 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0xdac7bd8, 0xc010c04c80})
	pkg/sql/distsql_running.go:254 +0x145
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func1({0xdac7bd8, 0xc010c04c80})
	pkg/util/stop/stopper.go:440 +0x116
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 56818
	pkg/util/stop/stopper.go:438 +0x24f

Parameters:

• attempt=1
• race=true
• run=3
• shard=42

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvserver: TestAdminScatterWithDrainingNodes failed #154316 kv/kvserver: TestAdminScatterWithDrainingNodes failed [C-bug C-test-failure O-robot P-2 T-db-server X-duplicate branch-release-25.4]
• kv/kvserver: TestAdminScatterWithDrainingNodes failed #154312 kv/kvserver: TestAdminScatterWithDrainingNodes failed [C-bug C-test-failure O-robot P-2 T-db-server branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-55974

[tbg]

Going through log.txt.

Hanging on a Drain RPC, here's the server side:

goroutine 60474 [semacquire, 74 minutes]:
sync.runtime_Semacquire(0xc00958f478?)
	GOROOT/src/runtime/sema.go:71 +0x25
sync.(*WaitGroup).Wait(0xc00958f470)
	GOROOT/src/sync/waitgroup.go:118 +0xa5
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).flush(0xc002ff1380, {0xdac7500, 0xc002f474f0}, 0xc00e68fe00, 0xc01224b800, {0xc007939e00, 0x14, 0x20}, {0xc001f32680, 0x9, ...})
	pkg/sql/sqlstats/persistedsqlstats/flush.go:220 +0x673
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).MaybeFlushWithDrainer(0xc002ff1380, {0xdac7500, 0xc002f474f0}, 0xc00e68fe00, {0xda8c570, 0xc008792000})
	pkg/sql/sqlstats/persistedsqlstats/flush.go:115 +0xb05
github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).MaybeFlush(0xc002ff1380, {0xdac7500, 0xc002f474f0}, 0xc00e68fe00)
	pkg/sql/sqlstats/persistedsqlstats/flush.go:52 +0x5b
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClientsInternal(0xc001d35cc0, {0xdac7bd8, 0xc001b4cbe0}, 0xc01276ac38, 0xe0?)
	pkg/server/drain.go:480 +0x91c
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClients(...)

The work in that WaitGroup in turn sits in DistSender

goroutine 61068 [chan receive, 74 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit(0xc008a4c608, {0xdac7bd8, 0xc005c87900}, 0xc001ac4780, {{0xc00bdd3aa0, 0x59, 0x60}, {0xc00bdd3aa0, 0x5a, 0x60}}, ...)
	pkg/kv/kvclient/kvcoord/dist_sender.go:1578 +0xd1d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc008a4c608, {0xdac7bd8, 0xc005c87860}, 0xc001ac4780)
	pkg/kv/kvclient/kvcoord/dist_sender.go:1278 +0x7d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc0052c9788, {0xdac7bd8, 0xc005c87860}, 0xc001ac4780)
	pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x26c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked(0xc0052c9738, {0xdac7bd8, 0xc005c87860}, 0xc001ac4780)
	pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:42 +0x1da

which sits on a Batch RPC:

goroutine 61096 [sync.Cond.Wait, 74 minutes]:
sync.runtime_notifyListWait(0xc0186db510, 0x0)
	GOROOT/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc0186db500)
	GOROOT/src/sync/cond.go:71 +0x75
storj.io/drpc/drpcstream.(*packetBuffer).Get(0xc0186db4f8)
	external/io_storj_drpc/drpcstream/pktbuf.go:65 +0x125
storj.io/drpc/drpcstream.(*Stream).MsgRecv(0xc0186db408, {0xbec23a0, 0xc0072dc780}, {0xda812a8, 0x13f38340})
	external/io_storj_drpc/drpcstream/stream.go:504 +0x174
storj.io/drpc/drpcconn.(*Conn).doInvoke(0x0?, 0xc0186db408, {0xda812a8, 0x13f38340}, {0xc1fe86f, 0x20}, {0xc018e00500, 0x2031, 0x2500}, {0x0, ...}, ...)
	external/io_storj_drpc/drpcconn/conn.go:152 +0x148
storj.io/drpc/drpcconn.(*Conn).Invoke(0xc019dcacc0, {0xdac7bd8, 0xc005c87900}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc00b4c7040}, {0xbec23a0, ...})
	external/io_storj_drpc/drpcconn/conn.go:131 +0x3fa
storj.io/drpc/drpcpool.(*poolConn[...]).Invoke(0xdb39580, {0xdac7bd8, 0xc005c87900}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc00b4c7040}, {0xbec23a0, ...})
	external/io_storj_drpc/drpcpool/conn.go:66 +0x366
storj.io/drpc/drpcclient.finalInvoker({0xdac7bd8, 0xc005c87900}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc00b4c7040}, {0xbec23a0, 0xc0072dc780}, ...)
	external/io_storj_drpc/drpcclient/clientconn.go:33 +0xe2
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13({0xdac7bd8, 0xc005c87900}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc00b4c7040}, {0xbec23a0, 0xc0072dc780}, ...)
	pkg/util/tracing/drpcinterceptor/drpc_interceptor.go:149 +0x4f0
storj.io/drpc/drpcclient.(*ClientConn).Invoke(0xc018ea3c80, {0xdac7bd8, 0xc005c87900}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc00b4c7040}, {0xbec23a0, ...})
	external/io_storj_drpc/drpcclient/clientconn.go:41 +0x18a
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*drpcKVBatchClient).Batch(0xc00e44de60, {0xdac7bd8, 0xc005c87900}, 0xc00b4c7040)
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api_drpc.pb.go:60 +0xc6
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*drpcKVBatchClientAdapter).Batch(0xc00e44de60, {0xdac7bd8, 0xc005c87900}, 0xc00b4c7040)
	bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api_drpc.pb.go:254 +0x45
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*unaryDRPCBatchServiceToInternalAdapter).Batch(0xc015030c60, {0xdac7bd8, 0xc005c87900}, 0xc00b4c7040)
	pkg/rpc/nodedialer/nodedialer_drpc.go:27 +0xd7
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc017406de0, {0xdac7bd8, 0xc005c87900}, 0x16e2788?, {0xda82d40, 0xc015030c60}, 0xc00b4c7040)
	pkg/kv/kvclient/kvcoord/transport.go:211 +0x22b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc017406de0, {0xdac7bd8, 0xc005c87900}, 0xc00b4c7040)
	pkg/kv/kvclient/kvcoord/transport.go:189 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xdb1b718?, 0xc017406de0?}}, {0xdac7bd8, 0xc005c87900}, 0xc00b4c7040)
	pkg/kv/kvclient/kvcoord/transport_race.go:70 +0x2ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc008a4c608, {0xdac7bd8, 0xc005c87900}, 0xc001ac4c80, {0xc0032df680, 0xc002385860}, 0x1)
	pkg/kv/kvclient/kvcoord/dist_sender.go:2768 +0x26ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc008a4c608, {0xdac7bd8, 0xc005c87900}, 0xc001ac4c80, {{0xc001f92540, 0x48, 0x60}, {0xc001f92660, 0x55, 0x60}}, ...)
	pkg/kv/kvclient/kvcoord/dist_sender.go:2227 +0x825
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc008a4c608, {0xdac7bd8, 0xc005c87900}, 0xc001ac4c80, {{0xc001f92540, 0x48, 0x60}, {0xc001f92660, 0x55, 0x60}}, ...)
	pkg/kv/kvclient/kvcoord/dist_sender.go:1775 +0xfd6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit.func1({0xdac7bd8, 0xc005c87900})
	pkg/kv/kvclient/kvcoord/dist_sender.go:1541 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit.func2({0xdac7bd8, 0xc005c87900})
	pkg/kv/kvclient/kvcoord/dist_sender.go:1560 +0x116
created by github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit in goroutine 61068
	pkg/kv/kvclient/kvcoord/dist_sender.go:1558 +0xaa5

Here is where it gets interesting because we don't see a single goroutine in (*Node).Batch or (*Node).BatchStream. This is actually quite strange and I wonder if goroutines are missing. The logs look healthy overall. Something is strange here.

The few other stacks that are in Invoke look similar to above - waiting on MsgRecv here:

func (c *Conn) doInvoke(stream *drpcstream.Stream, enc drpc.Encoding, rpc string, data []byte, metadata []byte, out drpc.Message) (err error) {
	if len(metadata) > 0 {
		if err := stream.RawWrite(drpcwire.KindInvokeMetadata, metadata); err != nil {
			return err
		}
	}
	if err := stream.RawWrite(drpcwire.KindInvoke, []byte(rpc)); err != nil {
		return err
	}
	if err := stream.RawWrite(drpcwire.KindMessage, data); err != nil {
		return err
	}
	if err := stream.CloseSend(); err != nil {
		return err
	}
	if err := stream.MsgRecv(out, enc); err != nil {

I still have no idea how it can be that we don't have a single BatchStream going on in this test.

We have ~83 stream managers:

$ grep manageReader log.txt  | wc -l
      83

This one sticks out for the wait time, but it could just be an idle connection.

goroutine 59977 [IO wait, 74 minutes]:
internal/poll.runtime_pollWait(0x7f3582154d78, 0x72)
        GOROOT/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc001da6fa0, 0x72, 0x0)
        GOROOT/src/internal/poll/fd_poll_runtime.go:84 +0xb1
internal/poll.(*pollDesc).waitRead(...)
        GOROOT/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001da6f80, {0xc013568800, 0x800, 0x800})
        GOROOT/src/internal/poll/fd_unix.go:165 +0x466
net.(*netFD).Read(0xc001da6f80, {0xc013568800, 0x800, 0x800})
        GOROOT/src/net/fd_posix.go:55 +0x4b
net.(*conn).Read(0xc018e3f188, {0xc013568800, 0x800, 0x800})
        GOROOT/src/net/net.go:189 +0xad
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0104cba40, {0xc013568800, 0x800, 0x800})
        external/com_github_cockroachdb_cmux/cmux.go:241 +0x2db
crypto/tls.(*atLeastReader).Read(0xc001d78150, {0xc013568800, 0x800, 0x800})
        GOROOT/src/crypto/tls/conn.go:809 +0x89
bytes.(*Buffer).ReadFrom(0xc00dd802b8, {0xda6de00, 0xc001d78150})
        GOROOT/src/bytes/buffer.go:211 +0x113
crypto/tls.(*Conn).readFromUntil(0xc00dd80008, {0x7f352f7921b0, 0xc0104cba40}, 0x5)
        GOROOT/src/crypto/tls/conn.go:831 +0x1da
crypto/tls.(*Conn).readRecordOrCCS(0xc00dd80008, 0x0)
        GOROOT/src/crypto/tls/conn.go:629 +0x365
crypto/tls.(*Conn).readRecord(...)
        GOROOT/src/crypto/tls/conn.go:591
crypto/tls.(*Conn).Read(0xc00dd80008, {0xc00513c000, 0x1000, 0x4a8ac8?})
        GOROOT/src/crypto/tls/conn.go:1385 +0x2dd
storj.io/drpc/drpcwire.(*Reader).read(0xc00c4fe460, {0xc00513c000, 0x1000, 0x1000})
        external/io_storj_drpc/drpcwire/reader.go:68 +0xaa
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc00c4fe460, {0xc003715020, 0x3c?, 0x60})
        external/io_storj_drpc/drpcwire/reader.go:113 +0x8ab
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc011aff4a0)
        external/io_storj_drpc/drpcmanager/manager.go:230 +0x199
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 59926
        external/io_storj_drpc/drpcmanager/manager.go:122 +0x8dd

There's also this which might be totally fine, I don't know.

goroutine 59562 [sync.Cond.Wait, 50 minutes]:
sync.runtime_notifyListWait(0xc019dd2fd0, 0x12)
	GOROOT/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc019dd2fc0)
	GOROOT/src/sync/cond.go:71 +0x75
storj.io/drpc/drpcmanager.(*streamBuffer).Wait(0xc019dd2fb8, 0x11f)
	external/io_storj_drpc/drpcmanager/streambuf.go:53 +0xba
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc019dd2f00)
	external/io_storj_drpc/drpcmanager/manager.go:282 +0x745
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 59536
	external/io_storj_drpc/drpcmanager/manager.go:122 +0x8dd

[tbg]

drpc is not in production, so no release blocker. Moving to server team for a look (slack)

[cockroach-teamcity]

kv/kvserver.TestAdminScatterWithDrainingNodes failed on release-25.4.0-rc @ 4f4179243a351bc48c00dd9243dc56b8af98e210:

crypto/tls.(*Conn).Read(0xc00d97d188, {0xc02ac3b000, 0x7000, 0xc02ac3b004?})
	GOROOT/src/crypto/tls/conn.go:1385 +0x2dd
storj.io/drpc/drpcwire.(*Reader).read(0xc008988ee0, {0xc02ac3b000, 0x7000, 0x7000})
	external/io_storj_drpc/drpcwire/reader.go:68 +0xaa
storj.io/drpc/drpcwire.(*Reader).ReadPacketUsing(0xc008988ee0, {0x0, 0x0?, 0x0})
	external/io_storj_drpc/drpcwire/reader.go:113 +0x8ab
storj.io/drpc/drpcmanager.(*Manager).manageReader(0xc0287bd4a0)
	external/io_storj_drpc/drpcmanager/manager.go:230 +0x199
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 200571
	external/io_storj_drpc/drpcmanager/manager.go:122 +0x8dd

goroutine 200585 [select]:
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc00d2203c0)
	external/io_storj_drpc/drpcmanager/manager.go:321 +0x1a6
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 200444
	external/io_storj_drpc/drpcmanager/manager.go:123 +0x946

goroutine 193383 [select]:
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc0325c8f00)
	external/io_storj_drpc/drpcmanager/manager.go:321 +0x1a6
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 56879
	external/io_storj_drpc/drpcmanager/manager.go:123 +0x946

goroutine 214141 [select]:
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
	external/com_github_cockroachdb_pebble/vfs/disk_health.go:254 +0x192
created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 57371
	external/com_github_cockroachdb_pebble/vfs/disk_health.go:249 +0xad

goroutine 186619 [select]:
storj.io/drpc/drpcmanager.(*Manager).manageStreams(0xc0034c8f00)
	external/io_storj_drpc/drpcmanager/manager.go:321 +0x1a6
created by storj.io/drpc/drpcmanager.NewWithOptions in goroutine 186606
	external/io_storj_drpc/drpcmanager/manager.go:123 +0x946

goroutine 145960 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00e7da0a8, 0xcf6)
	GOROOT/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc00e7da098)
	GOROOT/src/sync/cond.go:71 +0x75
github.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
	external/com_github_cockroachdb_pebble/record/log_writer.go:371
github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc00e7da000, {0xc0450d98d0?, 0xb4f1fe0?})
	external/com_github_cockroachdb_pebble/record/log_writer.go:673 +0xc07
runtime/pprof.Do({0xdac6ae8, 0x13f38340}, {{0xc001300be0?, 0x46a8e10?, 0xdac7bd8?}}, 0xc005574fc0)
	GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble/record.NewLogWriter.func2()
	external/com_github_cockroachdb_pebble/record/log_writer.go:587 +0x92
created by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 58985
	external/com_github_cockroachdb_pebble/record/log_writer.go:586 +0xa6e

Parameters:

• attempt=1
• race=true
• run=2
• shard=42

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

• kv/kvserver: TestAdminScatterWithDrainingNodes failed #154316 kv/kvserver: TestAdminScatterWithDrainingNodes failed [C-bug C-test-failure O-robot P-2 T-db-server X-duplicate branch-release-25.4]
• kv/kvserver: TestAdminScatterWithDrainingNodes failed #154312 kv/kvserver: TestAdminScatterWithDrainingNodes failed [C-bug C-test-failure O-robot P-2 T-db-server branch-master]

This test on roachdash | Improve this report!

[tbg]

^-- this does seem like some dRPC issue. We have a request stuck in .(*drpcKVBatchClient).Batch, but there (again) isn't a corresponding server-side goroutine.

goroutine 60770 [sync.Cond.Wait, 20 minutes]:
sync.runtime_notifyListWait(0xc019426610, 0x0)
        GOROOT/src/runtime/sema.go:587 +0x159
sync.(*Cond).Wait(0xc019426600)
        GOROOT/src/sync/cond.go:71 +0x75
storj.io/drpc/drpcstream.(*packetBuffer).Get(0xc0194265f8)
        external/io_storj_drpc/drpcstream/pktbuf.go:65 +0x125
storj.io/drpc/drpcstream.(*Stream).MsgRecv(0xc019426508, {0xbec23a0, 0xc001bf2f00}, {0xda812a8, 0x13f38340})
        external/io_storj_drpc/drpcstream/stream.go:504 +0x174
storj.io/drpc/drpcconn.(*Conn).doInvoke(0x0?, 0xc019426508, {0xda812a8, 0x13f38340}, {0xc1fe86f, 0x20}, {0xc011e16000, 0x173, 0x5c000}, {0x0, ...}, ...)
        external/io_storj_drpc/drpcconn/conn.go:152 +0x148
storj.io/drpc/drpcconn.(*Conn).Invoke(0xc01b191640, {0xdac7bd8, 0xc00bf8cb40}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc01951ab40}, {0xbec23a0, ...})
        external/io_storj_drpc/drpcconn/conn.go:131 +0x3fa
storj.io/drpc/drpcpool.(*poolConn[...]).Invoke(0xdb39580, {0xdac7bd8, 0xc00bf8cb40}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc01951ab40}, {0xbec23a0, ...})
        external/io_storj_drpc/drpcpool/conn.go:66 +0x366
storj.io/drpc/drpcclient.finalInvoker({0xdac7bd8, 0xc00bf8cb40}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc01951ab40}, {0xbec23a0, 0xc001bf2f00}, ...)
        external/io_storj_drpc/drpcclient/clientconn.go:33 +0xe2
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func13({0xdac7bd8, 0xc00bf8cb40}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc01951ab40}, {0xbec23a0, 0xc001bf2f00}, ...)
        pkg/util/tracing/drpcinterceptor/drpc_interceptor.go:149 +0x4f0
storj.io/drpc/drpcclient.(*ClientConn).Invoke(0xc007519a40, {0xdac7bd8, 0xc00bf8cb40}, {0xc1fe86f, 0x20}, {0xda812a8, 0x13f38340}, {0xc066da0, 0xc01951ab40}, {0xbec23a0, ...})
        external/io_storj_drpc/drpcclient/clientconn.go:41 +0x18a
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*drpcKVBatchClient).Batch(0xc0001406b0, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951ab40)
        bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api_drpc.pb.go:60 +0xc6
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*drpcKVBatchClientAdapter).Batch(0xc0001406b0, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951ab40)
        bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api_drpc.pb.go:254 +0x45
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.(*unaryDRPCBatchServiceToInternalAdapter).Batch(0xc001f5c630, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951ab40)
        pkg/rpc/nodedialer/nodedialer_drpc.go:27 +0xd7
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc008e50420, {0xdac7bd8, 0xc00bf8cb40}, 0x1752cd8?, {0xda82d40, 0xc001f5c630}, 0xc01951ab40)
        pkg/kv/kvclient/kvcoord/transport.go:211 +0x22b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc008e50420, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951ab40)
        pkg/kv/kvclient/kvcoord/transport.go:189 +0x13a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext({{0xdb1b718?, 0xc008e50420?}}, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951ab40)
        pkg/kv/kvclient/kvcoord/transport_race.go:70 +0x2ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc007178008, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951aa00, {0xc012f65020, 0xc00998c960}, 0x0)
        pkg/kv/kvclient/kvcoord/dist_sender.go:2768 +0x26ed
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc007178008, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951aa00, {{0xc009d6e120, 0x57, 0x60}, {0xc009d6e120, 0x58, 0x60}}, ...)
        pkg/kv/kvclient/kvcoord/dist_sender.go:2227 +0x825
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc007178008, {0xdac7bd8, 0xc00bf8cb40}, 0xc01951aa00, {{0xc009d6e120, 0x57, 0x60}, {0xc009d6e120, 0x58, 0x60}}, ...)
        pkg/kv/kvclient/kvcoord/dist_sender.go:1775 +0xfd6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc007178008, {0xdac7500, 0xc004839ea0}, 0xc01951aa00)
        pkg/kv/kvclient/kvcoord/dist_sender.go:1280 +0x82d
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc00dbd2d88, {0xdac7500, 0xc004839ea0}, 0xc01951aa00)
        pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:77 +0x26c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLocked(0xc00dbd28f0, {0xdac7500, 0xc004839ea0})
        pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:484 +0x61a
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeat(0xc00dbd28f0, {0xdac7500, 0xc004839ea0})
        pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:442 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).heartbeatLoop(0xc00dbd28f0, {0xdac7500, 0xc004839ea0})
        pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:412 +0x2e5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc00e375900, {0xdac7500, 0xc004839ea0}, {0xc20af42, 0x21}, 0xc010d7ff88)
        pkg/util/stop/stopper.go:318 +0x247
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).startHeartbeatLoopLocked.func1()
        pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:365 +0x21c
created by time.goFunc
        GOROOT/src/time/sleep.go:215 +0x45

[pav-kv]

Sounds like a dup of #154312