Unobserved null reference exception

[janpieterz]

Describe the bug

About once or twice a week on production (using Azure SQL) we get an unobserved null reference exception.
I've run some analyzers on our code and there's not a single task that is not awaited.

Today we had a caught exception right before the unobserved one (Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.).

The stack trace doesn't indicate any code from us and it doesn't seem related to above (I traced all code paths to ensure its error indeed caught the exceptions).

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Object reference not set to an instance of an object.)
 
---> System.NullReferenceException: Object reference not set to an instance of an object.
   
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadAsyncCallback(IntPtr key, PacketHandle packet, UInt32 error)
   
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadAsyncCallback(PacketHandle packet, UInt32 error)
   
at Microsoft.Data.SqlClient.ManagedSni.SniMarsHandle.HandleReceiveError(SniPacket packet)
   
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.HandleReceiveError(SniPacket packet)
   
at Microsoft.Data.SqlClient.ManagedSni.SniMarsConnection.HandleReceiveComplete(SniPacket packet, UInt32 sniErrorCode)
   
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   
--- End of inner exception stack trace ---

To reproduce

Not really any reproduction since it feels like there's nothing we're doing specifically, though maybe worth noting it relates to a relatively intense and regular SQL activity (we're using a sp_getapplock regulary inside this method).

Perhaps worth noting we're wrapping the SQL client with Dapper.

Expected behavior

No unobserved task exceptions

Further technical details

Microsoft.Data.SqlClient version: 6.1.2
.NET target: net9
SQL Server version: Azure SQL
Operating system: Docker container running Ubuntu

Additional context
None, but happy to expose anything useful.

Similar to #746

[apoorvdeshmukh]

Thank you @janpieterz for reporting this.
Do you see this issue only with SqlClient 6.1.2 or any of the supported earlier versions as well?

[janpieterz]

@apoorvdeshmukh We've seen an increase moving to 6.1.2 from 6.1.1, but can't speak for sure if that's a red herring or not.

This weekend we have a good couple more cases, with no previous exception (so the initial comment with the timeout exception is unrelated, though perhaps it does indicate a relation to some kind of underlying database instability causing it).

It's not reliable, as in, it happens now and then, some hours nothing this weekend, then a few hours regular ones in a row. See screenshot below.

[apoorvdeshmukh]

@janpieterz Is it possible for you to try with the earlier versions of the driver such as 5.1 or 5.2?