SqlClient event counters not enabled when listening with App Insights SDK

[empowerRob]

Describe the bug

We are using Azure App Insights SDK [2.23.0] configured to capture event counters from multiple dotnet sources.

Recently we upgraded the SqlClient to a new version which brought with it a recent change in the way event counters are handled internally. This seems to have stopped the capturing of event counters by the App Insights SDK

My investigation has brought me to the following information - please confirm if my findings are correct.

This change in SqlClientEventSource.cs‎:

if (command.Command == EventCommand.Enable)
{
    Metrics.EnableEventCounters();
}

The use of only EventCommand.Enable here appears to be specific to what I think is causing the issue.

App Insights SDK calls to enable events for all event sources / counters to tracked. It calls to the method System.Diagnostics.Tracing.EventListener.EnableEvents which takes in an EventSource and sends the command of type EventCommand.Update with EventCommandEventArgs that has an internal state of enabled set true.

Here is the command it sends

eventSource.SendCommand(this, 0, 0, EventCommand.Update, true, level, matchAnyKeyword, arguments);

The change made to SqlClientEventSource.cs only enables metrics collection if EventCommand.Enable is provided. Meaning it's just not enabled when the System.Diagnostics.Tracing.EventListener.EnableEvents method is used.

Does that sound right?

EDIT: looks like this was a bug in sequencing / initialization of the SqlClientEventSource, this bug #3575 was fixed in a newer version and a backport of it to version 6.1.x has been requested and is in progress - thanks Edward Neal!

Further technical details

Microsoft.Data.SqlClient version: 6.1.1
.NET target: .NET 9.0
ApplicationInsights: 2.23.0

[edwardneal]

The logic in SqlClientEventSource.cs corresponds to EventCommandMethodCall in the deleted file SqlClientEventSource.NetCoreApp.cs:

if(command.Command != EventCommand.Enable)
{
    return;
}

// <set up polling counters>

Are you able to reproduce this in 7.0-preview1 or later? This had another change (#3575) which handled cases where the underlying event source was enabled before the SqlClientEventSource's static constructor ran. The VS debugger does this, and perhaps Application Insights does something similar.

[empowerRob]

Thanks for the reply @edwardneal. I will have to give that a try, as I did not see that prior implementation leaning on the Enable value explicitly.

It could be a timing issue, but I did not see in the App Insights code where it would call enable, so more digging might be required!

I'll grab the latest with that later fix and see if it helps.

[empowerRob]

@edwardneal I couldn't easily get my solution to work with ver 7.0-preview1 due to dependency conflicts. Instead I decided to debug the event source wiring. Looks like the bug fix you have pointed to would indeed fix it. Also looks like my original theory on Update vs Enable was not true, it is sending Enable.

Is there any chance we can get that fix in #3575 to apply to version 6.1.x

Here are some screenshots of my findings

Metrics is null - so the bug fix for capturing the intended state will sort it