[API Proposal]: Replacement of forbidden characters in variable names in EnvironmentVariablesConfigurationProvider

[svick]

Background and motivation

On Unix systems, the names of environment variables are are often limited to just letters, numbers and the underscore. (This is not a limitation of the OS, but of many tools, including bash or Azure App Service.) This makes it impossible to use the EnvironmentVariablesConfigurationProvider to configure components that expect such names in their configuration. In particular, Microsoft.Extensions.Logging expects hierarchical keys that can look like Logging:LogLevel:Microsoft.Hosting, where the last part is a namespace, so it will very often include dots.

EnvironmentVariablesConfigurationProvider already replaces __ (double underscore) with :, so that hierarchical keys could be represented at all. The proposal here is to effectively expand this capability and make it configurable.

API Proposal

 namespace Microsoft.Extensions.Configuration.EnvironmentVariables;

 public class EnvironmentVariablesConfigurationSource : IConfigurationSource
 {
+    public IDictionary<string, char>? VariableNameCharacterReplacements { get; set; }
 }

 namespace Microsoft.Extensions.Configuration;

 public static class EnvironmentVariablesExtensions
 {
     public static IConfigurationBuilder AddEnvironmentVariables(this IConfigurationBuilder configurationBuilder);
     public static IConfigurationBuilder AddEnvironmentVariables(this IConfigurationBuilder configurationBuilder, string? prefix);
+    public static IConfigurationBuilder AddEnvironmentVariables(this IConfigurationBuilder configurationBuilder, string? prefix, IDictionary<string, char>? variableNameCharacterReplacements);
     public static IConfigurationBuilder AddEnvironmentVariables(this IConfigurationBuilder builder, Action<EnvironmentVariablesConfigurationSource>? configureSource);
 }

All replacements (including the implicit __ → :) would be performed at the same time. If the value is null, only the implicit : replacement happens.

API Usage

The code to replace three underscores with a dot would be:

configurationBuilder.AddEnvironmentVariables(null, new Dictionary<string, char> { { "___", '.' } });

With this, the environment variable assignment of Logging__LogLevel__Microsoft___Hosting=Debug would produce configuration equivalent to:

{
  "Logging": {
    "LogLevel": {
      "Microsoft.Hosting": "Debug"
    }
  }
}

Alternative Designs

More general

Allow arbitrary changes to keys and possibly even to values:

 public class EnvironmentVariablesConfigurationSource : IConfigurationSource
 {
+    public Func<string, string>? VariableNameTransformation { get; set; }
     // or
+    public Func<(string Name, string Value), (string Name, string Value)>? VariableTransformation { get; set; }
 }

In my opinion, the generality of these alternatives is not justified, since they lack a compelling use case and a more targeted API is more appropriate.

Though in the common case, usage of this alternative would not be any more complicated than of the main proposal:

configurationBuilder.AddEnvironmentVariables(null, name => name.Replace("___", "."));

More specific

If we only consider the Microsoft.Extensions.Logging case, then we only need to handle the dot character:

 public class EnvironmentVariablesConfigurationSource : IConfigurationSource
 {
+    public string? DotReplacement { get; set; }
 }

In my opinion, this is too narrow, since any of the forbidden characters can be expected by a library that is being configured. Specifically, we had a report of a library that expected a configuration key with a hyphen (-) (it was since changed to also accept the same key without a hyphen, though).

Risks

There shouldn't be any.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-extensions-configuration
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here are possible related and/or duplicate issues (I may be wrong):

• [Feature Request] Add a replacement for dot in EnvironmentVariablesConfigurationProvider #35989
• LogLevel Filter by Category may be not working on Linux (AppServices) when dots in CategoryName are replaced by underscore #60534

[mrek-msft]

Some validations will be needed. For example, following should be invalid, I think

configurationBuilder.AddEnvironmentVariables(null, new Dictionary<string, char> { { "__", '.' } });

Similarly it is questionable how should something like this behave or how to efficiently reject it

configurationBuilder.AddEnvironmentVariables(null, new Dictionary<string, char> { 
    { "?", '!' },
    { "!", '?' },
 });

[svick]

@mrek-msft Forbidding __ makes sense to me.

The other variant should switch ! and ? around, so I don't see a need to reject it, even though it indeed doesn't make sense.