[Rant] All the "wheel reinvention" going on in `Testing.Platform` is extremely disappointing

[julealgon]

Context

I opened another issue some time ago to show some of my frustration with what seemed to be untapped potential in the new Microsoft.Testing.Platform framework here:

• Testing.Platform should not be a blackbox: leveraging standard Hosting model and Program.cs for test application setup #4198

Some of my concerns then were addressed: it turned out I didn't fully comprehend how MTP worked behind the scenes and how one could actually manage their own Program.cs etc. That was a nice find for me.

However... after now having worked more closely with MTP as we start a strong push on our solution to migrate from NET472 to NET9 and add UI and API test projects (higher level test project types, basically), it became increasingly frustrating to see how barebones and limited most of the abstractions around MTP really are under the hood.

This is supposed to be a discussion/rant post based on my recent experience and hopefully it will have enough constructive criticism to steer some of these decisions in what I think would be a better direction.

Reinventing the wheel (and the entire universe while we are at it...)

I was super hopefull when I saw the TestApplicationBuilder class and how the "manual" Program.cs looked like.

"This really looks like they are following the modern pattern established for other application types, nice!"

But as soon as I actually started working with it, I realized how wrong I was. Starting with that IConfiguration abomination.

My first instinct:

"oh this is cool, so they expose builder.Configuration the same way other apps do. I'll just attach my Azure AppConfiguration provider here and..."

"...wait a second.... I can't call AddAzureAppConfiguration here..... oh...... WAIT...... this is a different IConfiguration type?"

So we have the first wheel reinvented right there. A completely custom IConfiguration, that has like 5% of the capabilities of the original, and of course doesn't work at all with all existing providers.

Related:

• [Microsoft.Testing.Platform] IConfiguration interface is too limited: expose keys or children for custom extensions #5492

Then I saw there is no builder.Services... weird. Surely I'm missing something.

And that's when everything clicked together: the entire thing is fake. Starting from TestApplicationBuilder down to all of its properties, everything was custom built and doesn't use any of the Microsoft.Extensions.* packages.

Why on earth would you even think about doing something like this! Just use the thing you already have abstracted that is super general purpose and battle-tested! It is absolutely insane to me that you'd recreate your own.

The consequence of this is that of course, that whole Program.cs is now completely useless to me. I can't register my services, I can't register my configuration providers, I can't tap into anything else that is standard Microsoft.Extensions.Hosting. I can't add OpenTelemetry, no hosted services, nothing.

It is a useless barely functional "full rewrite" of Microsoft.Extensions.Hosting that exposes a few convoluted interfaces for library authors only.

"All right.... I think I get this now. Let's try to make testconfig.json load environment variables now..."

• [Microsoft.Testing.Platform] testconfig.json should support setting environment variables #5491

And suddenly I'm dealing with another config file that was introduced to..... configure things... of course. Is it following the same pattern of appsettings.json? Of course not! It is fully custom!

Now we have a billion mechanisms for providing parameters and configuration settings:

• launchSettings.json: doesn't work properly with VS test integration. only applies to development regardless. currently useless
• runsettings: deprecated, apparently? fair enough
• testconfig.json a new thing... replacing runsettings, but of course, dropping a bunch of capabilities
• xunit.runner.json because why not have one custom made for a specific test framework? After all we are dealing with raw Json which is highly proprietary and not portable at all..... wait....
• appsettings.{env}.json: nah, tests won't use this by default. why would they? its just a general configuration abstraction that is used in all other project types... surely there is no reason at all to tap into it... right?
• dotnet CLI
• and of course, you have a custom MSBuild property where you can set cmd arguments as well... very discoverable, super idiomatic, great stuff

"But if you are using the MSBuild nuget package, testconfig.json is automatically copied and renamed to {application}.testconfig.json". Of course..... a random NuGet package, and not a project SDK like all other project types. Great idea guys.

But it doesn't stop there. You want to create an extension? Sure... just declare a random static method and then create a MSBuild property pointing to it in the target project. Surely super intuitive to do, particularly when the extension is not a NuGet package but a shared library in your solution. Very idiomatic, just ask people to add completely random custom properties and values to their csproj! Talk about ease of adding extensions!

Imagine if there was a hosting mechanism where you could..... add things, and configure them! That would be amazing, right? Nah... let's rely on extremely obscure MSBuild targets buried in transient package dependencies to automatically generate a bunch of code and hide that as much as possible from users! Those dumb users, they wouldn't be able to ever configure this themselves, we should hide all of it!

You know... the same way that ASP.NET Core and Worker projects hide all of their hosting and... no.... they don't do that? Oh... whatever, we will do it anyways!

Suggestions

Guys...... what on earth are you doing here? You have such a strong abstraction to build on top of, and you ditch all of it in favor of.... this mess?

Stop hiding host details from engineers

Just expose Program.cs. No more MSBuild shenanigans to "make it look like the old projects". That's completely useless. .NET engineers are used to the new hosting model for years now. They will be able to handle a new Program.cs file with 3 lines of setup, no problem.

All the MSBuild-based magic is insane right now.

Drop all this custom config nonsense

.NET already has an idiomatic configuration subsystem on Microsoft.Extensions.Configuration. All configuration should be through this, period. Drop this custom testconfig.json nonsense, use the standard appsettings.{env}.json, env vars, secrets providers. Tap into the existing IConfiguration interface. Encourage all framework authors to use it.

Libraries for anything else in the .NET ecosystem have extensions for being configured with the hosting model, usually via AddX methods. Those methods then rely on IOptions<T> to propagate configuration settings, that can be optionally bound to one of the many configuration providers (appsettings.json etc) or just be configured inline).

Just do that! AddXunit(XunitOptions...), AddHotReloadWhatever(HotReloadWhateverOptions...).

Again, get rid of the insane MSBuild-based magic and all that IExtension nonsense you invented.

Everything is DI-based

I don't want to have to use Xunit.DependencyInjection. I especially don't want to do it when the platform itself already has a hosting mechanism and even has a IServiceProvider.

Final Thoughts

This framework is an abomination. I had reservations before (as evidenced by that first issue I linked) but now that I've actually worked with it a bit more they became orders of magnitude worse.

I really really hope you'll reconsider (dramatically) some of the decisions you made thus far here. There is still time, this framework is brand new. Do thinks right, please.

I'll now go back to our new hodgepodge of a project with multiple workarounds and limitations.

• We use the .Worker SDK so that appsettings.{env}.json are copied automatically. Not intuitive at all, but it works...
• We force MTP runner via the UseMicrosoftTestingPlatformRunner MSBuild property (this has no reason to exist)
• We now maintain a custom envvar provider extension to be able to define environment variables (including DOTNET_ENVIRONMENT) when running the tests. Of course launchSettings.json doesn't work, why would it? That would be too easy
• We depend on Xunit.DependencyInjection, which itself uses the old callback-based host builder, because of limitations with xunit. If we want DI with another framework, well... we are screwed: this is xunit-specific, after all. Imagine if there was a central place where DI could be setup regardless of underlying test framework... wouldn't that be nice?
• We configure xunit settings, MTP settings, Azure settings, Playwright settings, and domain-specific settings, all using different mechanisms

[Youssef1313]

Thanks so much for your feedback. I read it, and will respond to the important points.

So we have the first wheel reinvented right there. A completely custom IConfiguration, that has like 5% of the capabilities of the original, and of course doesn't work at all with all existing providers.

One important design decision of MTP is that the core platform shouldn't have any dependencies. This was explained in more details by @Evangelink in #4198 (comment).

And that's when everything clicked together: the entire thing is fake. Starting from TestApplicationBuilder down to all of its properties, everything was custom built and doesn't use any of the Microsoft.Extensions.* packages.

That's the same answer, the core platform is expected to not have any dependencies. But that generally shouldn't be so much limiting for you. If it is, we will need more concrete cases of what you are trying to do and what's missing (similar to #5492 where it's more clear to us what feature is missing).

The consequence of this is that of course, that whole Program.cs is now completely useless to me. I can't register my services, I can't register my configuration providers, I can't tap into anything else that is standard Microsoft.Extensions.Hosting. I can't add OpenTelemetry, no hosted services, nothing.

Please have a read on Microsoft.Testing.Platform extensibility, which explains how you can extend the platform with your own stuff. If something is still not clear in that regard, please ask us back.

And suddenly I'm dealing with another config file that was introduced to..... configure things... of course. Is it following the same pattern of appsettings.json? Of course not! It is fully custom!

On that part, we don't have control over xunit, so xunit.runner.json is not in our hands. For RunSettings, that was the VSTest way of doing stuff, which has partial support via VSTestBridge (used only by MSTest, NUnit, and Expecto at the time of writing). For launchSettings, this is already supported today if you run with dotnet run (MTP apps are typical console apps after all, nothing special), and will be supported in dotnet test in .NET 10. Then testconfig.json is the MTP main way of configuration. It serves a very different purpose compared to launchSettings for example. launchSettings is read before the process starts, so it can correctly set the environment variables from the very beginning before starting the process. appsettings.*.json is irrelevant for MTP.

and of course, you have a custom MSBuild property where you can set cmd arguments as well... very discoverable, super idiomatic, great stuff

There are scenarios where setting command-line arguments via MSBuild property is important. For example, if you run dotnet test on a solution where you want to ignore a specific exit code only for a specific project. We have no other way to support this scenario very well apart from an MSBuild property. In the new dotnet test in .NET 10, we will be supporting RunArguments as well, which is more standard property that has existed in .NET SDK for long. What we will be doing is simply that TestingPlatformCommandLineArguments gets added to RunArguments. See here.

"But if you are using the MSBuild nuget package, testconfig.json is automatically copied and renamed to {application}.testconfig.json". Of course..... a random NuGet package, and not a project SDK like all other project types. Great idea guys.

We were trying to keep Microsoft.Testing.Platform package as minimal as possible, with anything "optional" kept separately (e.g, TrxReport is optional, so it's its own package, MSBuild stuff are optional, so they are in their own package, etc). Using a NuGet-provided MSBuild SDK has its own problems, so I don't think it's a good fit here. And having a custom MSBuild SDK as part of the .NET SDK itself puts us in the same release cycle as .NET SDK, which is also not so good.

But it doesn't stop there. You want to create an extension? Sure... just declare a random static method and then create a MSBuild property pointing to it in the target project. Surely super intuitive to do, particularly when the extension is not a NuGet package but a shared library in your solution. Very idiomatic, just ask people to add completely random custom properties and values to their csproj! Talk about ease of adding extensions!

What the MSBuild item does is that it simply calls the hook's AddExtensions in a generated file. You can always disable the auto-generated file and have the code on your own.

Just expose Program.cs. No more MSBuild shenanigans to "make it look like the old projects". That's completely useless. .NET engineers are used to the new hosting model for years now. They will be able to handle a new Program.cs file with 3 lines of setup, no problem.

At least I would love to do that, but we still want to consider the migration story from VSTest and try to make it easier. You still have the option to disable the auto-generated entry point. It's only that it's enabled by default.

We force MTP runner via the UseMicrosoftTestingPlatformRunner MSBuild property (this has no reason to exist)

UseMicrosoftTestingPlatformRunner is not our MSBuild property, it's xunit's (however, we have similar one for MSTest). The problem it tries to solve is simple: Support both VSTest and MTP, while keeping VSTest the default.

Of course launchSettings.json doesn't work, why would it? That would be too easy

As mentioned earlier, this is something we will change in dotnet test in .NET 10.

Overall, I understand why you find it confusing, but most of it really boils down to the fact that the core platform shouldn't have dependencies.

Please let us know if you have further questions.

[julealgon]

Thanks so much for your feedback. I read it...

Thank you for taking the time to go through it @Youssef1313 . I really do appreciate it.

One important design decision of MTP is that the core platform shouldn't have any dependencies. This was explained in more details by @Evangelink in #4198 (comment).
...
That's the same answer, the core platform is expected to not have any dependencies.

Yes, I know from that previous discussion why you guys went with this approach. It is not good though (of course, my opinion here).

You should very strongly reconsider this constraint and just use the common abstractions instead. I can't stress this point enough. If there is anything else I could do to highlight that please let me know.

But that generally shouldn't be so much limiting for you. If it is, we will need more concrete cases of what you are trying to do and what's missing (similar to #5492 where it's more clear to us what feature is missing).

When I say it becomes useless, it's not in the sense that "I can't create custom extensions" etc (after all, we did create one...), but in the sense that it just doesn't work at all for application concerns: that entire hosting model is only there to serve tool extensions and nothing else.

I can't configure my configuration providers (that I need for my test app), I can't configure my services (that I need to inject), I can't configure OpenTelemetry using OpenTelemetry.Extensions.Hosting because your hosting is incompatible, etc).

It is extremely limitting, to the point where we needed to add Xunit.DependencyInjection to be able to get some of those benefits back (although in a very convoluted, test framework-specific way as I mentioned). That entire package should be completely unnecessary if this was done "properly".

I want an open hosting model, where the framework's needs are defined (AddMsTest/AddXunit etc), but where I can also define my own needs.

A good example of how this works is isolated Azure Functions, which evolved over time to finally get a proper hosting experience similar to the generic host using HostApplicationBuilder, even though it has very complex and particular hosting requirements.

The consequence of this is that of course, that whole Program.cs is now completely useless to me. I can't register my services, I can't register my configuration providers, I can't tap into anything else that is standard Microsoft.Extensions.Hosting. I can't add OpenTelemetry, no hosted services, nothing.

Please have a read on Microsoft.Testing.Platform extensibility, which explains how you can extend the platform with your own stuff. If something is still not clear in that regard, please ask us back.

Again, I think you misunderstood my point here. I know what your entry point exposes, and I know I can use it. I'm just saying it is not good for application level concerns at all.

How do I add appsettings.{env}.json support in your pipeline? How do I connect to Azure AppConfiguration? How do I enable dependency injection for my test classes and inject a custom service into them?

None of this is possible out of the box, and attempting to implement any of that would mean recreating massive amounts of code from existing Microsoft.Extensions.* libraries. It completely defeats the point.

And suddenly I'm dealing with another config file that was introduced to..... configure things... of course. Is it following the same pattern of appsettings.json? Of course not! It is fully custom!

On that part, we don't have control over xunit, so xunit.runner.json is not in our hands.

Sure, I know that. But you have contacts and connections with each test library author. If you provided a robust configuration mechanism (...Microsoft.Extensions.Configuration....) there would be zero reason for a test framework author to ever come up with their own custom json/xml files: they would just tap into the existing, well-known idiomatic configuration system.

What you are doing here is similar to having each random ASPNET.Core library author come up with weird nonstandard ways to configure their libraries. Imagine we have a mvcSettings.json file, a blazorOptions.xml file, a automapper.yaml, a dapperConfiguration.ini etc, each one 100% custom for their specific needs. That does NOT happen because the ecosystem is standardized: everyone knows they are supposed to create a AddX method that takes in a configuration lambda that can be bound to the underlying IConfiguration mechanism. Anyone not doing that is doing it wrong.

It is the same thing here.

If MTP was standardized on a similar hosting model, all configuration options could be done via AddX methods and no library would ever need to define custom files/formats/environment variables/msbuild properties/anything of that sort.

Again..... please think about what you are doing here for the future. This should be easy to standardize, since the standard has already been defined for every other project type. You just need to take that and apply to test projects.

For launchSettings, this is already supported today if you run with dotnet run (MTP apps are typical console apps after all,
nothing special), and will be supported in dotnet test in .NET 10.

Yeah, I mentioned it for completeness but I knew it was going to be "fixed" in NET10. That's a step in the right direction there for sure if only for consistency alone.

Then testconfig.json is the MTP main way of configuration. It serves a very different purpose compared to launchSettings for example. launchSettings is read before the process starts, so it can correctly set the environment variables from the very beginning before starting the process. appsettings.*.json is irrelevant for MTP.

You are missing the point though. There is no reason why you couldn't just use the same appsettings.*.json mechanism for all settings and eliminate the need for testconfig.json. All of the things it does should be doable via the standard configuration, which you'd load at the very beginning of the process (again, like with all other application types) and then pass along those settings to the frameworks/extensions that need them.

You don't need to have everything depend on environment variables either. Things should depend on the configuration abstraction, IOptions<T>, which can be populated from multiple different sources including environment variables.

You are creating a brand new architecture/framework, it is the perfect time to modernize how you configure things using the new mechanisms that were invented since then.

And for environment variables, you said it yourself that .NET 10 is fixing the launchsettings.json compatibility issue, so it will just work for tests as well.

The reason we needed to customize testconfig.json to allow setting envvars as well was precisely because things in the test realm keep depending directly on them. You have the opportunity here to finally change that and have everything depend on IOptions instead which then opens it up for the consumer to use any configuration source they want: appsettings, secrets, appconfig, envvars, cmdline arguments, etc.

and of course, you have a custom MSBuild property where you can set cmd arguments as well... very discoverable, super idiomatic, great stuff

There are scenarios where setting command-line arguments via MSBuild property is important. For example, if you run dotnet test on a solution where you want to ignore a specific exit code only for a specific project. We have no other way to support this scenario very well apart from an MSBuild property.

Why would that be hard to setup at all? If you expose it as a setting on AddXWhatever, developers would be able to configure the status code mapping on a per project basis. This would be equivalent to configuring health checks, or ProblemDetails on an ASPNET.Core project.

I fail to see why this configuration can't be parameterized and set at startup time at all. Perhaps I'm missing something obvious, in which case I'd love for you to elaborate.

In the new dotnet test in .NET 10, we will be supporting RunArguments as well, which is more standard property that has existed in .NET SDK for long. What we will be doing is simply that TestingPlatformCommandLineArguments gets added to RunArguments. See here.

I hope you realize how convoluted that is. Once again you create test-specific things when an equivalent already exists natively, and now you have to reconcile.

If RunArguments existed, you should've just used that in the first place. This feels incredibly obvious to me but again, I don't have your internal context.

"But if you are using the MSBuild nuget package, testconfig.json is automatically copied and renamed to {application}.testconfig.json". Of course..... a random NuGet package, and not a project SDK like all other project types. Great idea guys.

We were trying to keep Microsoft.Testing.Platform package as minimal as possible, with anything "optional" kept separately (e.g, TrxReport is optional, so it's its own package, MSBuild stuff are optional, so they are in their own package, etc). Using a NuGet-provided MSBuild SDK has its own problems, so I don't think it's a good fit here. And having a custom MSBuild SDK as part of the .NET SDK itself puts us in the same release cycle as .NET SDK, which is also not so good.

I 100% get the intent of keeping optional extensions (such as Trx, Coverage, etc) in separate packages. I'm not disputing that at all and I think you should keep that approach for sure.

What I think is a massive mistake here is not to take a dependency on the core abstractions from Microsoft.Extensions.*. Configuration, DI, hosting, logging, metrics, all the core concepts have already been abstracted there over several years, and have standardized across all application types.

Even OpenTelemetry has full support for those abstractions, and that framework is built around a static SDK library with zero dependencies as well. I mentioned this earlier: these are other teams from Microsoft, all you need to do is talk amongst yourselves and share the knowledge on how it was built.

Apply the same thing that OTEL did: have your "zero dependencies" core thing, but have a Extensions.Hosting package of some kind that bridges your thing with "the modern development world of .NET" and brings in all the benefits that the standard hosting provides.

What you are doing currently is that you are baking in a poor version of those things into your library. This will never scale properly and you are going to keep getting infinite requests to make whatever you created there be on par with the Microsoft.Extensions.* libraries. "I want DI support", "I want to be able to add Json configs, azure configs", "I want to add OpenTelemetry to my tests". Are you going to completely reimplement all of Microsoft.Extensions inside of MTP?!

But it doesn't stop there. You want to create an extension? Sure... just declare a random static method and then create a MSBuild property pointing to it in the target project. Surely super intuitive to do, particularly when the extension is not a NuGet package but a shared library in your solution. Very idiomatic, just ask people to add completely random custom properties and values to their csproj! Talk about ease of adding extensions!

What the MSBuild item does is that it simply calls the hook's AddExtensions in a generated file. You can always disable the auto-generated file and have the code on your own.

Sure. Again, I know that. I'm just arguing it should be the default and the other "form" shouldn't even need to exist in the first place.

Instead, you guys went the opposite route and made the convoluted thing the default, and the super idiomatic, standard thing, the optional path.

With that said... I'll look into not using that MSBuild-driven extension point and move to the manual Program.cs for sure.

Just expose Program.cs. No more MSBuild shenanigans to "make it look like the old projects". That's completely useless. .NET engineers are used to the new hosting model for years now. They will be able to handle a new Program.cs file with 3 lines of setup, no problem.

At least I would love to do that, but we still want to consider the migration story from VSTest and try to make it easier. You still have the option to disable the auto-generated entry point. It's only that it's enabled by default.

I fail to see why you think it would be that challenging to enforce the Program.cs to be there... how many lines does it even need? 5 at most?

Compare that to any other project type and it's not even close. .NET engineers are used to the hosting model by now. This shouldn't be this big of a barrier you guys are making it out to be.

This constraint you imposed on yourselves is introducing all sorts of convoluted, archaic, nonstandard concepts that shouldn't need to exist at all (like that IExtension interface I mentioned).

We force MTP runner via the UseMicrosoftTestingPlatformRunner MSBuild property (this has no reason to exist)

UseMicrosoftTestingPlatformRunner is not our MSBuild property, it's xunit's (however, we have similar one for MSTest). The problem it tries to solve is simple: Support both VSTest and MTP, while keeping VSTest the default.

Just don't make VSTest the default... have MTP not work with it.

Again, you have such a great opportunity here to restart things from scratch, and you are clinging to these "compatibility concerns" that don't even need to exist. MTP is brand new. People creating new projects now would use it directly, and those who want to migrate, perform the migration.

It's not like VSTest is deprecated or anything. You can still "target" it just fine.

Overall, I understand why you find it confusing, but most of it really boils down to the fact that the core platform shouldn't have dependencies.

This is a terrible mistake. MTP will never properly scale if you don't tap into the well-known hosting abstractions. Those hosting abstractions will NOT move anywhere either anytime soon. There is no option here.

Once again thanks for at least listening to my rants. I'm coming at this as someone who has been working with .NET for 15 years, creating unit test projects since day 1, and I see such a gigantic opportunity with this that I am basically forced to chime in. If you can make MTP work with the existing hosting abstractions, you will have a winner in your hands. Otherwise, I just don't see this ever working "well", it's always going to be filled with hacks, duplicated solutions, framework-specific solutions, etc.

[Evangelink]

@julealgon thanks for the detailed, rough but honest, feedback!

I totally understand your point of view which I would 100% share if I didn't spent time working in testing space at Microsoft. We have many teams that are in need of testing at different levels (e.g. runtime is needing to test really low level APIs where most of Microsoft.Extensions.* would not work...). We have also had many experiences with VSTest where having a dependency is bitting us in the future because at Microsoft you cannot easily deprecate or remove something - back compat forever.

Let me take some examples:

1. let's say we depend upon Microsoft.Extensions.Configuration v10 and the team working on this tool is working on v11 that contains a breaking change then suddenly they can no longer test their library.

2. now, let's say MTP depends upon Microsoft.Extensions.Configuration v10 and your project depends upon v8, we will impact the runtime of your app through our test environment meaning you are not truly testing what you will deploy/ship. One solution here is to target the oldest version possible which comes with the limitation of hitting potential breaking change (see point 1) and could cause users not to be happy because they cannot use feature x or y.

Just don't make VSTest the default... have MTP not work with it.

If you manage to convince our leadership, consider that @Youssef1313 and I would work night and weekend and drop everything VSTest on the shortest notice ever.

Now we have a billion mechanisms for providing parameters and configuration settings:

I agree that's painful, sadly and although you are pointing the fingers at us, there is no global config file in .NET that we can extend (runtimeconfig.json is proprietary of runtime, appsettings.json to ASP.NET...) so yeah we had to add our config file. I again understand your view and frustration, if you are managing to get some approval from other teams we can reuse and extend their config file then we are happy to consider evolutions/replacements.

If you can make MTP work with the existing hosting abstractions, you will have a winner in your hands.

During my spare time, I have done some experiments with doing a wrapper from Microsoft.Extensions to our core platform. This is easy and work well for everything that is supported by the core platform interfaces and obviously doesn't work well for the rest.

The only other approach I could see is to have 2 versions of the platform (one with the deps and one without) which isn't that hard but it gets harder for extensions (including test frameworks) needing to be also duplicated.

I'll try to find some key devs/PMs from Microsoft.Extensions.* and runtime willing to discuss/brainstorm this topic so we can hopefully mitigate some of these issues.

@julealgon I am also happy to hear your suggestions

[julealgon]

I totally understand your point of view which I would 100% share if I didn't spent time working in testing space at Microsoft. We have many teams that are in need of testing at different levels (e.g. runtime is needing to test really low level APIs where most of Microsoft.Extensions.* would not work...). We have also had many experiences with VSTest where having a dependency is bitting us in the future because at Microsoft you cannot easily deprecate or remove something - back compat forever.

Let me take some examples:

1. let's say we depend upon Microsoft.Extensions.Configuration v10 and the team working on this tool is working on v11 that contains a breaking change then suddenly they can no longer test their library.
2. now, let's say MTP depends upon Microsoft.Extensions.Configuration v10 and your project depends upon v8, we will impact the runtime of your app through our test environment meaning you are not truly testing what you will deploy/ship. One solution here is to target the oldest version possible which comes with the limitation of hitting potential breaking change (see point 1) and could cause users not to be happy because they cannot use feature x or y.

Man.... I get these, but we are punishing the entire ecosystem because of these special cases. There needs to be some other solution to this besides "let's not depend on anything" IMHO. Think about it, how many engineers are testing projects that are potential dependencies of the test framework itself? Surely that's less than 0.001% of all test projects out there.

On 1:

For example.... say I was the owner of Microsoft.Extensions.Configuration and needed to test my changes using MTP that also depends on Microsoft.Extensions.Configuration... Wouldn't it be possible to just generate a "test version" of Microsoft.Extensions.Configuration with a different assembly name that would work independently of the "real" Microsoft.Extensions.Configuration dependency? Surely if you had both DLLs in place, and they were actually 2 distinct assemblies, you'd be able to use a global alias to alias your custom version and even keep the exact same namespaces in place.

There must be something like this that would enable MTP to use one version, and the actual tests test another version.

Even if that's a bit clunky, it only impacts a very small subset of the real world test projects and the added annoyance is not even comparable with the massive benefits it would bring everyone else.

On 2:

This should honestly be even easier. If a consumer wants to test using Microsoft.Extensions.* v8 to match their real apps v8 usage, they just use a version of MTP that targets those versions. If MTP releases a new version that now relies on v9 or v10 for newer features, then the users make an informed decision:

1. they are fine with testing against a different version of this shared library (super low risk here, these are incredibly stable libraries), so they just upgrade MTP without necessarily upgrading the dependencies on their real apps
2. they want to make sure to use the same versions across, so they update the dependencies on their app and update MTP so both match
3. they can't currently update the deps on their app, so they just avoid the MTP update and stick with the version that targeted v8

To me, these are all extremely viable options. And they are in a sense quite similar to, say, how ASP.NET Core works. If you have libraries targeting Microsoft.Extensions.* v8 and then decide to move to ASP.NET Core v9, you are also basically forced to upgrade those dependencies. Same thing would happen here.

All you'd need to do is potentially support multiple versions of MTP at any given time (again... something that is quite normal), until the frameworks are considered out of support.

For example, MTP right now would have a version depending on Microsoft.Extensions.* v8 (since NET8 is LTS), and optionally another version targeting v9 libraries, but that's really only necessary if MTP itself needs something out of those newer versions (or if vulnerabilities are found etc). Then once STS for NET9 ends, you drop support for anything referencing v9 extensions and publish a new version targeting v10 extensions.

This feels honestly very doable to me, but again I don't have all the context you guys do.

Just don't make VSTest the default... have MTP not work with it.

If you manage to convince our leadership, consider that @Youssef1313 and I would work night and weekend and drop everything VSTest on the shortest notice ever.

If there is any way I could do that, I would. If there are any channels beside github that I could voice my concern on this, please let me know. Hell... give me the PM's email or setup a call or whatever, I'm in.

People usually don't treat test libraries the same way they do with "real" libraries, but this moment here is such a key moment in test framework's timeline where we have this insane opportunity to finally unify things. This needs to be ingrained in the mind of whoever is leading this.

Now we have a billion mechanisms for providing parameters and configuration settings:

I agree that's painful, sadly and although you are pointing the fingers at us, there is no global config file in .NET that we can extend (runtimeconfig.json is proprietary of runtime, appsettings.json to ASP.NET...) so yeah we had to add our config file.

No no no, this is incorrect. appsettings.json started in ASP.NET Core, but it is not exclusive to it anymore, and it's been several years since that's been the case. If you take a look at HostApplicationBuilder itself, which is the new, modern general form of the old generic host, it registers all of the appsettings.*.json and secrets just as WebApplicationBuilder does for ASP.NET: this has been standardized now. We use appsettings.json on our workers, and even on our Azure Functions.

This is why I was so happy to see that TestApplicationBuilder, only to become super disapointed to learn it was really not following the same idea but reinventing every sub-component of the hosting model.

This is from one of the console apps we just migrated to .NET9:

We use this mechanism today in many different app types:

• ASP.NET Core apps
• legacy ASP.NET WebForms apps (yep... even there, altough we don't run the host on those, we use all the other capabilities of it)
• Windows Service apps (using the Worker SDK, which even handles these files automatically for us by copying them etc)
• console apps (also using the Worker SDK)
• Azure Functions
• etc

What you are saying there was true at one point, but it is not anymore. Most of the Microsoft.Extensions.* libraries started targeting ASP.NET Core use cases, but were quickly standardized to also work outside of it. This is even more true now that the newer property-based API was created (HostApplicationBuilder).

If this argument was used to justify not tapping into appsettings.*.json before, it needs to be reconsidered.

The only other approach I could see is to have 2 versions of the platform (one with the deps and one without) which isn't that hard but it gets harder for extensions (including test frameworks) needing to be also duplicated.

If you really want to have a super low level, zero dependencies "version" of MTP, then again my suggestion would be to check what the OpenTelemetry dotnet team did. They have a "raw" version, that models the OTEL SDK as it is "documented" by the general spec, and then they have OpenTelemetry.Extensions.Hosting which completely bridges that raw low level thing with the modern .NET hosting model, including full support for IConfiguration, DI, logging, etc.

Having just one version of MTP that is heavily compromised due to special cases is really not ideal at all. As I mentioned earlier, it punishes 99% of the population for 1% special cases that would need some workarounds. And those percentages are likely bloated... the number of people who would have problems running a test project that depends on Microsoft.Extensions.* is probably lower than that.

And maybe, even in that scenario, they could just use old test projects? Honestly, even that would be a better way to handle this than butchering the new platform.

I'll try to find some key devs/PMs from Microsoft.Extensions.* and runtime willing to discuss/brainstorm this topic so we can hopefully mitigate some of these issues.

Please... this is such a great opportunity. @davidfowl and others have worked a lot on the hosting standardization effort, I'm sure he could share his two cents here. It needs to be top priority somehow, or we will never get to a point where writing any type of test project is really seamless.

Thanks @Evangelink .

[Evangelink]

Again a big thank you for all the valuable comments.

[nohwnd]

Very useful comments👍🏻

If you really want to have a super low level, zero dependencies "version" of MTP, then again my suggestion would be to check what the OpenTelemetry dotnet team did. They have a "raw" version, that models the OTEL SDK as it is "documented" by the general spec, and then they have OpenTelemetry.Extensions.Hosting which completely bridges that raw low level thing with the modern .NET hosting model, including full support for IConfiguration, DI, logging, etc.

Looking at open telemetry code, and reading their docs, this is what I see. They provide OpenTelemetry.Api package that people can depend on when authoring instrumentation, and that implements the "raw" spec. As recommended by them here, in the following excerpt:

Libraries providing SDK plugins such as exporters, resource detectors, and/or samplers should take a dependency on the OpenTelemetry SDK package.
Library authors providing instrumentation should take a dependency on OpenTelemetry.Api or OpenTelemetry.Api.ProviderBuilderExtensions package.
OpenTelemetry.Api.ProviderBuilderExtensions exposes interfaces for accessing the IServiceCollection which is a requirement for supporting the .NET Options pattern.

Where "OpenTelemetry SDK package" refers to OpenTelemetry nuget package, and "OpenTelemetry.Api" refers to the nuget of the same name.

Looking at the dependencies of OpenTelemetry.Api, it is indeed pure and only depends on the runtime part that allows it to instrument stuff. This would be the low level, zero dependencies package that you refer to.

They also recommend that exporters etc, depend on the OpenTelemetry package, which does depend on the Microsoft.Extensions.* dependencies.

The glue between the two layers is the OpenTelemetry.Api.ProviderBuilderExtensions package (and the fact that IServiceCollection is part of runtime).

There we can see for example for Tracing, that OpenTelemetry.Api provides abstractions for creating a tracer (IDeferredTracerProviderBuilder, TracerProviderBuilder) which are used by the higher level packages to do composition via dependency injection and configuration via Configure callback.

The .Api package also provides TraceProvider.Default, which can create tracers and do tracing, but without all the additional features of reporting etc.

Looking at AddConsoleReporter for example, which is not part of the raw package, we can see that the extension for the builder is implemented here

And the overloads that actually configure it are used like this: https://github.com/OneUptime/oneuptime/blob/master/Examples/otel-dotnet/Program.cs#L61

The types mentioned there are also not part of the .Api package https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/Metrics/Reader/MetricReaderOptions.cs they come from the open telemetry SDK.

This is the ConfigureServices used there, https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Api.ProviderBuilderExtensions/Trace/OpenTelemetryDependencyInjectionTracerProviderBuilderExtensions.cs

I cannot find any "configuration" per se in the .Api package to make the scenario complete, but I can imagine that we simply use our own objects (as we do now) for the configuration, require users of the low level package to provide them, and otherwise we use defaults.

So for MTP to follow the same model, we would have to:

• split Microsoft.Testing.Platform into a Microsoft.Testing.Platform."api" package, that holds the real real core.
• take the current MTP and change how we construct it (probably).
• rename the types like IConfiguration to avoid confusion
• agree that we want to use the Microsoft.Extensions.* as dependencies for all additional services like any non-core extensions.
• extension authors (like xunit, or mstest) would have to choose if they want to depend on the full version that uses microsoft.extensions.* or not, and possibly provide 2 versions one for low-level only and one for full.
• use the oldest supported version of the microsoft.extensions* and make sure that is enough for our purposes, with user being able to update by installing newer packages (but this exposes us to breaking changes)
• write wrappers around our configuration map from microsoft.extensions* in places where it needs to propagate to the real core
• more?

[thomhurst]

I understand the zero dependency thing, I'm assuming because if your test platform has a dependency, that your system-under-test also has, then you might run into conflicts. It might upgrade your sut dependency which then isn't a true reflection of your app.

But I also get the huge potential you could get by using the Microsoft.Extensions.* packages.

To avoid transitive nuget dependency and potential conflicts, could you use Microsoft Extension packages, reference them as development dependencies (so it's not seen as referencing those packages when packaged up), and then do some clever msbuild logic where those packages' DLLs are placed within the MTP packages themselves under the /lib path, meaning they can still reference extensions code, it shouldn't conflict with users (because you essentially have your own sandboxed version if I'm thinking correctly?), and then you've opened up an entire massive ecosystem of possibilities!

I may be being naive or not considered something, but wouldn't that work?

The caveat would be that users wouldn't be able to manually update the backing version of the microsoft extensions packages, because they'll be whatever version was used at the time of packing MTP, but I think that's a small price to pay, and unlikely to affect most people tbh since they're mostly abstractions anyway. And MTP can just regularly keep them up to date if they don't then conflict with user's SUTs.

[Youssef1313]

If we place Microsoft.Extensions.* dlls under lib, NuGet will be considering them to be copied to output directory. So I think this will still have the same issue.

[thomhurst]

If we place Microsoft.Extensions.* dlls under lib, NuGet will be considering them to be copied to output directory. So I think this will still have the same issue.

Damn! I'm not an msbuild expert so is this a hard limitation?

Is there no way to provide them their own output path? e.g. $(OutputDirectory)/MTP/*.dll

Or can they be compiled directly into the app?

[Youssef1313]

I think at the end of the day, only a single version of the dll will be passed to Csc. We could hack it so that the user dll is the one that's always present in some way, but then we risk binary incompatibilities between the version MTP compiles against and the version the user compiles against.

[thomhurst]

You can load two different versions of the same assembly though if they're publicly signed can't you?