Skip to content

Full Microsoft.Testing.Platform support... Eventually #275

Open

Description

@farlee2121

Motivation

Right now YoloDev supports Microsoft.Testing.Platform (MTP) via Microsoft.Testing.Extensions.VSTestBridge. This allows a smooth transition.
However, this does not allow us to leverage new features in MTP, like better hierarchy support and more granular code locations.

Experiment

I threw together an experiment demonstrating direct MTP support for expecto: https://github.com/farlee2121/expecto-puremtp-experiment

This experiment does handle

  • discovering tests
  • running tests will all expected outcomes

It does not handle

  • filtering
  • runsettings
  • command line arguments
  • code locations

Conclusion

I suggest we not move forward with direct MTP support.

Unfortunately, Visual Studio doesn't actually respect the new hierarchy system in MTP. In fact, it's handling the hierarchy worse and flattening all tests to one level, including the groupings which are shown but never executed.

Rider, for unclear reasons, does not work with the experiment repo even with MTP support enabled. It does work with the current YoloDev MTP support, but only on windows.

I still wanted to post the experiment as a starting point for whenever MTP support is sufficiently mature to pick this up again

Activity

  1. Alxandr commented on May 22, 2026

    @Alxandr
    Contributor

    Maybe open issues on MTP repo about the issues you found? That way they can be tracked?

  2. farlee2121 commented on May 22, 2026

    @farlee2121
    ContributorAuthor

    I know they're already trying to work with the VS test explorer team to fix the hierarchy issue.

    I'll have to poke around the rider issue a bit more to characterize the issue. My guess is I'll need to open that one with Jetbrains

  3. farlee2121 commented on May 22, 2026

    @farlee2121
    ContributorAuthor

    Ionide also doesn't support direct MTP yet. That's waiting on an official MTP client package

  4. farlee2121 commented on Nov 7, 2026

    @farlee2121
    ContributorAuthor

    Visual Studio support for id-based hierarchy is in the works and Ionide/FSAC support for MTP is close.
    So, I decided to come back to this.

    The experiment repo is pretty close to ready.

    Some outstanding issues/questions

    • I haven't implemented a replacement for runsettings to enable expecto-specific configuration. The options are testconfig.json or forwarding command line arguments. I'd lean toward forwarding command line arguments
    • There's not an easy way to test if the test node hierarchies are working correctly. I'm guessing i'll need to run the mtp-based test project in server mode to get the needed information
    • How we should integrate with YoloDev. I'd hoped I could make MTP integration a function users call in their test project's main, but MTP won't detect any extensions unless you let it generate the main method
  5. Alxandr commented on Nov 16, 2026

    @Alxandr
    Contributor

    Could the failing tests be used in detecting the hierarchy? I have some tests that are failing to check that we actually run tests, with some simple stdout validation (iirc). Also, I'm not entirely sure I understand the last point? Don't we already generate main?

  6. farlee2121 commented on Nov 16, 2026

    @farlee2121
    ContributorAuthor

    Using failing tests is an interesting idea, but I don't think it'd work for this case.
    I can't think of any command we give MTP that evaluates based on test hierarchy.
    There's filter, but they just pass that back to us and we evaluate the filter consistently with hierarchy.

    As to generating main.

    My understanding is that the current Expecto project template creates a Main.fs that invokes the Expecto cli directly while VSTest discovers YoloDev.Expecto.TestSdk via convention and then YoloDev discovers tests by attribute. So, both the Expecto CLI and VSTest are supported simultaneously, with MTP partially supported using VSTestBridge.

    My experiment supports MTP with it's own runner, not MPT running under VSTest.

    For MTP's own runner, they expect the test project to be an exe that starts MTP. So MTP and the and the Expecto CLI can't both be supported at once, since they both expect to be what runs when the project is executed.

    My thought was to make mtp an option similar to runTestsWithCLIArgs (maybe runTestsWithMtp). This way users could easily swap between cli experiences.

    The problem is that MTP's extensibility model heavily prioritizes code generation.
    Extensions, like the TrxReporter, are discovered at build time and included in a generated main method.
    It's possible to register extensions manually, but it's not expected and so it's rarely documented very well and users coming from other libraries will expect that referencing an extension package will auto-register the extension.

    This means expecto projects using the MTP runner cannot have an explicit main without turning off code generation and manually register every extension.

    Circling all the way back, that means adding MTP runner support to YoloDev will be a breaking change to many projects, requiring them to delete their main method

  7. Alxandr commented on Nov 17, 2026

    @Alxandr
    Contributor

    https://github.com/YoloDev/YoloDev.Expecto.TestSdk/blob/main/src/YoloDev.Expecto.TestSdk/build/net8.0/YoloDev.Expecto.TestSdk.targets#L6

    You can opt-in to MTP since the last version. We should have a breaking release at some point where it turns to an opt-out (or just forced feature). I think tests are ran in both modes (if I remember correctly).

    The sample test-project does not have main:

    https://github.com/YoloDev/YoloDev.Expecto.TestSdk/blob/main/test/Sample.Test/Sample.Test.fsproj

  8. farlee2121 commented on Nov 17, 2026

    @farlee2121
    ContributorAuthor

    Hmm. It's true that the incompatibility is with the Expecto cli and the current expecto template, not with YoloDev.Test.Sdk.

    I'll have to experiment with how EnableExpectoTestingPlatformIntegration (which in turn sets IsTestingPlatformApplication) handles having both test bridge and a direct MTP extension available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions