Skip to content

Can Test Explorer match the Expecto folder hierarchy? #431

Open

Description

@charlesroddie

Given tests in a folder structure:

[<Tests>]
let tests = 
  testList "ListA" [
    testList "ListAA" [
      testCase "CaseAA1" id
      testCase "CaseAA2" id
    ]
    testCase "CaseAB1" id
  ]

Is there a way to have Test Explorer match this?

Project, Namespace, Class isn't right
image

Project, Class isn't right either
image

@cmeeren in #364 was getting some correspondence, but the samples there are very constrained - every test is exactly 2 levels deep in the hierarchy.

Is there a spec for the Test Explorer behaviour, or should I figure out by trial and error what Expecto structures result in what outputs and then try to make a set of consrained types (FolderLevel1, FolderLevel2) and then try to generate an altered Expecto folder structure to get the desired Text Explorer behaviour?

Activity

  1. charlesroddie commented on Apr 23, 2022

    @charlesroddie
    Author

    @baronfel I'm taking up your suggestion to try Expecto (via https://github.com/YoloDev/YoloDev.Expecto.TestSdk) and it's good so far but I'm stuck at this point.

  2. charlesroddie commented on Apr 25, 2022

    @charlesroddie
    Author

    Even "Rectangular hierarchy", in which folders of degree 0 contain only tests, and folders of degree n contain folders of degree n-1, and the main degree is 3, doesn't work:

    image

    There seems to be a big problem connecting Expecto to Test Explorer. Test Explorer may be hacked together with strings but it definitely allows for deeper hierarchies than this.

    Am I right in posting here rather than in https://github.com/YoloDev/YoloDev.Expecto.TestSdk ?

  3. olivercoad commented on Apr 30, 2022

    @olivercoad

    Hi.
    It looks like you're using Visual Studio.

    I don't use Visual Studio very much anymore, but back in the day I used to use adamchester/expecto-adapter. It looks like it's been archived since then (I guess people use YoloDev.Expecto.TestSdk for that now), but I remember getting the nesting in the VS Test Explorer to work fine. See screenshot in adamchester/expecto-adapter#47. You could try seeing if it still works but it might need an older version of VS.

    Usually I just use the terminal for running tests now (ftestCase and ftestList are useful) so don't know much about YoloDev.Expecto.TestSdk, but I tried your example in VSCode to see what it's like with Ionide.

    The nesting works fine for what you had in your "Rectangular hierarchy".
    image

    "Some other test" doesn't even show up in the right place though, it should be nested under "List".
    Also, it only seems to identify testCases with a function defined in-line (CaseAAA1 shows up but CaseAAB1 doesn't). And of course, tests generated in a for loop don't show up either.

    There seems to be an issue with it Ionide or VSCode though. When I try to run tests from the test explorer it just hangs indefinitely showing the spinner. Nothing shows up in any of the VSCode output logs.
    image
    The TestResults/Ionide.trx file is generated successfully and has all the expected tests and test names, so I suspect if it actually ran properly it might show all the tests and in the correct spots.

    The output of dotnet test -t is exactly what I'd expect it to be, as is the Ionide.trx output file, so I'd guess the issue is probably in the interaction between YoloDev.Expecto.TestSdk and Visual Studio. (And the Ionide/VSCode issue is probably unrelated)

    The following Tests are available:
        List.Some other test
        List.ListA.ListAA.ListAAA.CaseAAA1
        List.ListA.ListAA.ListAAA.CaseAAA2
        List.ListA.ListAA.ListAAB.CaseAAB1
        List.ListA.ListAA.ListAAB.CaseAAB2
        List.ListA.ListAB.ListABA-generated-list.CaseABA0
        List.ListA.ListAB.ListABA-generated-list.CaseABA1
        List.ListA.ListAB.ListABA-generated-list.CaseABA2
        List.ListA.ListAB.ListABA-generated-list.CaseABA3
        List.ListA.ListAB.ListABA-generated-list.CaseABA4
        List.ListA.ListAB.ListABB.CaseABB1
        List.ListA.ListAB.ListABB.CaseABB2
    ...
    
  4. baronfel commented on Apr 30, 2022

    @baronfel
    Contributor

    Note that the VSCode feature is in its infancy, so I'd err on the side of a bug in our test discovery and execution. This issue should focus purely on the discovery and execution of expecto tests via the Yolodev adapter, which provides the bridge to the 'dotnet test' world that powers the CLI and VS experiences.

  5. olivercoad commented on Apr 30, 2022

    @olivercoad

    How does the VSCode feature discover and execute tests then (and where is it implemented?) if it's not with the yolodev adapter? I thought it runs something like dotnet test --logger "trx;Ionide.trx" to generate the report, meaning it would be using the same "'dotnet test' world" as VS.

    But agreed, this issue should focus on the Yolodev adapter with VS

  6. baronfel commented on Apr 30, 2022

    @baronfel
    Contributor

    The discovery happens via syntax analysis, which is way faster but not as accurate. The execution should happen via dotnet test, however. Can you raise an issue about any test execution issues on the Ionide VSCode repo?

  7. charlesroddie commented on Nov 17, 2025

    @charlesroddie
    Author

    There may be progress on this in the future via Microsoft.Testing.Platform, by allowing node trees to be specified explicitly (i.e. not relying on syntax analysis), and putting pressure on Test Explorer to support these trees.

    microsoft/testfx#2537 (comment) @Evangelink
    We have designed this new testing principles by trying to be as open as possible to the various scenario we could see in testing in general (not .NET specific) and it seems that supporting a tree of tests is what's most generic as it allows flat list (as VSTest), test group and tests (as in F# Expecto for example)...
    The main problem is that Test Explorer (VS and VS Code) doesn't support this. We have some discussions with the team owning this part and we hope we will be able to make this component evolve in the future but there is currently no short-term plan for it.

  8. Evangelink commented on Nov 21, 2025

    @Evangelink

    The new platform allows this but sadly Test Explorer is not modified and it looks like it won't be done (despite our various pings - and customer requests).

  9. farlee2121 commented on Oct 6, 2026

    @farlee2121
    Collaborator

    @Evangelink Any updates on support for hierarchy based on parent ids?

    I've been experimenting with direct MTP support for Expecto (not TestBridge), and this is a significant hangup.
    I tried influencing the hierarchy with TestMethodIdentifierProperty, but I can't seem to get a palatable result.

  10. Evangelink commented on Oct 7, 2026

    @Evangelink

    @farlee2121 we have some good news. We (actually @drognanar) were able to take over parts of VS Test Explorer (not yet for VSCode) and we were able to have some PoC working, now we need to do tidy-up and ensure everything is in-order for MTP. I'll ping you here when we have something you can use.

  11. farlee2121 commented on Oct 7, 2026

    @farlee2121
    Collaborator

    That's excellent news!

    I think we're also close to supporting the MTP runner and id-based hierarchy in Ionide/FSAutoComplete
    ionide/ionide-vscode-fsharp#2180

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions