Including testsets from within testsets

[dpinol]

Hi,
The limitation to not include files containing testsets from within testsets is a showstopper for our project, since we have our tests within a hierarchy of folders mimicking the hierarchy in the src folder.
Eg
test/runtests.jl:

@testset "utils" begin
 include("utils/1.jl")
end

test/utils/1.jl:

@testset "1" begin
 @test ....
end

Is there any workaround? Do you plan to offer one in the future?

thanks

[rfourquet]

There is some partial improvement on this front, available when ReTest.hijack is used, via the include=:static keyword. It would be great if you can try it out and report if it works as intended for your test code. This is available on the "master" branch, and I'm about to make a release with it, which should be available later today.

If you want to use ReTest directly in your code, you could try to write your @testset with the undocumented option static_include=true (e.g. @testset "description" static_include=true begin ... end, this might not be the final API). If you want to keep your tests compatible with both Test and ReTest as highlighted at https://juliatesting.github.io/ReTest.jl/stable/#Keeping-the-ability-to-use-Test, I could maybe add the include=:static option also to the retest function.

But for ReTest in general, this option with include looks like a hack (as include can have other side effects beside defining testsets, which are then executed only once with include=:static. A more fit solution still has to be designed. Maybe a simple @testset_include macro or something like that would do.

[Sacha0]

We are exploring migration from XUnit to ReTest, and the ability to include files containing testsets within a testset is important to us as well: Much of our test code is structured as @dpinol describes above, with multiple levels of includes within includes. Maintaining compatibility with Test is not important to us; we haven't been compatible for some time. Is the status above current Rafael, i.e. the undocumented/experimental static_include=true argument to @testset is the only option for this use case at the moment, and the API may change going forward? Thanks and best wishes Rafael! Hope you're doing well! :)

[Sacha0]

cc @caseykneale :)

[rfourquet]

Hi @Sacha0 , nice to see you here :) Yes the status above is current. I don't think static_include is going anywhere, but give me a chance to revisit this issue to try to come up with a better API, I will try to slowly resume my work on ReTest in the upcoming weeks. Hope you are doing well! Best wishes

[Sacha0]

Cheers, thanks Rafael! Please let us know if we can help out somehow :).