Failed to run idflakies on following projects

[byjiang1996]

The latest testrunner(1.1)/iDFlakies(1.1.0) failed to run iDFlakies on Sentinel repo:

Project: https://github.com/alibaba/Sentinel
Branch: master
SHA: 8c4f251922365029a36ca6fe912b8aa88d6fe286
Module/submodule name: sentinel-demo-quarkus

Error logs:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (2 tests executed, 4 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter(), com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.(RandomDetector.java:26)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

[byjiang1996]

In addition, I have minimized the repro steps by modifying .dtfixingtools/original-order file:

If we tested on either com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter() or com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter() function, similar error will occur:

Only test com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter() in original-order file:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.OriginalDetector.(OriginalDetector.java:28)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:23)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

Only test com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter() in original-order file:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.OriginalDetector.(OriginalDetector.java:28)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:23)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

[byjiang1996]

The error message in this issue is same as in #22, but the cause may be different as this test class has a project-specific class annotation, while the other case seems to be about parameterized tests.

FYI, similar to #22, all error tests use Junit5 api.