Adding notes from a few video chats, so that there is a record of the discussion
From @tkonolige , confirmed that the current implementation of @tvm.testing.parametrize_targets shows skipped targets if they are explicitly listed in the decorator, but not if they come from TVM_TEST_TARGETS environment variable.
From @tkonolige , some of the unit tests have a significant amount of setup required before the loop over enabled_targets(), and repeating the setup for many targets would increase the runtime of the tests. I agree, this definitely would be an issue and should have a recommended style to avoid duplicating work. I think the best style would be to use xunit-style setup_class to perform the setup, then have the test methods within the class implemented using parametrized fixtures. I’ll test out a few options and get back on it.
From @areusch , recommended using xfail instead of skip for tests marked as known failing. I agree, and will make that change to the PR.
From @areusch , recommended removing the global variables of tvm_excluded_targets and tvm_known_failing_targets, having the decorator as the only method by which to apply these traits. The global variables can silently have a typo, whereas a typo in a decorator throws an error. I agree, and will make that change to the PR.
From @areusch , though perhaps out of scope of this particular RFC, it would be good to have some discussion as to which unit tests should be required to have parametrized targets, and which are allowed to be target-specific (e.g. where should changes in PRs be requested for having non-parametrized targets). My first thoughts would be to rearrange the current tvm/tests/python folder to mimic the current organization of the top-level src and python directories, rather than the current division into unit/integration tests. Then, the tests that are specific to the target subdirectory and the target-specific subdirectories in runtime would be allowed to have non-parametrized tests, while all others would be parametrized.
Someone mentioned that @masahi was looking into ways to compare cuda/vulkan correctness across all topi models, and that this might be of interest to him. The topi tests were exactly the ones I had in mind as a starting point for converting over to the parametrized targets, for exactly the same reason.