Disabling test cases for functionality configured out

mjs · October 11, 2019, 10:23am

Hi. Do we have a mechanism to disable python unit tests for functionality that has been configured OFF?

For example with USE_VM_PROFILER configured OFF the test test_runtime_vm_profiler.py still runs results in a failure.

I don’t see an obvious mechanism in place to disable such tests based on the USE_ flags… ?

thierry · October 11, 2019, 6:03pm

That’s a good question. Assuming that you’re referring to the Jenkins CI tests: the Jenkinsfile sets the build parameters (which flags are turned on or off) and the sequence of unit tests that are run for each build:

github.com

dmlc/tvm/blob/master/Jenkinsfile#L176-L202


'BUILD: CPU': {
  node('CPU') {
    ws('workspace/tvm/build-cpu') {
      init_git()
      sh """
         mkdir -p build
         cd build
         cp ../cmake/config.cmake .
         echo set\\(USE_SORT ON\\) >> config.cmake
         echo set\\(USE_MICRO ON\\) >> config.cmake
         echo set\\(USE_GRAPH_RUNTIME_DEBUG ON\\) >> config.cmake
         echo set\\(USE_VM_PROFILER ON\\) >> config.cmake
         echo set\\(USE_LLVM llvm-config-8\\) >> config.cmake
         echo set\\(USE_NNPACK ON\\) >> config.cmake
         echo set\\(NNPACK_PATH /NNPACK/build/\\) >> config.cmake
         echo set\\(USE_ANTLR ON\\) >> config.cmake
         echo set\\(CMAKE_CXX_COMPILER g++\\) >> config.cmake
         echo set\\(CMAKE_CXX_FLAGS -Werror\\) >> config.cmake
         echo set\\(HIDE_PRIVATE_SYMBOLS ON\\) >> config.cmake
         """

This file has been truncated. show original

Therefore, we should make sure that test_runtime_vm_profiler.py is only run when the USE_VM_PROFILER flag is enabled. If that’s not the case, I suggest that we create a new .sh test script for builds that enable the profiler.

tqchen · October 11, 2019, 6:08pm

We tried to encourage the writing test case though to be agnostic of availability of a certain features. Based on runtime checks(e.g. if cuda is not available skip cuda test).

In the case of VM profiler, because we see there is less dependencies needed to enable this feature, the testcase was written in a way that is not feature agnostic.

ramana-arm · October 12, 2019, 12:22am

@tqchen

Do we have helpers or tests to detect hardware run time features ?

We are going to have tests in the testsuite which expect particular instructions in cpus that are not necessarily tvm build time detectable. However the tests need to be able to run in the presence and get skipped in the absence of actual hardware.

regards
Ramana

tqchen · October 14, 2019, 5:05am

We have functions fo detecting cuda features, but as far as i know perhaps we lack cpu feature detection.