TVM runtime unit test infrastructure

Development
#1

While TVM supports a unit test infrastructure backed by gtest for compiler constructs, there is no such unit test infrastructure for the TVM runtime.

This leaves TVM runtime developers with poor choices when it comes to testing. In most cases, developers rely on integration tests using pytest. The downside of this approach is the need for a “full stack” (e.g. codegen) support prior to testing and difficulty in testing corner cases in the runtime code.

This discussion topic proposes a TVM runtime unit test infrastructure backed by gtest. The idea is to use a packed function (rather that gtest_main) to execute all registered tests as follows:

src/runtime/my_runtime/tests/my_runtime_tests.cc

#include "gtest/gtest.h"

TEST(MyRuntimeClass, answer_to_everything) {
  MyRuntimeClass x;
  ASSERT_EQ(x.GetAnswer(), 42);
}

TVM_REGISTER_GLOBAL("MyRuntime.run_all_tests").set_body([](TVMArgs args, TVMRetValue* rv) {
  *rv = RUN_ALL_TESTS();
});

This packed function could be executed through device specific runtime using pytest as follows, using Hexagon as an example:

@requires_hexagon_toolchain
def test_remote_call(hexagon_session):
    func = hexagon_session._rpc.get_function("hexagon.run_all_tests")
    func()

This change will require linking the TVM runtime to gtest, or similar solution.

4 Likes
#2

Thanks @adstraw!

And just underlining that this should also allow for unit testing remote runtimes over RPC. Hexagon is a reasonable example to specify given that there are multiple layers of indirection required to get to the remote hardware:

PC (x86) → Android (Arm) → Hexagon

with RPC channels existing between each layer.

1 Like
#3

It should also be noted that the gtest library should only optionally be linked in to the TVM runtime conditionally given user specified build options to avoid bloat when the testing environment is not needed.

1 Like
#4

It would definitely be useful to define a retargettable test strategy for libtvm_runtime.a. Let’s assume it’s the .a for a couple of reasons:

  1. We may not always link into .so.
  2. Sometimes in C++ unittesting we prefer to take a whitebox approach e.g. reference unexported (non-TVM_DLL) symbols.

The RPC mechanism makes sense based on how we’ve built out the Hexagon infrastructure, but it might not always make sense. I think your solution addresses this by adopting a two-part approach:

  1. Implementing gtest on the target system.
  2. Adding a PackedFunc to invoke a gtest test suite over RPC.

Just wanted to start by making that explicit–it could be there is a system where we could not easily define an RPC server.

The next question in my mind is whether we can compile GTest/GMock for a non-POSIX system. If memory serves, the answer is yes, but we may need to make some hacks. Want to chase down this answer first before we decide how to proceed?

Assuming that’s possible, I favor adding a libtvm_runtime_gtest.a target which links those objects all together. But let’s see how the gtest question looks first.

#5

I was able to create a proof-of-concept for this RFC specific to Hexagon:

With this PR I am able to invoke gtest based unit tests for HexagonBuffer class and run them on device. These unit tests are invoked using python frontend to call through the existing layer of RPC. Runs on simulation or on hardware depending on configuration. Runs in CI on simulation.

NOTE: The HexagonBuffer unit tests had previously run using the existing gtest solution (cpptest) and separate implementation for x86.

The PR still needs some work, even for Hexagon. Not sure whether / when I will publish. Posting here as proof-of-concept as mentioned above.

#6

The Hexagon on-device unit testing PR has landed:

This discuss forum calls for a general solution for on-device unit testing. What has landed is Hexagon specific. We plan to learn from and iterate on the Hexagon solution and grow it into a general solution that works across multiple devices.