Build Error with tensorized call_packed in AutoTVM

dsr91 · February 20, 2019, 7:48am

Hi there,

I ran into a build-error in autoTVM, when using call_packed and a custom python function for tensorization.
This is the used python function:

    @tvm.register_func("tvm.contrib.gemv_update")
    def gemv_update(c, a, b, m, l, stride):
        C = c.asnumpy()
        B = b.asnumpy()
        A = a.asnumpy()
        C = A.dot(B.T)
        c.copyfrom(C)
        return 0

and changed the call from tensorize.py to :

ib.emit(tvm.call_packed("tvm.contrib.gemv_update",cc,aa,bb,m,l,bb.strides[0])

Without AutoTVM, this approach builds and runs successfully.

However, If I use this code in AutoTVM (I added the code to tune_simple_template.py ) , the following error appears when running the tuner:

(RuntimeError('Except caught from RPC call: TVMCall CFunc Error:
    Traceback (most recent call last):  File "/home/rid2rng/tvm/tvm/python/tvm/_ffi/_ctypes/function.py", line 55, in cfun
    rv = local_pyfunc(*pyargs)
    File "/home/rid2rng/tvm/tvm/python/tvm/rpc/server.py", line 50, in load_module
        m = _load_module(path)
        File "/home/rid2rng/tvm/tvm/python/tvm/module.py", line 241, in load
            _cc.create_shared(path + ".so", files)
              File "/home/rid2rng/tvm/tvm/python/tvm/contrib/cc.py", line 33, in create_shared
                  _linux_shared(output, objects, options, cc)
                    File "/home/rid2rng/tvm/tvm/python/tvm/contrib/cc.py", line 58, in _linux_shared
                        raise RuntimeError(msg)\nRuntimeError: Compilation error:
                        /usr/bin/ld: /tmp/tmpt5frypyt/lib.o: relocation R_X86_64_32S against `.rodata.cst16\' can not be used when making a shared object; recompile with -fPIC
                        /usr/bin/ld: final link failed: Nonrepresentable section on output\ncollect2: error: ld returned 1 exit status\n\n',),), error_no=4,

The error Message appears in the AutoTVM tuning log print, in MeasureResults(costs=,…)
It looks like the object file used by LocalBuild in AutoTVM isn’t built with the -fPIC flag.
Is this fixable by adding the -fPIC flag to build of the object? If so, where would I have to do this?

eqy · February 20, 2019, 11:53pm

You might be able to modify the AutoTVM build function to see if a change there can support your needs (e.g., compare it with building without AutoTVM to see what changes) https://github.com/dmlc/tvm/blob/master/python/tvm/autotvm/measure/measure_methods.py#L352. Have you configured some different build options when building without AutoTVM?

dsr91 · February 21, 2019, 8:45am

The build options with and without AutoTVM are the same and the target is llvm for both builds. I also realized that embedding the “tensorization.py” tutorial into “tune_simple_template.py” will also fail, resulting in exactly the same error message. Is there a verified example of combining AutoTVM and and tensorize?

eqy · February 21, 2019, 7:33pm

I don’t know of any examples of combining tensorization and AutoTVM in the past (though it may be used in VTA without any documentation). However, this is an interesting problem and a use case that we should cover. Currently I am wondering if this issue is a result of AutoTVM itself or an artifact of some other part of the build process e.g., how we export/import modules.

Concretely, one way to debug this is to just compare the outputs commands of cc.py to check if there is a mismatch. Can you share what these look like to rule out the issue being a different command being issued here?

dsr91 · February 22, 2019, 7:44am

I compared the outputs of cc.py for tune_simple_template.py and my version with tensorization.
Original:

['g++', '-shared', '-fPIC', '-o', '/tmp/tmp8dv4bqeo/tmp_func_a42e465cef1a9a85.tar.so', '/tmp/tmpza7pyr3_/lib.o']

tensorized:

['g++', '-shared', '-fPIC', '-o', '/tmp/tmpmnkpdbs2/tmp_func_617b44656cfdac39.tar.so', '/tmp/tmp83ihok0m/lib.o']

There is no visible difference. Corresponding to the error message from the initial post, the problem seems to be the missing -fPIC flag while generating /tmp/tmp83ihok0m/lib.o. However, the generation of this lib.o seems to happen somewhere in the C++ part of TVM.

I looked through tvm/codegen, but I couldn’t pinpoint the place where exactly I could add the flag to build command. Could you hint me to right place, @eqy?

eqy · February 22, 2019, 9:13pm

I do not have much experience with this stage of the build process, but my guess is that the llvm build happens somewhere around here: https://github.com/dmlc/tvm/blob/master/src/codegen/llvm/llvm_module.cc

dsr91 · February 25, 2019, 7:05am

Thanks for the advice, appreciate it! @tqchen, could you weigh in on this issues? According to git, you contributed a major of the build process. It would be enough to know the location where the flags for the llvm build are defined.

denis0x0D · February 25, 2019, 8:38am

Hi @dsr91, I’m not familiar with auto tvm part, but the relocations defines while creating a llvm::TargetMachine

github.com

dmlc/tvm/blob/master/src/codegen/llvm/llvm_common.cc#L146


}


std::string err;
const llvm::Target* target =
    llvm::TargetRegistry::lookupTarget(target_triple, err);
if (target == nullptr) {
  CHECK(allow_null) << err << " target_triple=" << target_triple;
  return nullptr;
}
llvm::TargetMachine* tm = target->createTargetMachine(
    target_triple, mcpu, mattr, opt, llvm::Reloc::PIC_);
return std::unique_ptr<llvm::TargetMachine>(tm);
}


}  // namespace codegen
}  // namespace tvm
#endif  // TVM_LLVM_VERSION

hope, it might be helpful.