Error: HIP should dso exportable or binary serializable

yogeesh · August 9, 2023, 10:10am

I wrote a relay function containing just one operation (subratract).

    input1 = relay.var("i1" , shape = (N,) , dtype = "float16")
    input2 = relay.var("i2" , shape = (N,) , dtype = "float16")
    sub_out = relay.subtract(input1 , input2)
    func = relay.Function([input1 , input2] , sub_out)
    mod = tvm.IRModule.from_expr(func)
    mod = relay.transform.InferType()(mod)

and used

with tvm.transform.PassContext(opt_level=2):
        graph, lib, params = relay.build(mod, target="rocm", params=None)

to build with “ROCM” as target. Till here i was able to run the code.
But when i export the library using

lib.export_library("lib.so", fcompile=False)

i get following error:

Traceback (most recent call last):
  File "tvm_half_test.py", line 33, in <module>
    lib.export_library("lib.so", fcompile=False)
  File "/home/user/pim-workspace/rib_tvm/tvm/python/tvm/runtime/module.py", line 503, in export_library
    modules = self._collect_dso_modules()
  File "/home/user/pim-workspace/rib_tvm/tvm/python/tvm/runtime/module.py", line 440, in _collect_dso_modules
    return self._collect_from_import_tree(lambda m: m.is_dso_exportable)
  File "/home/user/pim-workspace/rib_tvm/tvm/python/tvm/runtime/module.py", line 429, in _collect_from_import_tree
    ), f"Module {module.type_key} should be either dso exportable or binary serializable."
AssertionError: Module hip should be either dso exportable or binary serializable.

Could anyone please point me right direction for this issue?
thanks

junrushao · August 9, 2023, 9:36pm

CC @sunggg if you wanted to take a look

sunggg · August 10, 2023, 12:13am

Hi, @yogeesh. That seems odd to me since ROCMModuleNode is set as binary serializable correctly.

github.com

apache/tvm/blob/main/src/runtime/rocm/rocm_module.cc#L66


~ROCMModuleNode() {
  for (size_t i = 0; i < module_.size(); ++i) {
    if (module_[i] != nullptr) {
      ROCM_CALL(hipSetDevice(static_cast<int>(i)));
      ROCM_DRIVER_CALL(hipModuleUnload(module_[i]));
    }
  }
}


const char* type_key() const final { return "hip"; }
int GetPropertyMask() const final {
  return ModulePropertyMask::kBinarySerializable | ModulePropertyMask::kRunnable;
}
PackedFunc GetFunction(const String& name, const ObjectPtr<Object>& sptr_to_self) final;


void SaveToFile(const String& file_name, const String& format) final {
  std::string fmt = GetFileFormat(file_name, format);
  std::string meta_file = GetMetaFilePath(file_name);
  // note: llvm and asm formats are not laodable, so we don't save them
  ICHECK_EQ(fmt, fmt_) << "Can only save to format=" << fmt_;
  SaveMetaDataToFile(meta_file, fmap_);

Would you check if you are failing at ROCMModuleNode and how their module property is set?

github.com

apache/tvm/blob/main/python/tvm/runtime/module.py#L255-L288


def is_binary_serializable(self):
    """Returns true if module is 'binary serializable', ie can be serialzed into binary
     stream and loaded back to the runtime module.


    Returns
    -------
    b : Bool
        True if the module is binary serializable.
    """
    return (self.get_property_mask() & ModulePropertyMask.BINARY_SERIALIZABLE) != 0


@property
def is_runnable(self):
    """Returns true if module is 'runnable'. ie can be executed without any extra
    compilation/linking steps.


    Returns
    -------
    b : Bool
        True if the module is runnable.

This file has been truncated. show original

I don’t have AMD machine so I couldn’t reach where you are at unfortunately. Build failed because of missing rocm-related dependency.

  File "/home/ubuntu/tvm/python/tvm/contrib/rocm.py", line 171, in callback_rocm_bitcode_path
    raise RuntimeError("could not find bitcode " + n)
RuntimeError: could not find bitcode oclc_daz_opt_on

tqchen · August 10, 2023, 1:59am

I think it was patched as of last week, so maybe a matter of updating the source to latets

yogeesh · August 10, 2023, 6:09am

hi @junrushao , @sunggg i was able to rectify the issue by returning the property mask for rocm asdso_exportable. I was wondering if it was actually missing in tvm , but as @tqchen mentioned it has been fixed, i will pull the latest. thanks

tqchen · August 10, 2023, 1:07pm

Thank you @yogeesh . glad it is resolved