Hi, I’m working with the android_deploy sample app that uses the three artifact version of a compiled model (.so library, .json graph, .params weights) and the code to load this is available here. Attaching a minimal snippet below:
// load json graph
String modelGraph = null;
String graphFilename = 'modelGraph.json';
modelGraph = new String(getBytesFromFile(assetManager, graphFilename));
// upload tvm compiled function on application cache folder
String libCacheFilePath = null;
String libFilename = 'modelLib.so';
libCacheFilePath = getTempLibFilePath(libFilename);
byte[] modelLibByte = getBytesFromFile(assetManager, libFilename);
FileOutputStream fos = new FileOutputStream(libCacheFilePath);
fos.write(modelLibByte);
fos.close();
// load parameters
byte[] modelParams = null;
String paramFilename = 'modelParams.params';
modelParams = getBytesFromFile(assetManager, paramFilename);
// create java tvm device
Device tvmDev = Device.cpu();
// tvm module for compiled functions
Module modelLib = Module.load(libCacheFilePath);
// get global function module for graph executor
Function runtimeCreFun = Function.getFunction("tvm.graph_executor.create");
TVMValue runtimeCreFunRes = runtimeCreFun.pushArg(modelGraph)
.pushArg(modelLib)
.pushArg(tvmDev.deviceType)
.pushArg(tvmDev.deviceId)
.invoke();
graphExecutorModule = runtimeCreFunRes.asModule();
// get the function from the module(load parameters)
Function loadParamFunc = graphExecutorModule.getFunction("load_params");
loadParamFunc.pushArg(modelParams).invoke();
// release tvm local variables
modelLib.release();
loadParamFunc.release();
runtimeCreFun.release();
TVM is now deprecating this output structure in favor of a single .so file which contains the graph and weights structures that can be loaded in python as follows:
lib = tvm.runtime.load_module(os.path.join(base, "modelLibrary.so"))
module = tvm.contrib.graph_executor.GraphModule(lib['default'](tvm.cpu()))
Is there an example available to load this kind of combined binary in Android?
you can use this code snipped:
// depending on target device wich was used for compiled model
DLDevice ctx;
if (device == "CPU") {
ctx = {kDLCPU, 0};
} else if (device == "Metal") {
ctx = {kDLMetal, 0};
} else if (device == "OpenCL") {
ctx = {kDLOpenCL, 0};
} else { // etc for other target devices
...
}
tvm::runtime::Module mod_factory = tvm::runtime::Module::LoadFromFile(input_model);
// create the graph runtime module
tvm::runtime::Module gmod = mod_factory.GetFunction("default")(ctx);
tvm::runtime::PackedFunc set_input = gmod.GetFunction("set_input");
tvm::runtime::PackedFunc get_output = gmod.GetFunction("get_output");
tvm::runtime::PackedFunc run = gmod.GetFunction("run");
// in its turn, the context for data should be CPU only so far:
DLDevice ctxCPU{kDLCPU, 0};
tvm::runtime::NDArray x = tvm::runtime::NDArray::Empty({1, 3, 224, 224}, DLDataType{kDLFloat, 32, 1}, ctxCPU);
// filing of the input to x->data
// loading data to model
set_input(0, x);
// run inference
run();
// get ouptut
tvm::runtime::NDArray y = tvm::runtime::NDArray::Empty({1, 1000}, DLDataType{kDLFloat, 32, 1}, ctxCPU);
tvm::runtime::NDArray y2 = get_output(0);
y.CopyFrom(y2);
Thanks for sharing this.
I updated my code to the following:
String libCacheFilePath = null;
String libFilename = MODEL_CPU_LIB_FILE;
try {
libCacheFilePath = getTempLibFilePath(libFilename);
byte[] modelLibByte = getBytesFromFile(assetManager, libFilename);
FileOutputStream fos = new FileOutputStream(libCacheFilePath);
fos.write(modelLibByte);
fos.close();
} catch (IOException e) {
LOGGER.e("Problem uploading compiled function!" + e);
}
// tvm module for compiled functions
Module modelLib = Module.load(libCacheFilePath);
// get global function module for graph executor
graphExecutorModule = modelLib.getFunction("default")
.pushArg(Device.cpu().toString())
.invoke().asModule();
I get the following error:
error: no suitable method found for pushArg(org.apache.tvm.Device)
graphExecutorModule = modelLib.getFunction("default").pushArg(tvmDev).invoke().asModule();
^
method org.apache.tvm.Function.pushArg(int) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to int)
method org.apache.tvm.Function.pushArg(long) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to long)
method org.apache.tvm.Function.pushArg(float) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to float)
method org.apache.tvm.Function.pushArg(double) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to double)
method org.apache.tvm.Function.pushArg(java.lang.String) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to java.lang.String)
method org.apache.tvm.Function.pushArg(org.apache.tvm.NDArrayBase) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.NDArrayBase)
method org.apache.tvm.Function.pushArg(org.apache.tvm.Module) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.Module)
method org.apache.tvm.Function.pushArg(org.apache.tvm.Function) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.Function)
method org.apache.tvm.Function.pushArg(byte[]) is not applicable
(argument mismatch; org.apache.tvm.Device cannot be converted to byte[])
Should I pass the kDLCPU value 1 to the argument?
I also tried passing an integer to the function (1, which is the kDLCPU value), and it throws:
Check failed: type_code_ == kDLDevice (0 vs. 6) : expected DLDevice but got int
Ups, I thought about c/c++ flow. I don’t know if there are java bindings for my above code. There might be a suggestion to create your own c library, call TVM from it like I proposed above and call your functions through JNI, but it might not be convenient.
Hi, I am trying to use the same demo and I put these three files of mobilenet in asset folder. But it shows that I could not load the .so file.
Could you please tell me why this happens.
Here is the code that generates these three files.
model = models.MobileNetV2(width_mult=1)
model.eval()
input_name = "input_1"
shape_list = [(input_name,(1,3,224,224))]
img = torch.rand([1,3,224,224])
model = torch.jit.trace(model,img)
shape_list = [(input_name,(1,3,224,224))]
mod, params = relay.frontend.from_pytorch(model, shape_list)
return (mod, params)
and
print("dumping lib...")
lib.export_library(output_path_str + "/" + "deploy_lib_cpu.so", ndk.create_shared)
print("dumping graph...")
with open(output_path_str + "/" + "deploy_graph.json", "w") as f:
f.write(graph)
print("dumping params...")
with open(output_path_str + "/" + "deploy_param.params", "wb") as f:
f.write(tvm.runtime.save_param_dict(params))