Android Deploy with new .so format

#1

Hi, I’m working with the android_deploy sample app that uses the three artifact version of a compiled model (.so library, .json graph, .params weights) and the code to load this is available here. Attaching a minimal snippet below:

// load json graph
String modelGraph = null;
String graphFilename = 'modelGraph.json';
modelGraph = new String(getBytesFromFile(assetManager, graphFilename));

// upload tvm compiled function on application cache folder
String libCacheFilePath = null;
String libFilename = 'modelLib.so';
libCacheFilePath = getTempLibFilePath(libFilename);
byte[] modelLibByte = getBytesFromFile(assetManager, libFilename);
FileOutputStream fos = new FileOutputStream(libCacheFilePath);
fos.write(modelLibByte);
fos.close();

// load parameters
byte[] modelParams = null;
String paramFilename = 'modelParams.params';
modelParams = getBytesFromFile(assetManager, paramFilename);

// create java tvm device
Device tvmDev = Device.cpu();

// tvm module for compiled functions
Module modelLib = Module.load(libCacheFilePath);

// get global function module for graph executor
Function runtimeCreFun = Function.getFunction("tvm.graph_executor.create");
TVMValue runtimeCreFunRes = runtimeCreFun.pushArg(modelGraph)
        .pushArg(modelLib)
        .pushArg(tvmDev.deviceType)
        .pushArg(tvmDev.deviceId)
        .invoke();
graphExecutorModule = runtimeCreFunRes.asModule();

// get the function from the module(load parameters)
Function loadParamFunc = graphExecutorModule.getFunction("load_params");
loadParamFunc.pushArg(modelParams).invoke();

// release tvm local variables
modelLib.release();
loadParamFunc.release();
runtimeCreFun.release();

TVM is now deprecating this output structure in favor of a single .so file which contains the graph and weights structures that can be loaded in python as follows:

lib = tvm.runtime.load_module(os.path.join(base, "modelLibrary.so"))
module = tvm.contrib.graph_executor.GraphModule(lib['default'](tvm.cpu()))

Is there an example available to load this kind of combined binary in Android?

#2

you can use this code snipped:

    // depending on  target device wich was used for compiled model
    DLDevice ctx;
    if (device == "CPU") {
        ctx = {kDLCPU, 0};
    } else if (device == "Metal") {
        ctx = {kDLMetal, 0};
    } else if (device == "OpenCL") {
        ctx = {kDLOpenCL, 0};
    } else {    // etc for other target devices
        ...        
    }

    tvm::runtime::Module mod_factory = tvm::runtime::Module::LoadFromFile(input_model);
    // create the graph runtime module
    tvm::runtime::Module gmod = mod_factory.GetFunction("default")(ctx);
    tvm::runtime::PackedFunc set_input = gmod.GetFunction("set_input");
    tvm::runtime::PackedFunc get_output = gmod.GetFunction("get_output");
    tvm::runtime::PackedFunc run = gmod.GetFunction("run");

    // in its turn, the context for data should be CPU only so far:
     DLDevice ctxCPU{kDLCPU, 0};
    tvm::runtime::NDArray x = tvm::runtime::NDArray::Empty({1, 3, 224, 224}, DLDataType{kDLFloat, 32, 1}, ctxCPU);
    // filing of the input to x->data
    // loading data to model
    set_input(0, x);

    // run inference
    run();

    // get ouptut
    tvm::runtime::NDArray y = tvm::runtime::NDArray::Empty({1, 1000}, DLDataType{kDLFloat, 32, 1}, ctxCPU);
    tvm::runtime::NDArray y2 = get_output(0);
    y.CopyFrom(y2);
#3

Thanks for sharing this.

I updated my code to the following:

String libCacheFilePath = null;
String libFilename = MODEL_CPU_LIB_FILE;
try {
    libCacheFilePath = getTempLibFilePath(libFilename);
    byte[] modelLibByte = getBytesFromFile(assetManager, libFilename);
    FileOutputStream fos = new FileOutputStream(libCacheFilePath);
    fos.write(modelLibByte);
    fos.close();
} catch (IOException e) {
    LOGGER.e("Problem uploading compiled function!" + e);
}

// tvm module for compiled functions
Module modelLib = Module.load(libCacheFilePath);
// get global function module for graph executor
graphExecutorModule = modelLib.getFunction("default")
                              .pushArg(Device.cpu().toString())
                              .invoke().asModule();

I get the following error:

error: no suitable method found for pushArg(org.apache.tvm.Device)
        graphExecutorModule = modelLib.getFunction("default").pushArg(tvmDev).invoke().asModule();
                                                             ^
    method org.apache.tvm.Function.pushArg(int) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to int)
    method org.apache.tvm.Function.pushArg(long) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to long)
    method org.apache.tvm.Function.pushArg(float) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to float)
    method org.apache.tvm.Function.pushArg(double) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to double)
    method org.apache.tvm.Function.pushArg(java.lang.String) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to java.lang.String)
    method org.apache.tvm.Function.pushArg(org.apache.tvm.NDArrayBase) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.NDArrayBase)
    method org.apache.tvm.Function.pushArg(org.apache.tvm.Module) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.Module)
    method org.apache.tvm.Function.pushArg(org.apache.tvm.Function) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to org.apache.tvm.Function)
    method org.apache.tvm.Function.pushArg(byte[]) is not applicable
      (argument mismatch; org.apache.tvm.Device cannot be converted to byte[])

Should I pass the kDLCPU value 1 to the argument?

#4

I also tried passing an integer to the function (1, which is the kDLCPU value), and it throws: Check failed: type_code_ == kDLDevice (0 vs. 6) : expected DLDevice but got int

#5

Ups, I thought about c/c++ flow. I don’t know if there are java bindings for my above code. There might be a suggestion to create your own c library, call TVM from it like I proposed above and call your functions through JNI, but it might not be convenient.

#6

Hi, I am trying to use the same demo and I put these three files of mobilenet in asset folder. But it shows that I could not load the .so file. Could you please tell me why this happens. Here is the code that generates these three files.

model = models.MobileNetV2(width_mult=1)
        model.eval()
        input_name = "input_1"
        shape_list = [(input_name,(1,3,224,224))]
        img = torch.rand([1,3,224,224])
        model = torch.jit.trace(model,img)
        shape_list = [(input_name,(1,3,224,224))]
        mod, params = relay.frontend.from_pytorch(model, shape_list)
        return (mod, params)

and

print("dumping lib...")
    lib.export_library(output_path_str + "/" + "deploy_lib_cpu.so", ndk.create_shared)
    print("dumping graph...")
    with open(output_path_str + "/" + "deploy_graph.json", "w") as f:
        f.write(graph)
    print("dumping params...")
    with open(output_path_str + "/" + "deploy_param.params", "wb") as f:
        f.write(tvm.runtime.save_param_dict(params))