Loading Quantized PyTorch model using relay failing

Troubleshooting
#1

I have a quantized Transformer PyTorch model that I’m trying to load using relay, but I’m getting an obscure error when doing so. Here’s a code sample:

scripted_model = torch.jit.trace(model, input_data).eval()

input_shape = [1, 128] input_data = torch.randint(1, 100, input_shape) input_name = “input0”

shape_list = [(input_name, input_shape)]

mod, params = relay.frontend.from_pytorch(scripted_model, shape_list)

This is the error I’m getting:

Traceback (most recent call last): 6: TVMFuncCall 5: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::transform::Pass, tvm::IRModule)>::AssignTypedLambda<tvm::transform::{lambda(tvm::transform::Pass, tvm::IRModule)#7}>(tvm::transform::{lambda(tvm::transform::Pass, tvm::IRModule)#7}, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) 4: tvm::transform::Pass::operator()(tvm::IRModule) const 3: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const 2: tvm::transform::ModulePassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const 1: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::InferType()::{lambda(tvm::IRModule, tvm::transform::PassContext const&)#1}>(tvm::relay::transform::InferType()::{lambda(tvm::IRModule, tvm::transform::PassContext const&)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) 0: tvm::DiagnosticContext::Render() File “tvm/src/ir/diagnostic.cc”, line 105

#2

Do you have more error message?

If you can share your model, I can look into this.

#3

Amazing thanks @masahi. Unfortunately that’s the entire error message I get. You can find the PyTorch model here: isakbos/Q8BERT_COLA_L_512 at main

Let me know if I can provide code to load the PyTorch model.

#4

Oh if this is really a quantized BERT in pytorch I’d be very interested.

Yes, please show us your complete code to reproduce this error.

#5

Sorry for the delay here @masahi. Please see the entire code sample below to load the model

import torch

from transformers import ( AutoConfig, AutoModelForSequenceClassification, )

import transformers config = AutoConfig.from_pretrained( ‘path_to_model_dir’, num_labels=2, finetuning_task=“cola” )

input_shape = [1, 128]

input_data = torch.randint(1, 100, input_shape)

scripted_model = torch.jit.trace(model, input_data).eval()

input_name = “input0”

shape_list = [(input_name, input_shape)]

mod, params = relay.frontend.from_pytorch(scripted_model, shape_list)

#6

@masahi following up one more time. Would appreciate if you can take a look. Thanks

#7

Please check your code. It doesn’t run, but after I made some changes to make it run, there was no error.