I found that I can simply use DLPACK to convert tvm tensor to pytorch tensor and vice versa. The problem has been solved!
from torch.utils.dlpack import to_dlpack, from_dlpack
from tvm.runtime import ndarray
...
m.set_input('input1', ndarray.from_dlpack(to_dlpack(input1)))
...
pytorch_output = from_dlpack(tvm_output.to_dlpack())