Confusing error "CUDA_ERROR_INVALID_PTX"

#1

I have two computer.
Computer A has two 24G memory nvidia gpu (RTX 6000).
Computer B only has a 2000M memory nvidia gpu (MX330).
The two computer all use same cuda-9.0 , same tvm version, same gcc version, same system ubuntu18.04 and same code.

There are two scene,
scene 1,
I export a tvm model on computer A.
Then i run the model on computer B.
TVM report error “Check failed: ret == 0 (-1 vs. 0) : CUDAError: cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX”.

scene2,
I export a tvm model on computer A.
Then i run the model on computer B.
It runs correctly.

So, what is the reason about the above error?
It is about gpu memory ?

1 Like
#2

INVALID_PTX could be many reasons, although running out of shared memory should be the most common one (note that it’s not about the global memory).

For your issue, I didn’t see the difference between two scenarios? Also did you try to export and run the model on the same GPU? Did the problem persist?

#3

Thank you for your reply.
I export and run the model on the same GPU( on computer A and B individually), it runs correctly.

And I have a computer C (aarch64 board with GTX 1060). It can run the model exported from computer b correctly. When running the model exported from computer A, it reports the same error “CUDA_ERROR_INVALID_PTX”.

The GPU memory size relationship is A (RTX 6000) > C (GTX 1060) >B (MX330).

#4

Well it’s not guaranteed the generated PTX can work on another GPU. If you want to do cross-compile, you can try to set the target GPU architecture during the compilation, such as

tvm.target.Target("cuda -arch=sm_75")
#5

I tried to set “target=tvm.target.Target(“cuda -arch=sm_75”)”, it still report above error.
It is unexpected that TVM PTX generated from one GPU can not work on another GPU.

#6

Any updates here ? I got the same error in a single cuda 11.0 + titanv environment.

1 Like
#7

Hello,I have same problem with you,have you sovled it?

1 Like
#8

Please take a look at Nvcc fatal : Value 'sm_86' is not defined for option 'gpu-architecture' - #5 by nicklhy and see if that helps.