[Tensor array] Performance of tensor array in tvm is poor than tf

Questions
#1

Hi, sorry to bother u, but does anyone have tested tensor array in tvm? I found that the performance of tensor array in tvm is pool, about 10x+ slowly than tf.

I read the code about tensor array in tvm and found tvm use adt(list) to express tensor array in relay, the operations about tensor array will be converted to some recursive functions such as nth/update, and will generate many packed funcs.

In the meantime, tf use array-like structure to implement tensor array and can read the nth element in O(1) (tvm will use O(n)).

Does anyone have some experiences on this? How could I make tensor array more efficient in tvm? Will be glad if anyone could discuss it.

#2

@wweic @masahi Could u give some advises? :grinning: :blush:

#3

Help cc @junrushao @kevinthesun Do you have any advice on this?

#4

@lgmt Sorry I forgot to answer. I’ve met the exact same issue of super slow tensor array concat in MaskRCNN compared to PyTorch (which has batched concat). The reason is precisely as you described (repeated calls to small concat and lack of constant time random access).

This is a limitation of Relay/TE. We currently don’t have a solution for this problem. We need a first class support for tensor array. @jroesch @tqchen @altanh @electriclilies

#5

Like @masahi said, I don’t think there’s a short term fix for this. In the long term, though (hopefully next 6-8 months), we are working on redesigning parts of Relay, and will be fixing this as part of that work.