[BYOC] Problem about subgraph with TupleTypeNode inputs

Pcheng-L · April 28, 2020, 9:18am

Now, I want to use BYOC to run SSD-ResNet34 model and I met some problems. About the “concatenate” operator, if it is a subgraph, the partitioned graph is:

def @ssdnn_0(%ssdnn_0_i0: (Tensor[(64, 4, 5776), float32], Tensor[(64, 4, 2166), float32], Tensor[(64, 4, 600), float32], Tensor[(64, 4, 150), float32], Tensor[(64, 4, 36), float32], Tensor[(64, 4, 4), float32]), Compiler="ssdnn", Inline=1, Primitive=1, global_symbol="ssdnn_0") -> Tensor[(64, 4, 8732), float32] {
  concatenate(%ssdnn_0_i0, axis=2) /* ty=Tensor[(64, 4, 8732), float32] */
}

After codegen, the cSource code is generated:

extern “C” int ssdnn_0_wrapper_(DLTensor* arg0, DLTensor* arg1) { ssdnn_0_(static_cast<float*>(arg0->data), static_cast<float*>(arg1->data)); return 0; }

extern “C” void ssdnn_0_(float* ssdnn_0_i0, float* out) { … ssdnn_concat(1, buf_0, 0, 2, 3, 64, 4, 8732, 1, ssdnn_0_i0); std::memcpy(out, buf_0->data(), 4 * 2235392); … }

The parameter of “ssdnn_0_i0” is a tupleTypeNode, But I need to know each field’s data after codegen. I wonder if there is any idear to get the each element of TupeTypeNode after codegen, or after graph partition, not input the TupleTypeNode but give each element of the TupleTypeNode? @comaniac @manupa-arm

comaniac · April 28, 2020, 5:47pm

At this moment we suggest your codegen flatting a tuple here:

github.com

apache/incubator-tvm/blob/master/src/relay/backend/contrib/dnnl/codegen.cc#L141


class CodegenDNNL : public MemoizedExprTranslator<std::vector<Output>>, public CodegenCBase {
 public:
  explicit CodegenDNNL(const std::string& id) { this->ext_func_id_ = id; }


  std::vector<Output> VisitExprDefault_(const Object* op) final {
    LOG(FATAL) << "DNNL codegen doesn't support: " << op->GetTypeKey();
    return {};
  }


  std::vector<Output> VisitExpr_(const VarNode* node) final {
    ext_func_args_.push_back(GetRef<Var>(node));
    Output output;
    output.name = node->name_hint();
    return {output};
  }


  std::vector<Output> VisitExpr_(const TupleGetItemNode* op) final {
    auto res = VisitExpr(op->tuple);
    CHECK_GT(res.size(), static_cast<size_t>(op->index));


    // Only keep the item we want for the child node.

In addition, when processing concatenate nodes, your codegen can retrieve the tuple information when processing concatenate nodes by accessing their args.

Apart from that, it’s not recommended offloading concatenate nodes especially this subgraph contains only one concatenate node.

ramana-arm · April 28, 2020, 9:08pm

@mbaret : you may find this interesting.

Pcheng-L · April 29, 2020, 6:16am

I will have a try. Thanks very much.

mbaret · April 29, 2020, 10:26am

Hi, welcome to the forum I’m working on fixing this exact issue at the moment. It comes about because constant tuples are not correctly propagated into the partitioned regions so you can’t see the data of the tuple, only its type. I hope to have a fix in review either later today or tomorrow.

mbaret · April 29, 2020, 2:43pm

I have a candidate fix with this PR: https://github.com/apache/incubator-tvm/pull/5476

comaniac · April 29, 2020, 5:07pm

@mbaret the PR you posted solved the issue of tuple constant propagation, but it seems not solving the tuple var node issue. In this particular case, for example, we will still have a tuple of data in the first argument.