[BYOC][ACL] Unexpected layout transform op after graph partitioning

Troubleshooting
#1

Hi, I am trying to offload the supported part of a MobileNet-V2 model to Arm Compute Library with partition_for_arm_compute_lib(mod) provided by the ACL BYOC pipeline.

However, it seems when building the partitioned graph with relay.build, the layout transform part required for ACL operators is integrated into the sub-graphs that are annotated with ‘arm_compute_lib’, and these transform ops aren’t folded away as expected.

Since ACL runtime engine can’t recognize this layout_transform op, I get the unsupported op error when trying to run the lib. Any suggestions on resolving this issue? Or is there anything I did wrong?

Thanks!

mod
mod1737×83 30.5 KB
backtrace
backtrace1737×371 125 KB

@lhutton1 @comaniac

#2

Hi @mgeek, it seems as though the constants aren’t bound to the function you are running and are instead being treated similar to a variable input. For this reason constant folding will not work and this is why the layout_transform operator remains. After importing your model from a frontend, but before partitioning for ACL, I would suggest running the bind_params_by_name pass - there is a short example which might help here: Constant params should be constants - #3 by lhutton1. After running this you should see the params appear as Constant[] when printing the Relay module.

I would also recommend taking a look at the user-facing TVMC Python interface, which should automatically take care of these types of issues.

1 Like
#3

The bind_params_by_name works! Thanks!

And I just realized that I can enable this param binding by passing params to the partition_for_arm_compute_lib function.

def partition_for_arm_compute_lib(mod, params=None, **opts):
    if params:
        mod["main"] = bind_params_by_name(mod["main"], params)

    seq = tvm.transform.Sequential(
        [
            transform.InferType(),
            transform.MergeComposite(arm_compute_lib_pattern_table()),
            transform.AnnotateTarget("arm_compute_lib", False),
            transform.PartitionGraph(),
        ]
    )

    return seq(mod)
#4

Ah yes, I should have also mentioned that. Great to hear you got it working!

#5

Many thanks for your help!

Btw, is there any multi-thread support for using ACL in TVM via BYOC? I tried to tweak the num_threads by os.environ[“TVM_NUM_THREADS”], it is effective for TVM-defined operators, but for offloaded ACL operators it makes no difference.

#6

Hi, yes you’re correct that TVM_NUM_THREADS is only effective for TVM-defined operators. It’s been a while since I last looked at the ACL integration, but I believe this is just a case of not having been explored in much detail as of yet

1 Like
#7

I see. Thanks for your help!