From the system infra’s perspective, I would recommend to use BYOC for any domain-specific accelerators.
From the implementation’s perspective, this would depend on your accelerator compiler. In short, if your compiler is based on DNN operators (e.g., conv2d, dense, etc), then BYOC is naturally fit. If your compiler is based on program statements (e.g, loop, if, etc), then it should be easier to be another TVM target, although we can discuss more to see if we can make this case work with BYOC as well.
For AutoTVM, all BYOC use cases we have met so far only need Relay IR with operator names and attributes for their tool chain to perform in-house optimization. In other words, their compilers will schedule operators by themselves so they do not need AutoTVM to do so.
Could you elaborate more about how would you expect to use AutoTVM in BYOC to support your accelerator?