Hi everyone,
I was wondering if somebody knows whether there is an auto-tuning template for relay local response normalization “relay.nn.lrn”. If so, do you know where I can find it (folder), and if there is a template for both CPU and GPU?
I appreciate any help you can provide on this issue
I think this is the one: https://github.com/apache/incubator-tvm/blob/7c5ff50873e91e9ad27b5f08847c27d58e8b5c4c/topi/src/topi.cc#L584
Thank you for your help @comaniac. I was able to find the schedule for CUDA in
tvm/topi/include/topi/cuda/normalization.h
This made me wonder why some schedules are given in C++, as the one above, whereas some others are written in Python, as the ones in:
/topi/python/topi/cuda/
Do you know why this is the case? In addition, Do you know of any tutorials/examples on how to create a relay layer including its schedule/schedule template? This could be very useful to interface new modules with the relay API.
AFAIK, it has some historical reasons and concerns. I’d like to call out @masahi for this question 
hey thanks for the ping yeah, this is my favorite topic to rant about. Discussion in the PR [TOPI] Remove cpp upsampling and resize op by masahi · Pull Request #4769 · apache/tvm · GitHub gives some context.
In short, there was a time when people wanted to reduce dependency on Python by porting existing schedules to cpp. But that effort fade away after some time and we are left with half-baked cpp implementation that is duplicate or even worse, inconsistent with py impl.
These days people do not seem to care and only py impl are being updated. Some important op like pooling still use cpp topi impl.
Thank you for providing some context on this issue @masahi. Speaking about the Python implementation, do you know of any tutorials/examples on how to create new relay layers with their schedule templates using Python?
I don’t know if such doc exist. You can refer to impl of other ops similar to lrn.
Keep in mind that Relay and topi are different things, and writing a new relay op needs a lot of boilerplate. Recently we changed how relay and topi ops are connected (look for “strategy” in the code).
Ah in fact, the doc about how to register your TOPI compute/schedule functions to Relay is being reviewed: https://github.com/apache/incubator-tvm/pull/5078
Thanks a lot for this information @comaniac . It would be great to have a document on how to add new operations to relay, this could help many TVM users and also increase the number of contributions. I look forward to the final version of the document