[AutoTVM] Interface for fixed input data

[Tantalus13A98B5F]

Background

Currently AutoTVM uses random input data to measure the generated functions. However, there are circumstances where we want fixed input. For example, when we are tuning a sparse operator (current sparse operators in TOPI are not tunable), fixed indices are required to ensure continued execution and stable access pattern, or we will be failing most cases and get non-progressing tuning results.

Solutions in similar occasions

In AutoScheduler we have the argument task_inputs on SearchTask to specify fixed input.

In AutoTVM, prior to PR-7250 we have the field RPCRunner.ref_input to do the job. Post to PR-7250 we can do monkey-patching on RPCSession.get_function to use an alternative of tvm.contrib.random.random_fill.

Design discussion

We have two options for this interface,

1. A field to specify a fixed input data;
2. A field to specify an alternative input generator (proposed by areusch).

For sparse operators, fixed input data should be preferred; or we may run into problems of measuring different kernels for different cases. Still, there may be some operators that can benefit from varied input. Or else we can make it both, wrapping fixed input with a function or branching on its callable-ness.

[tkonolige]

Ideally we want to use the same code to handle fixed inputs between autotvm, autoscheduler and autotir. Right now only autoscheduler supports fixed inputs. It uses a global registry (see https://github.com/apache/tvm/pull/7635) in which inputs are registered by name. Maybe you can reuse this code to provide fixed inputs for autotvm?

Note that the autoscheduler approach does have some difficulties. It’s hard to figure out where to register the fixed inputs. For sparse its done in the pass that converts dense matrices to sparse (which means you don’t get fixed inputs if you model is already sparse). Also, because inputs are identified by a string, if you have two of the same relay nodes with different inputs, you won’t be able to pass different inputs to each.

If we use fixed inputs (as opposed to the input generator), what do we do if inputs are not supplied? I’d suggest add fallback input generators for functions that need them (sparse, indexing operations).

[comaniac]

In fact, AutoTVM was having a feature to accept a fixed input for correctness checking, but it was already removed: Remove check_correctness in AutoTVM, which is busted by areusch · Pull Request #7250 · apache/tvm · GitHub. Please also be aware of the issues mentioned in the PR.

[areusch]

The main issue with the previous approach is that it wasn’t possible to transform the reference input when a schedule needed layout transformations. I’m okay with bringing something back so long as we have a way to either

1. handle layout transformations to ref_input
2. determine when layout transformations have occurred and not do output checking or warn/fail autotuning when output checking can’t be done

[Tantalus13A98B5F]

Thank you all for your replies.

@tkonolige

It’s hard to figure out where to register the fixed inputs. For sparse its done in the pass that converts dense matrices to sparse (which means you don’t get fixed inputs if you model is already sparse).

Seems that autoscheduler already supports tuning a whole model from dense to sparse. Initially I was considering tuning a single operator, and I’d like to hold on to it for now and come back later with more background. Agreed that we should have a better, unified mechanism for sparse inputs.

@areusch

I’m okay with bringing something back so long as we have a way to either

1. handle layout transformations to ref_input
2. determine when layout transformations have occurred and not do output checking or warn/fail autotuning when output checking can’t be done

I think the real problem is that autotvm does not officially support layout transformation, thus we all go implicitly, and then the correctness check fails. Following R1 might suggests a larger-scale design change.

On the other hand, previously ref_input can be enabled without setting the check_correctness option, and it just works implicitly (the attribute is always submitted to the executor). IMO output checking is not a requirement for this piece of code, it can be safely decoupled, and easily accomplished with external developer efforts.

[areusch]

@Tantalus13A98B5F

IMO output checking is not a requirement for this piece of code, it can be safely decoupled, and easily accomplished with external developer efforts.

I agree with this. Would you like to send a PR to re-enable ref_input with some appropriate documentation warnings about correctness?