I am developing in Ansor, which tunes and evaluates workloads in a model independently, and can evaluate them standalone using the utilities in measure.py.
I want to compile a single workload in a model using the default schedule, to compare against the time for an auto-scheduled approach.
E.g.:
- Workload 1 untuned time: 1 ms
- Workload 1 tuned time: 0.5 ms
- Workload 2 untuned time: 1.5ms
- Workload 2 tuned time: 1ms
- …
- Full model untuned inference time: 50ms
- Full model tuned inference time: 35ms
Given a tuned log file for a model, I can easily measure the inference time of a single workload with this function (full code linked below):
def test_single_workload(workload_num: int, logfile: os.PathLike) -> float:
"""Measures the time for a single workload with a given autoschedule file in Ansor
:param workload_num: the workload number to evaluate
:param logfile: the path to the logfile containing workload and schedule information
:returns: Inference time of workload
"""
reader = RecordReader(logfile)
inputs, _ = reader.read_lines()
timeout, n_parallel = 3, 1
runs = 5
inputs = [inputs[workload_num] for _ in range(runs)]
results = local_builder_build(inputs, timeout, n_parallel)
total = 0
vs = []
for r in results:
if r.error_no == 0:
# acceptable exit code
vs.append(r.time_cost)
else:
print("Unexpected exit code:", r.error_no)
exit(1)
cost = np.median(vs)
return cost
However, if I want to get the performance of that workload with the default schedule (no logfile), then I am unsure what I need to do.
If I wanted an equivalent function with the untuned task, I can create an input like this:
input = MeasureInput(task, state)
However I need a State object, which describes the schedule we are evaluating, and I do not have a logfile which contains a description of the default schedule for that workload. It cannot be set to None, at least the way I have tried it.
How would I get a State object for this workload in its untuned state?
A simple standalone example of my code is available at this gist.
EDIT: I have found that setting the state to task.compute_dag.get_init_state() gives a valid state. However I wonder if this is the right schedule or not.