[AutoTVM] Exception handling in measure_methods seems incorrect

vinx13 · August 8, 2018, 7:10am

An new exception is raised in a except block. However, this will lead to searching for an exception handler in the outer scope (beyond the entire try-except here). So the exception raised here will not be caught.

github.com

dmlc/tvm/blob/0241fdc5d23a410cff63a0ffeab7c548fadfaa98/python/tvm/autotvm/measure/measure_methods.py#L348


except TVMError as exc:
    tstamp = time.time()
    msg = str(exc)
    if "Stack trace returned" in msg:
        msg = msg[:msg.index("Stack trace returned")]
    if "InstantiationError" in msg:
        try:
            msg = msg.split('\n')[-2].split(": ")[1]
        except Exception:  # pylint: disable=broad-except
            pass
        raise InstantiationError(msg)
    else:
        res_pack.append(MeasureResult((RuntimeError(msg),),
                                      MeasureErrorNo.COMPILE_HOST,
                                      tstamp - tic, tstamp))
    continue
except InstantiationError as e:
    tstamp = time.time()
    res_pack.append(MeasureResult((InstantiationError(str(e)),),
                                  MeasureErrorNo.INSTANTIATION_ERROR,
                                  tstamp - tic, tstamp))

@merrymercy Could you check this?

merrymercy · August 8, 2018, 10:39pm

Good catch. I fixed it in https://github.com/dmlc/tvm/pull/1566.

But it is not a fatal bug since current tuning scripts still work.

vinx13 · August 9, 2018, 7:39am

@merrymercy thanks, however I encountered another problem that exceptions are not caught, which made subprocesses not responding.

github.com

dmlc/tvm/blob/41d4dd6ef16445c4ea7c7835f139d9f30398f027/python/tvm/autotvm/tuner/xgboost_cost_model.py#L273






_extract_space = None
_extract_target = None
_extract_task = None


def _extract_itervar_feature_index(index):
"""extract iteration var feature for an index in extract_space"""
config = _extract_space.get(index)
with _extract_target:
    sch, args = _extract_task.instantiate(config)
fea = feature.get_itervar_feature_flatten(sch, args, take_log=True)
fea = np.concatenate((fea, list(config.get_other_option().values())))
return fea


def _extract_itervar_feature_log(arg):
"""extract iteration var feature for log items"""
inp, res = arg
config = inp.config
with inp.target:
    sch, args = inp.task.instantiate(config)

Traceback attached for your information.
Traceback (most recent call last):
File “at.py”, line 139, in
tuner.tune(n_trial=200, measure_option=measure_option, callbacks=[autotvm.callback.log_to_file(log_name)])
File “tvm/python/tvm/autotvm/tuner/tuner.py”, line 125, in tune
self.update(inputs, results)
File “tvm/python/tvm/autotvm/tuner/model_based_tuner.py”, line 250, in update
self.cost_model.fit(self.xs, self.ys, self.plan_size)
File “tvm/python/tvm/autotvm/tuner/xgboost_cost_model.py”, line 142, in fit
x_train = self._get_feature(xs)
File “tvm/python/tvm/autotvm/tuner/xgboost_cost_model.py”, line 255, in _get_feature
feas = pool.map(self.feature_extract_func, need_extract)
File “/usr/lib/python2.7/multiprocessing/pool.py”, line 251, in map
return self.map_async(func, iterable, chunksize).get()
File “/usr/lib/python2.7/multiprocessing/pool.py”, line 567, in get
raise self._value
tvm._ffi.base.TVMError: [15:22:26] tvm/src/op/tensorize.cc:127: Tensorize failed, split condition likely((((k.inner.inner + (k.inner.outer4)) + (k.outer1)) < 2048)) relies on var defined inside tensorize scope

This happened when some auto-tuned tile size did not meet constraints but I didn’t detect it and explicitly call raise_error. Expected behavior is that AutoTVM catches this exception and continues iterations.

merrymercy · August 10, 2018, 4:16pm

Can you try other tuners that needn’t feature extraction for now ? For example, GATuner(pop_size=100), XGBTuner(feature_type=‘knob’).

There are some constraints on the feature extraction of XGBTuner. I can fix it later.