masahi
January 18, 2021, 9:00am
7
@comaniac @FrozenGene @merrymercy
I figured out what’s going on. The issue is random_fill function used by AutoTVM:
costs = (MAX_FLOAT,)
error_no = MeasureErrorNo.COMPILE_DEVICE
error_msg = make_traceback_info()
if error_no == 0:
try:
args = [ndarray.empty(get_const_tuple(x.shape), x.dtype, ctx) for x in build_res.args]
random_fill = tvm.get_global_func("tvm.contrib.random.random_fill", True)
assert random_fill, "Please make sure USE_RANDOM is ON in the config.cmake"
for arg in args:
random_fill(arg)
ctx.sync()
costs = time_f(*args).results
# pylint: disable=broad-except
except Exception:
costs = (MAX_FLOAT,)
error_no = MeasureErrorNo.RUNTIME_DEVICE
error_msg = make_traceback_info()
shutil.rmtree(os.path.dirname(build_res.filename))
toc = time.time()
I’m trying to tune scatter op, which has an integer input as one of inputs. This poses two problems for random_fill.
random_fill implementation is busted if the input is 32 or 64 bit integer tensor, see
} else if (tensor->dtype.bits == 32) {
std::generate_n(static_cast<float*>(tensor->data), size, [&]() { return dist(rnd_engine_); });
} else if (tensor->dtype.bits == 64) {
std::generate_n(static_cast<double*>(tensor->data), size,
[&]() { return dist(rnd_engine_); });
As a result of this, the index tensor is filled with values like
array([1091617754, 1090832347, 1091032493, ..., 1084215127, 1073655395,
1090813385], dtype=int32
The value of index tensor cannot be any random values, since the entry is the index into the scattered tensor. In our relay test for example, the index tensor is initialized as follows:
def verify_scatter(dshape, ishape, axis=0):
d = relay.var("d", relay.TensorType(dshape, "float32"))
i = relay.var("i", relay.TensorType(ishape, "int64"))
u = relay.var("u", relay.TensorType(ishape, "float32"))
z = relay.op.scatter(d, i, u, axis)
func = relay.Function([d, i, u], z)
data_np = np.random.uniform(size=dshape).astype("float32")
updates_np = np.random.uniform(size=ishape).astype("float32")
indices_np = np.random.randint(-dshape[axis], dshape[axis] - 1, ishape).astype("int64")
ref_res = ref_scatter(data_np, indices_np, updates_np, axis)
for target, ctx in tvm.testing.enabled_targets():
for kind in ["graph", "debug"]:
intrp = relay.create_executor(kind, ctx=ctx, target=target)
op_res = intrp.evaluate(func)(data_np, indices_np, updates_np)
tvm.testing.assert_allclose(op_res.asnumpy(), ref_res, rtol=1e-5)
def verify_dynamic_scatter(dshape, ishape, axis=0):
How should we solve this issue? I think I’ll simply skip the random init when the workload is from scatter op.
2 Likes