Small config bug in autotvm

Development
#1

There is a small bug in ConfigSpace where if __repr__ is invoked before the config has finished collecting knobs / spaces, subsequently collected items will not count towards the size of the whole config space which will result in an incomplete run of autotuning. This is due to the caching compute of the config space here.

The most straightforward fix would be something like this:

if self._length is None or self._collect:
     self._length = int(np.prod([len(x) for x in self.space_map.values()]))
 return self._length
#2

It’s an interesting finding. Could you elaborate a case that __repr__ is invoked before config collection?

On the other hand, your suggested change will make the caching useless, because ConfigSpace._collect is always true. A more safer solution could be like:

def _add_new_transform(self, space_class, name, axes, policy, **kwargs):
    self._length = None
    # do the rest process.

In this way, we “invalid” any existing length when a new knob is added.

Another solution is maintaining the length on the fly:

def __init__(self):
      self._length = 1

def _add_new_transform(self, space_class, name, axes, policy, **kwargs):
      # do the rest process.
     self._length *= len(self._entity_map[name])

def __len__(self):
    return self._length
#3

@comaniac in my case it happened when I was logging the config at the top of a compute / schedule function, before all knobs were defined, which is sometimes useful given the dual collect / apply behavior of config.

Good solution btw, a lot better than my suggestion.

ConfigSpace._collect is always true.

True, but on subsequent calls to the same compute / schedule functions config is a ConfigEntity which has _collect always false, no?

#4

@comaniac in my case it happened when I was logging the config at the top of a compute / schedule function, before all knobs were defined, which is sometimes useful given the dual collect / apply behavior of config.

Make sense. That’s also the only case I can think of.

True, but on subsequent calls to the same compute / schedule functions config is a ConfigEntity which has _collect always false, no?

I see. Your solution prevents ConfigEntity from calculating the length. It also works, but just a bit indirect.

Thanks for the discussion. Would you mind sending a PR to fix this?