[AutoTVM GPU] training data did not have the following fields

FrozenGene · August 26, 2018, 5:09pm

Training meet error: training data did not have the following fields: f1121, f1125, f1170, f1127, f1158, f1142, f1141, f1132, f1153, f1147, f1173, f1123, f1160, f1137, f1152, f1150, f1157, f1145, f1161, f1164, f1140, f1172, f1134

n_trials too large? I set 2000. @merrymercy

eqy · August 26, 2018, 10:59pm

n_trials should not be the problem here. We have used > 10,000 trials before without an issue.

FrozenGene · August 27, 2018, 1:03am

I find if we use transfer_learning to train, we have problem.

merrymercy · August 27, 2018, 3:16am

Do you use your modified tuner?
The code in my PR should not have this problem

FrozenGene · August 27, 2018, 4:02am

Yes. But it is very strange that everything is ok when not use transfer_learning.

merrymercy · August 27, 2018, 5:05am

Because there are some “if” statements in the schedule.

github.com

merrymercy/tvm/blob/6189f8d06aaf7c07f687f8b05bfe20f46a6b3fa5/topi/python/topi/cuda/conv2d_winograd.py#L196


s[data_pack].bind(tt, tvm.thread_axis("threadIdx.x"))


s[data_l].compute_at(s[data_pack], pi)
s[input_tile].compute_at(s[data_pack], pi)
s[pad_data].compute_inline()


# transform kernel
if not pre_computed:
    kernel, G = s[kernel_pack].op.input_tensors
    eps, nu, co, ci = s[kernel_pack].op.axis
    if autotvm.GLOBAL_SCOPE.in_tuning:
        s[G].pragma(s[G].op.axis[0], 'debug_skip_region')
        s[kernel_pack].pragma(eps, 'debug_skip_region')
    else:
        s[G].compute_inline()
        r_a, r_b = s[kernel_pack].op.reduce_axis
        for axis in [eps, nu, r_a, r_b]:
            s[kernel_pack].unroll(axis)


        fused = s[kernel_pack].fuse(co, ci)
        bb, tt = s[kernel_pack].split(fused, 128)

The output of feature extraction will be different for different branches.
When loading history data, we should set the context GLOBAL_SCOPE.in_tuning = True. (This is a patch I sent several days ago https://github.com/dmlc/tvm/pull/1615)

github.com

merrymercy/tvm/blob/6189f8d06aaf7c07f687f8b05bfe20f46a6b3fa5/python/tvm/autotvm/tuner/model_based_tuner.py#L266-L289


def load_history(self, data_set):
    # set in_tuning as True to make the feature extraction consistent
    GLOBAL_SCOPE.in_tuning = True


    # fit base model
    base_model = self.cost_model.spawn_base_model()
    success = base_model.fit_log(data_set, self.plan_size)


    if not success:
        GLOBAL_SCOPE.in_tuning = False
        return


    # use base model to select initial points
    if not self.trials:
        # no plan yet, use base model to select initial trials
        maximums = self.model_optimizer.find_maximums(base_model, self.plan_size, self.visited)
        self.trials = maximums
        self.trial_pt = 0


    self.cost_model.load_basemodel(base_model)

This file has been truncated. show original

FrozenGene · August 27, 2018, 7:34am

Thanks. I haven’t updated this code. I think it is the reason. However, I have started the training and I don’t want restart training again. Because it occupy much time. I want to ask, if I don’t use transfer_learning, it will affect my result performance or not?

merrymercy · August 27, 2018, 8:32am

No, it won’t affect the performance

ehsanamiri1354 · May 30, 2019, 9:00am

On a related note, can you please point to the local and global cost model (Eq. 4 in the paper).

eqy · May 30, 2019, 10:57am

Please open a separate thread for new questions.