Recall curve score for XGBoost

Ediolot · June 21, 2021, 2:22pm

Hello ! I have been reading the custom_callback function that autotvm uses to train the XGBoost cost model.

There are a few things that I would like to ask for clarification if possible:

The init function checks if the model (bst) has a best_score attribute which as I understand means that the model has been already trained to some degree so we set the state variables and we can keep doing incremental training. However since the init function only happens once per train at the beginning and each train restarts the model, this code has no use at least by default. Am I understanding this correctly ?

The code from github:

apache/tvm/blob/0f41d47bb43ba7509771a6254b986dcecc144479/python/tvm/autotvm/tuner/xgboost_cost_model.py#L493


    bst = env.model


    state["maximize_score"] = maximize
    state["best_iteration"] = 0
    if maximize:
        state["best_score"] = float("-inf")
    else:
        state["best_score"] = float("inf")


    if bst is not None:
        if bst.attr("best_score") is not None:
            state["best_score"] = float(bst.attr("best_score"))
            state["best_iteration"] = int(bst.attr("best_iteration"))
            state["best_msg"] = bst.attr("best_msg")
        else:
            bst.set_attr(best_iteration=str(state["best_iteration"]))
            bst.set_attr(best_score=str(state["best_score"]))
    else:
        assert env.cvfolds is not None


def callback(env):

The custom evaluation function that is generated by xgb_average_recalln_curve_score returns the average of N elements of the recall curve score. This N elements are the max N elements according to the pred values. Why dont we just average for all the elements instead ? What am I missing here ?

The code from github:

github.com

apache/tvm/blob/0f41d47bb43ba7509771a6254b986dcecc144479/python/tvm/autotvm/tuner/xgboost_cost_model.py#L614






def xgb_average_recalln_curve_score(N):
    """evaluate average recall-n curve score for xgb"""


    def feval(preds, labels):
        labels = labels.get_label()
        trials = np.argsort(preds)[::-1]
        ranks = get_rank(labels[trials])
        curve = recall_curve(ranks)
        return "a-recall@%d" % N, np.sum(curve[:N]) / N


    return feval




def xgb_recallk_curve_score(N, topk):
    """evaluate recall-k curve score for xgb"""


    def feval(preds, labels):
        labels = labels.get_label()
        trials = np.argsort(preds)[::-1]

Thanks !