Question about the dlight

anders · July 10, 2023, 10:35am

regarding with the relax auto-scheduling issue, I heard about a site-package or 3rd-plugins which name “dlight” is under development，which would replace DefaultGPUSchedule which offers great performance at no auto-tuning cost.

So, whats the status now, I 'm expecting and can not wait for any second.

[Unity] Schedule Needed while building relax model with GPU Target - Questions - Apache TVM Discuss

yzh119 · July 19, 2023, 5:37am

The first version of dlight has already been integrated into TVM Unity and MLC-LLM, you can try this feature by upgrading your relax and MLC-LLM to the latest version.

spoilers: it’s super fast

cydia · August 7, 2023, 2:43am

I have seen dlight in mlc-llm, but default dl.gpu.Matmul() does not seem to use nv’s tensor core, which makes matrix matmul quite slow. Any good suggestions?

yzh119 · August 7, 2023, 8:19am

Tensorization has been supported in matmul schedule of dlight (contributed by @adfwer233 ):

github.com

apache/tvm/blob/6c38001fe1bf5213e27afb83b547073649edc932/python/tvm/dlight/gpu/matmul.py#L282


        return None


    return reduction_blocks




def check_sm_version(arch: str) -> int:
    sm_version = arch.replace("sm_", "")
    return int(sm_version) if sm_version.isdigit() else -1




class MatmulTensorization(ScheduleRule):
    """
    The schedule rule for float16 tensor core matmul computation.
    func with attr 'dlight.do_not_tensorize' will not be tensorized.
    """


    def apply(  # pylint: disable=too-many-locals,missing-docstring
        self,
        func: tir.PrimFunc,
        target: Target,
        _: bool,