The main reason is that buffer map represented a specific semantics of variable defs(eg the vars in the buffer).
MakePackedAPI desugars that into a low level form that uses load and intrinsics to define those vars. As a result, the buffer map information can now be stale(or duplicated with the desugared impl).
I would try to defer MakePackedAPI as late as possible and do most of the transformations before that, which allows us to take benefit of the high level mapping info.