Can anyone tell me whether the dram described in the architecture diagram is on the device side or the host(cpu) side?
It is on device side, I think.
The VTA is implemented on the FPGA fabric of the Xilinx ZYNQ, which includes an ARM processors (host).
I believe the DRAM is part of the host, since the FPGA fabric has its own type of memory which is called BRAM (mostly used to implement the SRAM of the VTA).
So basically the host needs to prepare the DRAM content and then activate the Load instruction of the VTA runtime which offloads the content into the BRAMs of the FPGA fabric. This transfer is (I think) actually done by a DMA controller.