Nn.dense attribute question

Questions
#1

https://tvm.apache.org/docs/api/python/relay/nn.html#tvm.relay.nn.dense

Can someone explain to me what units is supposed to represent here? I’m implementing my own nn.dense function to be lowered from Relay IR, and I’m unsure what this attribute is supposed to be used for. For context, my background in ML is almost none, so I had trouble finding further information since other dense functions I’ve seen don’t have a units attribute.

So far I’ve seen it used in the VGG_Net example:

 let %x139 = Tensor[(1, 4096)] ...;
 let %fc8_weight = Tensor[(10, 4096)] ...;
 let %x140: Tensor[(1, 10), int32] = nn.dense(%x139, %fc8_weight, units=10);

Also, if the default is None, then what occurs in that case? Maybe that is simple to answer after the explanation :slight_smile:

#2

Hi there.

Units is the 2nd dimension of the weight matrix applied to the input matrix. If it was a dense layer in a neural network, it would be the number of neurons. I’m not sure what would happen if you set it to None, perhaps a graceful fail?

#3

Thanks for the response! I’m failing to understand what you mean by number of neurons. Isn’t the number of neurons pre-determined by the inputted matrix size?

In the example above, this would be 10 since we have a matrix multiply of (1, 4096) x (4096, 10). To me, it seems redundant to also have units=10; this thought is likely a gap in my knowledge.

#4

If you have a matrix multiply of (1, 4096) x (4096, 10), then the output will be of shape (1, 10). If you think of this in terms of inputs and neurons, you will have 4096 inputs, to 10 neurons. Each neuron will have a different weight associated with it, and you will end up with 10 outputs (or outputs of shape (1, 10)).

If we look at this example:

image
image840×901 114 KB

We have 7 inputs on the left side of the graph, and 5 outputs on the right side. This corresponds to a dense layer (or fully connected layer) of (1, 7) x (7, 5). We have 5 neurons, which take 7 inputs each.

#5

Thanks! This example is extremely helpful.

With that said, lowering Relay IR with units=None doesn’t fail. Looking at some example programs, e.g. VGG Net, it seems that units is always equal to the number of neurons, which is why it seemed redundant. It also seems nonsensical (and fails to lower) if units is not equal to the number of neurons.