BatchNormConfig num_features and num_channels

[wangjiawen2013]

Hi,
I always thought that num_features in BatchNormConfig::new() means the number of features, but in fact it means the number of channels ? All the features in a channel share the same gamma and beta ? For example, here is an image:

[[
[1,2,3],  // channel1
[4,5,6]  // channel2
]]

I thought num_features should be 3 because there are 3 elements (features) in a channel , but in fact it's 2 in BatchNorm because there are two channels. So BatchNormConfig::new(2) is right and BatchNormConfig::new(3) is wrong, is it ?

And so the dimensions of input tensor of BatchNorm must >=3, right ?

[wangjiawen2013]

This let me think about another case. For a linear layer, are the following two methods different ?

method1:

let tensor = Tensor::<MyBackend, 3>::from_data([[[1, 2, 3],[4,5,6]]], &device);
let fc: nn::Linear<MyBackend> = nn::LinearConfig::new(3, 4).with_bias(true).init(&device);
let output = fc.forward(tensor.clone());

method2:

let tensor = Tensor::<MyBackend, 3>::from_data([[1, 2, 3, 4,5,6]], &device);
let fc: nn::Linear<MyBackend> = nn::LinearConfig::new(6, 4).with_bias(true).init(&device);
let output = fc.forward(tensor.clone());

method1 will run full connection for [1,2,3] and [4,5,6] indepdently and method2 will run full connection for [1,2,3,4,5,6] as a whole, is it?

[laggui]

Yeah the two operations are different. The weight matrix for the first case is [3, 4] while the second is [6, 4], so the transformation applies to different elements (as you noted).

[laggui]

Hi, I always thought that num_features in BatchNormConfig::new() means the number of features, but in fact it means the number of channels ? All the features in a channel share the same gamma and beta ? For example, here is an image:

[[
[1,2,3],  // channel1
[4,5,6]  // channel2
]]

I thought num_features should be 3 because there are 3 elements (features) in a channel , but in fact it's 2 in BatchNorm because there are two channels. So BatchNormConfig::new(2) is right and BatchNormConfig::new(3) is wrong, is it ?

And so the dimensions of input tensor of BatchNorm must >=3, right ?

You're right that it should be BatchNormConfig::new(2). This might be confusing, but each channel can be thought of as representing a "feature map", leading to the use of "features". In fully connected layers, "features" more evidently refers to the dimensionality of the activations.