what is the purpose of detach here?

[Weitian-Wang-Bosch]

Hi @kyegomez. Thanks for your work. When reading your code I'm a bit confused about the purpose of these two lines:

BitNet/bitnet/bitlinear.py

Lines 57 to 58 in 9c3e7dc

	x_quant = x_norm + (activation_quant(x_norm) - x_norm).detach()
	w_quant = w + (weight_quant(w) - w).detach()

Can you shortly explaining it?

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[Mrw33554432]

Ask ChatGPT about STE

[Weitian-Wang-Bosch]

Ask ChatGPT about STE

Thanks for the advice. It actually works way better than googling it. I will just paste the answer below for anyone who is wondering the same question:

"STE" typically stands for "Straight-Through Estimator." STE is a technique used during the training of neural networks, particularly in scenarios where there are discrete inputs or outputs, such as in quantized neural networks.

When gradients are backpropagated through discrete operations, such as rounding or quantization, the gradients are usually undefined or zero, which can cause training instabilities. STE addresses this issue by using a straight-through approximation during backpropagation.

Essentially, during forward propagation, the input is passed through the discrete operation, but during backpropagation, the gradient is passed straight-through without modification. This allows gradients to flow through the network properly, enabling effective training even with discrete operations.