Hello, what is the point in using the max in J=max(d_P - d_N, 0) in the triplet cost function? It seems like an unnecessary complication for the optimizer if the initial weights make the d_P - d_N positive (the gradient will be zero)…

No, it is just plain math. For example, \newcommand{\dv}[2]{\frac{\mathrm{d} #1}{\mathrm{d} #2}} J = \sum_{i = 1}^m \max(0, w x_i) \max(0, w x_i) = \begin{cases} w x_i & \text{ if } w x_i > 0 \\ 0 & \text{ if } w x_i \le 0 \end{cases} \dv{\max(0, w x_i)}{w} = \begin{cases} x_i & \text{ if } …

[image] Loss function in triplet loss Convolutional Neural Networks The reason we have a margin and the max operator is that, when the representations produced for a negative pair are distant enough, no efforts are wasted on enlarging that distance, so further training can…

Mmm okay, but how does the training “focus” on the more difficult pairs? The optimizer as we learnt it simply computes all the derivatives, no matter if the values are zero or not. There would have to be some hard coded instruction to skip the derivatives computations if the loss came out as zero?

Thanks, I think I understand now. The “hard coded” instruction to skip the computation is in your analytical expression of the derivative itself (0 if wx_i <= 0). I imagined that the whole derivative expression (just x_i in this case) is firstly evaluated for the whole batch in a vectorized way an…

Yes, using the max function acts as a derivative gate and is a common trick in machine learning. I just illustrated without using vectorization. You can as easily add that to my example as well. You can try it out for yourself. Vectorization or no vectorization doesnt matter, the derivative will ge…

I am not sure how to vectorize it in such way to be honest. I would do np.sum(dwX/dw * (wX>0), axis=1), but this doesn’t avoid the gradient computation also for the zero components. Any hints?

X \in \mathbb{R}^{1 \times m} wX > 0 \in \mathbb{R}^{1 \times m} \text{ which acts as a mask of ones and zeros} \frac{dJ}{dw} = (wX > 0) X^T \in \mathbb{R}^{(1 \times m) \times (m \times 1) = 1 \times 1} should do it :cowboy_hat_face:

Thanks for the reply, I think you will be able to see what I mean now. \phantom{x} Your formula in numpy is np.dot(wX > 0, X.T). \phantom{x} This is computationally better than what I wrote before, but you can see that you haven’t avoided evaluating \frac{\text{d}wx_i}{\text{d}w} for all the …

Yes, but we don’t want to focus on examples where we already are good enough, we don’t want those examples interfering with how we try to do better on the “hard” examples.

Ah ok, so it’s not about computational effort then. I think I get it now. Perhaps the key point to realize is that a global solution to \text{minimize } J_1=\sum_{i=1}^{m}(d_P-d_N) may lead to weights where the predictions are extremely good for some samples, but quite bad for others. We prefer…

[image] jm1e16: may lead to weights where the predictions are extremely good for some samples, but quite bad for others. We prefer weights that satisfy d_P-d_N+\alpha\leq0 dP−dN+α≤0d_P-d_N+\alpha\leq0 for as many samples as possible, no matter how closely, which we can achieve yes …

Week 4 Triplet loss

Course Q&A Deep Learning Specialization Convolutional Neural Networks

jonaslalin October 27, 2021, 11:07am 2

Topic		Replies	Views
Loss function in triplet loss Convolutional Neural Networks coursera-platform	4	811	October 1, 2021
Error in the triplet_loss function (based on expected output) Convolutional Neural Networks coursera-platform	6	444	December 13, 2023
Triplet Loss - why bother maximizing with 0 Convolutional Neural Networks week-module-4 , coursera-platform	2	135	January 24, 2025
Class4 Week4 Ex1 Function triplet_loss Convolutional Neural Networks coursera-platform	1	729	July 15, 2021
C4 W4 A1: Programming Assignment: Face Recognition UNQ_C1 possibly wrong unit test answer Convolutional Neural Networks coursera-platform	2	524	May 4, 2023

Week 4 Triplet loss

Related topics