In the above picture, I am unable to understand what does “This enables multiple units to contribute to the resulting function without interfering” mean?
I think the interference or combination is still possible for multiple units right?
ex: ReLU(u1) + ReLU(u2) could possibly result in a new line that is different from both ReLU(u1) and ReLU(u2) is this not considered as interference?
Yes! It is possible, and indeed, it is quite hard to be impossible.
I think when the lab author wrote that sentence, they were focusing on how just the zero level of one ReLU NOT interfering with any other ReLUs. This makes sense, right?
However, when there are three or more units, it is very much likely that there should be at least one ReLU that its non-zero slope INTERFERES with another ReLU’s non-zero slope. But I believe this part is not at the spirit of writing that sentence down, and it does not mean it is not possible.
Cheers,
Raymond
1 Like
thanks for the confirmation!
1 Like
You are welcome, @abhilash341!