Using different activations in the same layer

Course Q&A Machine Learning Specialization Advanced Learning Algorithms
1

Does anybody mix in practice nodes with different activation in the same layer?

For example, what if after hidden layer #1 and before hidden layer #3 there is a layer #2 with the following structure:

  • several nodes are activated with relu (continuous features)
  • several nodes are activated with sigmoid (classification features)
  • several nodes are convolutional nodes
2

I have not seen that before.

3

@tenzink,

I have written another reply to you here.

Regarding the idea you have in this thread, we hope the question presenter will offer some supporting reasons or supporting experiment results. They are important because obviously ideas and imaginations are unbounded, but everyone’s experience is limited. Therefore, while we are happy to see new ideas from you, we would also need something more concrete such as reasons supported by research, or experiment results, so that the discussion can be much more fruitful.

I personally have never seen anyone done that before, so it’s hard for me to comment any further. However, when you have time, I would welcome you to try that out yourself and share with us what you find.

Raymond

4

I haven’t seen that before either. But there are lots of researchers trying new things, and it’s entirely possible someone has tried it and written a paper.

An academic paper search (for the correct keywords) would be required to find out.

5

maybe take a look at the section titled Manipulate complex graph topologies on this page of the Keras documentation and let us know if it gets at the objective you had in mind?

… For example, if you’re building a system for ranking customer issue tickets by priority and routing them to the correct department, then the model will have three inputs:

  • the title of the ticket (text input),
  • the text body of the ticket (text input), and
  • any tags added by the user (categorical input)

This model will have two outputs:

  • the priority score between 0 and 1 (scalar sigmoid output), and
  • the department that should handle the ticket (softmax output over the set of departments).

my emphasis added

2 Likes