Let me please ask a question.
In one of introductory videos Prof. Andrew shows how we go from input parameters (size, bedrooms, zip, wealth) to a new set of features (family size, affordability, school quality). I cannot understand the intuition behind creating these new features… Why could not we train the NN with initial set of features?
Thank you.
Vasyl
Kyiv, Ukraine.
Hi @vasyl.delta,
The more features you have, the better your prediction will be.
Talking about the house price prediction, let’s say your features only include number of bedrooms. This model will always give you the same price for, lets say, 3 bedroom house.
Let’s say you have another model and this time in the features, you include number of bedrooms and bedroom size. This model will not give you the same price for 3 bedroom house as it will also consider size of the bedrooms. More bedroom size, more price and vice versa.
As humans, we know, bigger bedroom size means more price so that’s what we are trying to capture…more information by adding more features for making better predictions.
Best,
Mubsi
1 Like
Thank you for the answer!
The question was rather on how to find new features…
Besides, In the mentioned example the input features remained the same, but the new ones were introduced in the next layer of neural network.
Basically you have two approaches:
-
hand crafting your features with your domain knowledge. This works often well with classic ML and < 20 features, see also: W1_Quiz_Large NN Models vs Traditional Learning - #4 by Christian_Simonis
-
using deep learning which kind of automates feature engineering using lots of data. This often suits with big and unstructured data, see also this thread. With these tons of data the model can learn abstract patterns: Deep Learning models with advanced architectures (like transformers but also architecture w/ convolutional | pooling layers) are designed to perform well on very large datasets and also process highly unstructured data like pictures or videos in a scalable way: basically the more data, the merrier! Compared to classic ML models DNNs possess less structure and can learn more complex and abstract relationships given sufficient amount of data, see also this thread: Why traditional Machine Learning algorithms fail to produce accurate predictions compared to Deep Learning algorithms? - #2 by Christian_Simonis
There are several reasons why feature engineering is key. Couple of them are:
-
performance: e.g. you can model non-linearity already in your features and make it easy for your ML model for fit the data well. So whenever you can model „distilled knowledge“ in features this is valuable for modelling
-
costs: often this also helps you in terms of requiring less data for a good model which often reduces your costs, especially when labels are expensive to create
-
interpretability: often features have a nice domain interpretation (e.g. a physical meaning or an aggregation of knowledge) which also helps for plausibility checks and interpreting (or explaining model results to your stakeholders).
Hope that helps, @vasyl.delta.
Best regards
Christian
Thank you very much for the detailed answer!
Just to make one point a little bit clearier:
In the task of digit recognition we have a set of pixels with their colours as an input to the NN.
In the next layer these values are modified so to detect different parts of the digits. How do we come to the idea that this layer should extract parts of the digits? Has it to be decided by the engineer or DL algorithm itself can “understand” that such features are reasonable (because sometimes it is said that DL eliminates the stage of feature extraction)?
I humbly hope that I expressed my question clearly.