Using correlation matrix for feature selection

[Christian_Simonis]

@mehmet_baki_deniz

here in the following picture, I understand that the decision boundery is circlear(something like x1^2 +x^2 right? ).

You can describe the data with sin(t), cos(t) relation, see:

Can we start with the circle equation as decision boundary?

Hi @tbhaxor, in addition to @rmwkwok 's excellent answer. I agree: Feature Engineering will do the trick: Of course you can also apply a transformation as we know it from polar coordinates to get rid of the non-linearity in the data and solve the problem with a linear classifier. [Bildschirm­foto 2023-01-14 um 20.36.23] import numpy as np import math import matplotlib.pyplot as plt # Some Parameters dim = 100 noise = 0.2 #Create inner circle x_1 = np.cos(np.linspace(0,2*math.pi, dim)) + …

After transforming the data, they become linearly separable.

An other way: you can also solve a problem like this with polynomial approaches, see: https://towardsdatascience.com/the-kernel-trick-c98cdbcaeb3f

@mehmet_baki_deniz

if the correct model were sth like x1^4+x1^2Xx3^4 would we be able to visualize it?

When analysing the residuum you do not want to see any systematic pattern. If the model did a bad job you could see at least some pattern in the residual data and no random “white noise”.

@mehmet_baki_deniz

wouldnt a better approach be to add polynomials of the features(that already seems important) to the dataframe and see if the new (linearized) features have correlation ? I just presume 3 degree would be enough to avoid overfitting.

This can absolutely help in data understanding.
Why do you think it would be “better” to add new features?

I think the transformation w/o growing the dimensional space is already making the data linearly separable in a minimum dimensional space. To me it seemed quite elegant this way. But many approaches solve the issue.
In general: The most suitable approach depends on the data as well your business problem you are solving. Often in reality it is sufficient to find a solution which is just “good enough”.

Best regards
Christian