Univariate selection - ANOVA F-values

Hi there, I’m looking to understand a bit better what SelectKBest is doing under the hood when using “f_classif” as a feature selector.

As far as I understand, using SelectKBest is a univariate feature selection approach. If the approach is univariate (ie. only looks at one individual feature at the time and assess correlation to the target), why do we use ANOVA / f_classif? ANOVA looks at correlation between multiple features and the target no? (instead of only one feature)

Wouldn’t it make more sense to use something like a simple sort of linear regression between the input feature and the target, and use something like the R squared? (or get a t-test value from this regression? though actually in this case, if we have only one input feature, the t value and the f value I believe would be the same)

Hi @Diego_Carreno ,

The following lecture notes explain well why the F-test is better than just a simple R2 (see chapter 2):


Hope that helps.