Hi Everyone, in the first lab of week 3 it was mentioned that we should use the same scaling from the training set for both CV and test scaling which makes sense. However, in the code snippet, I am attaching when we were looking for the best polynomial degree for the model it wasn’t the case anymore and fit_transform was used for both CV and later for test sets and I am lost why?
Hi @MohamedAbdo1
Welcome to the community!
That’s a good question,
According to the scaling, label encoding you should do the transformation in CV, and test data like what you did in the training as it’s compute or give specific values for example in the label encoding to the words in training so that if the transformation see the same word in test it must give it the same value,
But in the polynomial it’s just compute the combination between the feature it isn’t dependent on weights or something like that for example if you have 2 features x1, x2 and the polynomail of degree 2 the combinations should be x1,x2,x1^2,x2^2,x1*x2
Note if the number of the CV examples you have is large(enough) you can use different tranformation in the scaling only not do that with label encoding, also make sure you suffle the data before doing transformation
Best Regards,
Abdelrahman