Why do we use Odd Sized Filters?

I guess this question would’ve been asked before . I have came across different answers that talk about symmetry, central pixel etc. But it really doesn’t click.
Could anyone give an easy to understand intuitive explanation to this?

Because they are symmetric about the center column.

I know… but how does it actually affect?

Another benefit is that you can make a much more versatile filter with a 3x3 grid (81 possible arrangements) than you can a 2x2 grid (16 possible arrangements).

Since images tend to have an even number of pixels in each direction, using a filter with an odd number of elements also gives an even number of pixels in the output.
Due to this formula: output = N - (f -1)
…where N = the number of pixels in each direction.
… and f is the size of the filter (f x f)
Example:
The image is 32 x 32. The filter is (5 x 5)
So there are (32 - (5 - 1)) = 28 positions as you slide the filter across the image in each direction.
So the resulting image is (28 x 28)

I totally understand your concern and I would say first it is mainly because it affects the padding by the formula: p=(f-1)/2
It is obvious that we need to have a number that when substracting 1 and dividing by 2 we still have an integer.
That is my humble contribution

I mean the image size being an even number in each layer doesn’t affect the performance…does it?

Typically the image sizes are even numbers. This is due to how the image sensors are designed.

There’s technically no reason you could not have an odd-sized image.