There are several points to make here:
- For starters the whole point of pooling layers is to reduce the size of the height and width dimensions.
- It turns out that “same” padding in general (for “conv” layers as well as pooling layers) doesn’t work the way you might expect: you only get the same output size when
stride = 1. But in the pooling case, it is rare thatstride = 1. Here’s a thread which discusses this general point about “same” padding. Make sure to read the “Update” section at the end of that post.
My suggestion would be to try some experiments with pooling layers and watch what difference it makes in the output dimensions when you use “same” versus the default “valid” padding.