I have no idea what the setup is in the studio (live and if/how the podcast version is mixed), but I have noticed this issue. It's not so big, and I haven't noticed it much with Tom as with guests. I think Tom knows what he's doing, and knows how to use his vocal dynamics to achieve certain effects.
As for guests, for example, every time Julie Klausner is in studio I cry because she needs a compressor/limiter so badly... the dynamic range between her speaking voice and her laughter is huge. I have to crank up the volume to hear her speaking, then spring for the volume whenever she laughs before my eardrums explode.
I am a sound tech, so I might notice or have more problems with such things than the average listener.