I've been a WFMU listener on and off since 1990 and generally sort of a comedy dilettante. With high-speed internet and podcasting, I really started listening in earnest to everything except Dave Emory, pretty much.
I didn't get TBS at all at first, and the length was tough for me, tougher than anything else, really (though now I wish it was six hours long instead of three). I found an open door in the form of Rock, Rot, & Rule, and I was hooked. It didn't take very long for me to become a full-fledged sycophant.
The other thing is, part of what makes the show so funny is that Tom is so good at staying deadpan all the time. It's really hard to tell when he's kidding. This becomes really hilarious and rewarding with repeat listening, but it becomes a little bit of an obstacle. It's kind of like modernist literature in that way - you have to work at first, but it gets easier with time, and the rewards are big.
My wife loves the show, but interestingly, I don't think she quite gets most of the Wurster calls - her favorite parts are Tom's rants and interactions with callers.