But I think part of the reason that baseball was removed from the Olympics is just that MLB was unwilling to make any moves whatsoever towards letting MLB players participate.
The blame for the sport being removed lays as much on MLB as the IOC.
That said, (apart from relays) team sports are a kind of a tricky fit in the Olympics. But it's crazy that handball makes the cut and not baseball or rugby.