I have a toddler, technically still live in Brooklyn, and I love taking him into Manhattan, in fact, much more than I do going in by myself. He loves it too. The only thing is, I have an external-frame kid-carrier backpack which is much more convenient than a stroller on the subway and the sidewalk, and he's up high and can see all kinds of great interesting stuff.
We have had fun at MoMA, The Met, The Museum of Natural History, Times Square (which I used to HATE but it's actually kind of fun with a baby), Central Park, FAO Schwartz, The East River Ferry, The Statue of Liberty (not convenient for a day trip via Grand Central), Governor's Island (ditto). Never did the Empire State Building in all my years as a NJ/NYer, and now it's too late.