For starters...
-Mercer Street Books and Records is a great hole in the wall with vinyl, CD's and obscure books.
-Other Music is pretty good, but small, and I sometimes find the help a little snooty.
-The Strand Bookstore is a New York mainstay, but can be overwhelming.
-St. Marks Bookshop is a little more manageable and has a wonderful selection of books, magazines, and zines.