Also, if you've by chance snagged a ticket to the Studio Ghibli Museum in Mitaka (just outside Tokyo - advance purchase of tickets is required) - there are several exclusive soundtracks to their short films that can only be purchased from their gift shop. There are about 8 CDs, that retail tax inc. for 1000yen each. I've never been but they're on my wish list.
I just came back a few weeks ago. It's the best place in the world for Soundtrack CDs. Disk Union has a dedicated store in Shinjuku (with large amount of vinyl). Recofan & Tower Records in Shibuya are incredible, the headquarters of Book-Off are there too. There's branches of Book-Off at many of the stations on the JR circular line, google for locations. They can occupy you for several days. Sorry but I don't know too much about Vinyl (though I think Recofan has large section).
I don't think Book Off has vinyl at least not as far as I looked. But you should be ready with the Kanji names of these composers you are looking for as you may not see their name printed in Romanji on the CD anywhere. If you don't find a used store be prepared to spend what might seem like a lot. When I was last in Tokyo I was more interested in finding Manga in Japanese and I found a few in the Book-Off.
