Sunshine Street in Ikebukuro is the setting for some scenes in Durarara!!; it's also just a generally very famous vista that I've seen in a few different manga I can't even remember the names of. If you walk down that street when you're in Japan, then continue watching anime and reading manga, chances are high that you will see it in something you watch or read later on.
Tokyo Tower is pictured in tons of anime and manga and is definitely a must-see if you spend any time in Tokyo at all.
Several locations in Tokyo were featured in X/1999. Besides Tokyo Tower and the Sunshine 60 building, the other non-fictional sites that were mentioned were the Tokyo Metropolitan Government Building in Shinjuku (great place to get a /free/ panoramic view of the city from the top of a high-rise building), Rainbow Bridge (hard to miss if you spend time in Tokyo), the Diet Building, the Yamanote train line (which you will almost certainly ride if you spend time in Tokyo), and a certain building in Nakano--Nakano Sun Plaza, I think it's called.
Tokyo Dome and the famous Kaminarimon gate near the temples Sensoji and Asakusajinja are both such famous sights that they're featured in multiple anime/manga; I'm pretty sure I've seen each of those in at least one anime or manga, but I'm not sure which.
Speaking of Nakano, these are not particularly featured in any anime or manga that I have seen, but both Nakano Broadway and Akihabara are big shopping areas for geeky interests such as anime, manga, and electronics, along with Ikebukuro. (All of these are in Tokyo.)
Definitely anime-related but not featured in any anime that I know of is the futuristic water bus you can catch in Asakusa (in Tokyo) called the Himiko; it's designed by Mastumoto Leiji and has cutouts of the Galaxy Express 999 characters in the front of the boat.
I also highly recommend a trip to the Studio Ghibli museum in Mitaka (suburb west of Tokyo). While it is not featured in anime, it is all about anime and is absolutely enchanting. You do have to purchase your tickets in advance, though.
Kyoto has a manga museum, and you can see statues of Tezuka Osamu characters right in front of Kyoto Station.
Near Kyoto is a little village named Otsu, which is where Kenshin and Tomoe go to live together when Kenshin is going into hiding in the Ruruoni Kenshin Trust and Betrayal OVAs. This town is a little hard to get to (you have to take a bus that's a different company than the regular Kyoto city buses), but in it is a temple called Hosen-in which is renowned for its bloody ceiling made from planks taken from the floor of a castle where a large group of samurai, surrounded by enemies, all committed ritual suicide. However, if that doesn't interest you and you choose to ignore the traces of blood which can still be seen on the ceiling of the temple, it is still one of the most beautiful and enchanting temples I saw while in the Kyoto area and definitely worth checking out.
If you want to consider places featured in manga but not anime, there's a famous castle in Okinawa, Shuri Castle, that appears in the Basara manga. It is a great place to visit if you happen to be in Okinawa, but Okinawa is pretty remote from the rest of Japan, so your travels might not take you there.