The biggest downside to Japan: loneliness. I'm terribly slow to make friends even when I speak the same language as them, and when I don't have that? Nngh. In some ways, life in Tokyo is somewhat reminiscent of my days in Boston, Denton, and a good portion of Ann Arbor: an inability to find people I liked, much less hanging out with them. AA took about two years before I finally met people outside of work - and in the Bay Area, I met the majority of people I know during my first year there.
I'm pretty happy with most other aspects of Tokyo, although the weather sucks a lot in the summer, things in general are more expensive, and I get occasional cravings for food that's either mostly unavailable or quite costly (deep-dish pizza, cajun food, and spicy food in general.) It tends to be much more compartmentalized than places in the US, though - I've been walking down the main road in Nakameguro pretty much every day for the past ten months and I still find shops and whatnot that I hadn't noticed before.
There's a lot here that's really nifty, but without the ability to connect to people, it's tough.