I suspect it's more likely to be the former, honestly. Even if none of the landmarks we have access to are as strikingly similar to something from our worlds, Johto and Kanto do have a superficial resemblance to Japan... plus, I'm really not sure how long people have been arriving from other worlds in the first place.

I think four or five years may be the longest this has happening, but it's hard to tell without access to other regions.