Maybe "transported to an unfamiliar world" is a concept that's existed in fiction for thousands of years

Heck, it wouldn't even surprise me if some of the earlier stuff was men whisked away to another world to find love with gorgeous ladies. Just look at Celtic stuff, Tír na nÓg and the like. Plenty of tales of men getting swept up into the world of hot fairies and the like. Plenty of darker tales too, and it's not like it always ended well, but the point stands. Men have been getting whisked away to spend time with gorgeous ladies in fiction for who knows how many thousands of years. Vice versa, too, there are plenty of tales of otherworldly men pulling women into their worlds.

Like you said, otherworldly journeys aren't exactly a new trope.

Does Lucian's "A True Story" count?