How do we know that story isn't just a cheating woman with a very active imagination?

That's actually addressed in the bible. When Mary said that God made her pregnant, Joseph's first instinct was to assume she's a lying whore and went to divorce her. But then an angel showed up and was like "my man, it's all real" and everything ended up fine.

Well not fine, but you know. Relatively good, minus the part where they tortured his son to death.