The downside is a lopsided population pyramid, which does come with its own fair share of issues. Alleviated with technology in the modern era, though, and knowing it is inbound would drive specific tech advances to further help, but that would be the main drawback, all of the problems of an inverted population pyramid. Not to say whether it would be "worth it" or not, just that a downside DOES exist.