It is immoral but does not violate rights: life, liberty and pursuit of happiness. Same as how being fat is bad for society but you should be allowed to eat what you want. We simply should make sure others aren't forced to pay for bad choices, like broken homes (divorce due to adultery) leading to single moms on welfare, or obese people raising insurance or socialized health cost for others. You do that by getting rid of such programs, less government.