Traditionally they are legally classified as "trailers" and are built to a separate Federal code (colloquially referred to as the "HUD code/standard" because it's maintained by the Department of Housing and Urban Development). There are two issues with this:

• Legally they are classified as "personal property" like an RV/camping trailer would be and not "real property" like a house would be, so insurance and loans are different
• The HUD standard at least used to be less rigorous (I don't know if this is still the case), so trailers had less of a lifespan.

But nowadays most manufacturers will build them to either HUD code or building code, and in the latter case they are built to exactly the same standards as if the house were built on-site and in a lot of states are legally classified the same as a "traditionally" built house.

Of course there's the larger issue of perception and the simple fact that any method of "cheap construction" is going to be seen as low status. In large part by definition, because it's going to attract a lot of people who can't afford any other option