Asbestos does its worst damage in the lungs. Here's the thing about big objects: They turn into tiny objects. "Wear and tear" on asbestos means it is releasing into the air around it. Dust is a thing that exists, and it doesn't form out of the aether. Any wearing away, from friction or erosion or strong airflow or any number of things, turns that shelf-stable aesbestos item into a source of dangerous aesbestos dust.

That stated, IF no one is ever touching it to wear it away, and IF wear and tear is essentially zero, then yeah, it's "fine". I've been to plenty of schools and universities with asbestos in the ceilings and inside-walls. It's a durable, flame-retardant construction material. It's just also really bad if you get any dust from it inside you. Any construction/demolition workers are fucked, but in the interim, it's "fine".

Kind of like Uranium in that regard: Very useful, very durable, very stable, don't breathe in the dust. We just aren't retarded enough to build buildings out of uranium so it doesn't enter the same kind of infamy, but uranium munitions/bullets aren't uncommon, and are handled with bare hands.