Cowardice. They're all part of the same cult, so if they speak out against said cult, they'll get kicked out. Once a cultist gets excommunicated, they have nothing. No friends, no family, no community. Everywhere they go they'll be mocked and harassed.

This is by design. If they all simultaneously left, the cult would have no power over them, but they're kept unorganized and isolated so fear keeps them in line.