Basically it turns everything into government by rabble and special aggrieved interests.

Well, I agree that it'd turn out pretty rough if applied to our present large scale society, so let me walk it back and try again.

I actually think that'd be a way to describe the feature of small scale communities. It rapidly becomes a problem when you have multiple communities participating in the same vote, but if each community is only allowed to vote for affecting their own borders, what's the problem? Any border sharing of multiple communities will carry strife and negotiations with it, so that doesn't change much - cities can even continue being like a nightmare.

You could even require a majority vote of 90%+ in such small scale communities without totally paralyzing legislation, since it's not far off to assume that members of a community would have shared interests.

Edit (unclear point): direct democracy should only be a problem when people have changes forced upon them by people that do not (or can not) represent their interests. Reduction of scale should diminish this problem.

what you're looking for is personal responsibility. Democracy defuses that.

That is something I'd like, but how does democracy defuse it? Is there a form of government that does not? I thought it'd be the best at permitting the interests of all involved to be represented, even if no one is able to get a win for their interest. If I want to pursue something unpopular, like permittinng non-pasteurized dairy to be sold, I'd prefer throwing my vote into an empty bucket rather than trying to find a representative that promises to help and totally won't betray any other interests.

I'd probably advocate for a meddling system to make sure that the rats fleeing sinking ships are known to be associated with failure when they try to vote in their new homes. No need to get harsh with it, allow each community to handle that information in their own way. If they're fully integrating, the new host community likely won't care. But I also realize that this is reminiscent of a social credit score and feel like that marks some failure on my part.

Within specific communities, any legislative failures could have repurcussions for the people that voted for it, as each community would be free to have whatever rules they like for their members (excluding the prevention of leaving, ideally). Yes, it could get stupid, but that's a feature to me, because success means nothing when you could never fail. I think it's only cruel or accelerationist when scaled up.

The smaller governments get together to refuse the authority of the bigger ones. The bigger ones always face internal power struggles. The smaller ones face shifting alliances.

Isn't this basically returning to our roots? It sounds like the vision of the founding fathers might fit with this, though maybe I don't know their vision really well.