Your scenario only works for small towns where everyone knows each other. In a large enough city even among civilized folk there will always be disputes and you can't always be sure who is wandering through, and normal people are too busy with work to police the city themselves.

For large cities with peaceful residents (not the kinds of cities we have today), I sometimes wonder if the best setup would be a massive smart camera network - not run by a government, just a publicly accessible network - and voluntary police/militia groups. Or maybe those voluntary police would end up like jannies and moderate everyone to an annoying degree.