That makes me all the more curious about their back end workflow

Probably not unlike other studios tbh. Outside of specific requirements, like character creation or certain art styles, game character pipelines usually don't differ a great deal. I also doubt the characters are to blame for the performance problems. Open world in engines not tailored to it isn't an easy optimisation target.

I wonder if there's some real maths that can be done on topology still

No hard maths, unfortunately. There's simply too much variability between renderers, RHI's, target architectures, hardware configurations and scenes themselves.

The only advice I can really give is that game optimisation is a bit like software optimisation. Always profile and avoid premature optimisation (primarily micro-optimisation, like obsessing over topology). Dev time is finite, after all. Structural changes usually yield the greatest returns, but can become impractical in late development.

In practice, any number of things can introduce performance issues. That said, unless you have severely sub-optimal game systems or tricky requirements (eg. obscene NPC counts will require specific approaches to character animation, rendering and AI), environments are the most likely cause of poor performance in simpler projects.

Design your environments with your renderer in mind, starting with known limitations and general assumptions based on the rendering pipeline (forward, forward+, deferred all have "general" performance do's and don'ts). Starting in blocking, identify problematic sight lines and areas with poor occlusion or requiring a large number of dynamic and/or shadow casting lights. If you can't resolve these outright, you'll have to compensate with lesser fidelity in those areas. That should at least prevent you painting yourself into a corner/incurring significant rework.

Beyond that, efficient texture use, reasonable shader complexity, good use of instancing, good use of LODs and mesh aggregation, and a lot of care in regards to lighting and shadowing will be your primary means of producing something reasonably performant.