It's gotta be Josephine, but the problem is that that's historically accurate too. He's a bit of a cuck when it comes to her. That is how she kinda kept one of the most ambitious men in human history in line. I don't doubt she probably told him something like that to his face. If she didn't, he certainly believed it.

And she has a point. He had no capacity to navigate through the French aristocracy or upper class. He wasn't even French to a lot of them. He did absolutely need her social graces at a political level.