When you kill that many people, it seems odd to assume every single one was a jew when we already know the nazis killed communists, gypsies, dissenters, freedom fighters and etc.

Its more likely that they'd just classify you as a "jew" if you were one of those things, seized your property, sold it via the state (they were a socialist country - they were running out of money fast so jewish seizures was a big source of income) and sent you to the bulldozers to die in Poland.