I read a book describing their history in Poland. They were disliked because they disrespected the local community (would spit or make rude gestures towards catholic churches), and would refuse to assimilate. They preferred to learn Hebrew, English, and maybe even German, but would not learn Polish even though that was the country they lived in because they looked down upon the natives as ignorant and uncivilized.

They actually admired the sophistication of the Germans, ironically enough. From what I understand, Polish nobility used them as tax collectors due to their education, furthering resentment towards them. Even then, they were never kicked out of Poland as far as I know. It was the Germans who accomplished that.