Yep, this was a huge reason as to why West Virginia counter-seceded from Virginia when the latter joined the CSA during the early stages of the American Civil War. Most of WV was rugged country where it wasn't at all profitable to establish slave plantations, and the poor whites who did settle there to work the land had largely come to resent the Tidewater planter class, who in turn looked down on them as basically the 'white trash' of that era. (This is not to suggest the West Virginians had a modern progressive's attitude to race, of course. Actually they wanted to include a provision for total 'Negro exclusion', which is to say 'no blacks allowed in WV', in their statehood bill, but had to pull it under threat of being stonewalled by influential Radical Republicans like Benjamin Wade.)

Aside from the whole slavery dealio, the planters had also spent decades screwing them out of effective representation in the Virginia state legislature by denying them universal white male suffrage in favor of an older propertied system that favored the slaveowners. There was also a significant culture clash between the generally Episcopalian, Anglo-descended planters of the east & the mostly Scots-Irish, Methodist or Baptist settlers of WV (and the Appalachians in general, which is why similar phenomena happened in parts of many other Confederate states, most famously East Tennessee. IIRC the only CSA state that didn't have poor white regions contributing men to the Union Army was South Carolina, the very 'cockpit of secession' itself).