It very well could be racism or violence holding them back. The sheer number of stories I've personally heard from black people of sufferring physically or socially because they're attempting to succeed and thus "acting white", seemingly a crime in their culture, is staggering. Fewer black racists committing black crime against the black people trying to improve themselves and their standings would likely lead to a better outcome in those communities.