I would be personally affected by said policy obviously

How so? Do you think all the White business owners around you are rabid racists and only let you in their stores because it's the law?

And you still want to give them money?

You don't think there are any other businesses around that would gladly cater to you for money? What about large corporate stores, or "Black Owned" businesses?

If you're talking about banks redlining, I'll just point out that most Black people I've heard talk about this wouldn't want to live in a majority black neighborhood either. I'm sure there were ways to get around it.

the CRA violates freedom of assembly in the view of people who have that opinion

It violates freedom of association, objectively. Arguably you can say the government has the right to regulate commerce and businesses offering services is different than individual associations - but it's still a form of association.

I would be personally affected by said policy obviously

How so? Do you think all the White business owners around you are rabid racists and only let you in their stores because it's the law?

And you still want to give them money?

You don't think there are any other businesses around that would gladly cater to you for money? What about large corporate stores, or "Black Owned" businesses?

If you're talking about banks redlining, I'll just point out that most Black people I've heard talk about this wouldn't want to live in a majority black neighborhood either. I' sure there were ways to get around it.

the CRA violates freedom of assembly in the view of people who have that opinion

It violates freedom of association, objectively. Arguably you can say the government has the right to regulate commerce and businesses offering services is different than individual associations - but it's still a form of association.