Freedom of association is as critical as freedom of expression. The two are inextricably linked - if you cannot associate with those around whom you wish to express yourself, or disassociate from those you cannot express yourself in the presence of, then you do not have freedom of expression any more either.