Can't we build it somewhere nearby?

There might be room elsewhere on the grounds, but it probably becomes a question of how far you want your distinguished state guests to have to walk across the grounds, in their formal wear, at various times of the year. You certainly don't want it outside the grounds entirely; I'm sure that would create security headaches and would also diminish the connection to the presidency.

Also, a ballroom as I understand it is not typically a detached structure—hence "room."