Both the family and the Hall take their name from the same place, the village of Brougham and the origins of that name and place go much further back. 

The most likely derivation of the name comes from the Old English words ‘bru’ (brow of a hill) and ‘ham’ (homestead) [which] became fused together, 

Brougham = brow of a hill + homestead

…or that the word is a metathesised form of the Old English word ‘Burh ham’ (meaning homestead near the fortification).

Brougham: homestead near the fortification

That description matches the place well. Brougham was a place on a hill overlooking a river near a fortification. That fortification is Brougham Castle, a 20 minute walk down river from Brougham Hall.