To hide chatter groups from users, use “Unlisted Groups” in Salesforce.com.

To know more about “Unlisted Groups” in Salesforce.com, use the below link

https://www.infallibletechie.com/2014/10/unlisted-groups-in-salesforcecom.html

Cheers!!!