Non-Setup objects are standard objects like Account or any custom object.

 

Setup Objects are used to interact with metadata for declarative development. Setup objects are Group1, GroupMember, QueueSObject, User, UserRole, UserTerritory, Territory, etc..

List of Setup objects can be found in the following link

https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/reference_objects_setup.htm

 

Note: 

We cannot do DML operations on Setup and Non-Setup objects in the same transaction. We will hit the “MIXED_DML_OPERATION” exception.