Custom Permission can be checked in Validation Rule. Use $Permission.Custom_Permission_Name in the Validation Rule to check. If the User have Custom Permission via profile or permission set, then it will return true. Else, it will return false.

Check the following simple Validation Rule on Account object to prevent Account records creation with Type “Business Account” and without Custom Permission “Create_Business_Account”.

Custom Permission:

Validation Rule:

Output:

Custom Permissions can be assigned to the Users using Profile or Permission Set.