When the users upload files, it will be stored in ContentDocument Entity. So, develop a trigger on ContentDocument entity to fire when the users upload files in Salesforce.

Sample Trigger:

trigger ContentDocumentTrigger on ContentDocument ( before insert ) {
    
    for ( ContentDocument objCD : trigger.new ) {
        
        if ( objCD.Title == 'test' ) {
            
            objCD.addError( 'Do not upload Test File' );
            
        }
    }

}

Output: