How to remove deactivated CRM contacts from Outlook

Is there a possibility to remove not synced contacts from outlook? Currently only active contacts get synced by default but not synced contacts stay in outlook and do not get removed. Is there a way to easily remove those contacts in bulk without having to reinstall outlook or deleting the SQL Server instance from the computer?