Out of Office being sent to Regarding Contact

I ran into very strange issue in CRM. When a contact is created, I have created a workflow to send an email to one user (User A) from system user(User B) and I am setting the regarding with current contact. Everything works fine and emails are getting sent to User A. When User A set his Out Of Office Email, the regarding Contact is now getting that Out Of Office email. That should not happen because he is not the intended recipient of the email only User B should get the Out of Office email. For the time being i removed the Regarding Contact from workflow and that resolved the issue. But I need to set regarding contact as well and Out Of Office Email should not be sent to Regarding Contact. Anyone has any ideas how to achieve it??