getting plugin error "Access team for team template for record of type {1} and id {0} and not found"

there is a scenario where a user is added to lookup we have to add the user to access team under record and when ever we remove the same user from lookup we have to remove the access under access team and for this we have developed plugin everything is working fine but rarely we are getting this plugin error message

Access team for team template for record of type {entity = project} and id {id = projectID} and not found

below is the code used in plugin for removing user from plugin

RemoveUserFromRecordTeamRequest adduser = new RemoveUserFromRecordTeamRequest()

{
Record                 = record,
SystemUserId      = systemuserid,
TeamTemplateId  = teamTemplateId
};