D365 Portal Redirect still redirects even when de-activated

We have configured a portal redirect which takes an inbound URL and redirects to another portal web page.  This works fine but when we de-activate the redirect and carry out a portal cache refresh the redirect still happens.  Is this by design?

Thanks