Multi-tenant "Application User app" security concern

I am building an app to be distributed on the App Source. I need to authenticate to the Dynamics instance of my clients to access their data. I currently do this with a username and a password.

I would like to allow the Application User functionality available with CRM 8.2. This means an easier configuration for my clients.

My concern is: when configuring my app, the user has to:

1. Provide Admin consent.
2. Configure the URL of his CRM instance.

The problem is: if client A knows that client B uses my app, client A could set the URL of the CRM instance to the one of client B and steal its data this way.

How do you guys cope with this?