Configuration → User
You can restrict every control plugin individually by applying a permission class to it. Simply select one of your permission classes for the plugin, the changes become effective immediately.
Hint: To completely disable a control plugin, create a permission class with no rights at all and assign it to the control plugin.