During testing we discovered the following issues:

• If you edit a region to another it won't generate new keys.
• It did not successfully clean up an old key pair.

I lumped these together as the fix to one will probably fix the other. Maybe there should be an option to clean up all keys and regenerate them for all current compute resources.