Updating encryption key of a secret in AWS Secrets Manager gone wrong
I spent more than 24 hours to debug why an update of the encryption key is not working for my secret in AWS Secrets Manager. The final conclusion is just a one-liner, and hence if the read time makes you TLDR, you can jump to the conclusion directly. :)
Quick Intro to AWS Secrets Manager
AWS Secrets Manager helps you store, manage, retrieve and rotate any secret information like database credentials, API key, etc. Secrets Manager uses envelope encryption to protect your secrets.
What is Envelope Encryption?
When you encrypt secret value (DB Credentials) with a key (Key_1/also called the Data Key), you need to protect this key. To protect Key_1, you can encrypt it with another key (Key_2). But you now need to protect Key_2. So, you can encrypt Key_2 with another key (Key_3). You now need to…, well, you get the idea.
This chaining of keys is called envelope encryption.
Eventually, one key must remain in plaintext so you can decrypt the keys and your data. This top-level key is known as the master key.
How Secrets Manager handles your secrets?
Secrets Manager uses a unique data key for each secret value. Each data key is encrypted under a master key. The encrypted data key is stored along with the secret value in the Secrets Manager.
To decrypt the secret, Secrets Manager fetches the decrypted data key (discussed in the next section) and then uses it to decrypt the secret value. The decrypted data key is never persisted by Secrets Manager.
Who generates the data key and the master key?
Introducing AWS Key Management Service (KMS) which lets you generate and store master keys in a highly secured environment. A master key NEVER leaves the AWS KMS space.
AWS KMS also lets you generate data keys using generateDataKey API. It generates a new data key and encrypts it with the master key. In the response, KMS returns both plaintext and an encrypted version of the data key. The client uses the plaintext version to encrypt the secret value and then throws it away. The client must persist the encrypted data key, which will be later used for decryption of secret value.
The encrypted data key, if compromised, will cause no harm. :D
What went wrong when I updated the encryption key of my secret?
Secrets manager allows you to update the encryption key of a secret value, thus changing the master key associated with the secret.
By now, you should have a fundamental understanding of how all this is working. So let us do a step by step walkthrough of what I did.
- Created a new secret (Sec_1) and a corresponding new Master Key (MsKey_1)
- Successfully retrieved the secret value stored in Sec_1
- Deleted MsKey_1 (associated with Sec_1)
- Deleted Sec_1
- Created another Master Key (MsKey_2) (not associated with anything yet)
- Restored Sec_1
- Updated Sec_1 to use MsKey_2 as the encryption key (No other parameter of Sec_1 updated here)
- Attempted to retrieve the secret value stored in Sec_1
- Got error “KMSInvalidStateException: {KeyARN} is pending deletion”
The catch here is, when you change the master key of a secret, Secrets Manager does not re-encrypt the existing secret value under the new master key. However, the next time that the secret value changes, Secrets Manager encrypts it under the new master key.
To emphasize on above, I have explicitly mentioned in Step 7: “No other parameter of Sec_1 updated here”
24 hours for this?
Honestly, all of this now looks pretty obvious to me. So I went back and retrospected why it took me more than 24 hours to debug this:
- Mislead: AWS Secrets Manager console displays the updated Master Key, even when it is NOT using it for encryption; there is no message or WARNING.
- Generic Error Message: Using Secrets Manager console to retrieve the secret in the above state returns a generic error message “Failed to retrieve”. I had to crawl through a lot of CloudTrail logs to figure out what is happening.
- RTFM!!! (I was still going through the in-depth details of the manual, where I eventually found my hidden gem. In my poor defense, when I started working on this, I had a basic understanding of how Secrets Manager and KMS work.)
Conclusion
If you change the master key of a secret, you should also update the secret value. Only then will the updated master key be used for encryption.
And also, RTFM! Always.
Edit 1: My mentor pointed out if updating the encryption key and secret value in a single call will do the trick. Yes, it does; and that saves one extra network call. As expected, the update steps are ordered in AWS Secrets Manager.
Documentation: Modifying a secret in AWS Secrets Manager
Like what you read? From a quick cheer to a standing ovation, clap to show how much you enjoyed this story. Follow to get regular updates.
