In this codelab, you'll learn how to use some advanced features of Google Cloud Security and Privacy APIs, including:

You'll take abridged data from the Enron Corpus, encrypt it and load it into Cloud Storage.

What you'll learn

What you'll use

Self-paced environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one. Sign-in to Google Cloud Platform console ( and create a new project:

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Developers Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document). KMS pricing is documented here.

New users of Google Cloud Platform are eligible for a $300 free trial.

Codelab-at-a-conference setup

The instructor will be sharing with you temporary accounts with existing projects that are already setup so you do not need to worry about enabling billing or any cost associated with running this codelab. Note that all these accounts will be disabled soon after the codelab is over, but you can always do it again at home with your own account.

Once you have received a temporary username / password to login from the instructor, log into Google Cloud Console:

Here's what you should see once logged in :

Note the project ID you were assigned ( "codelab-test003" in the screenshot above). It will be referred to later in this codelab as PROJECT_ID.

Google Cloud Shell

While Google Cloud and KMS can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:

IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

In order to store the data in this codelab, you'll need your own Cloud Storage bucket. Pick a name for it, such as enron_corpus.


Create the bucket by running the following commands in your shell:

gsutil mb gs://${BUCKET_NAME}

Cloud Storage bucket names have to be unique across all buckets, so change YOUR_NAME above to your own name or some other unique identifier. For more information, see the Cloud Storage bucket naming guidelines.

After your bucket has been created, you are ready to go. Running this command should also help to verify that you've got the gsutil command line client setup correctly, authentication is working, and you have write access to the cloud project you're operating under.

The Enron Corpus is a large database of over 600,000 emails generated by 158 employees of the Enron Corporation. This data has been copied to the GCS bucket gs://enron_corpus/.

Let's download one of the source files locally so that we can see what it looks like. Run the command to copy down an email text file:

gsutil cp gs://enron_corpus/allen-p/inbox/1. .

Now tail the downloaded file to verify the email text is there:

tail 1.

This should display the contents of the plaintext mail file. There are two types of files we'll be looking at for this codelab: plaintext mail files and image files. If you're interested, use the same mechanism to check out what is in those other files.

Before you can use KMS, you need to enable KMS in your project. This can be done in the using the gcloud command-line utility. Run the following command in your shell:

gcloud service-management enable

Cloud KMS is now enabled for your project!

In order to encrypt the data, you need to create a KeyRing and a CryptoKey. KeyRings are useful for grouping keys. Keys can be grouped by environment (like test, staging, and prod) or by some other conceptual grouping. For this codelab, our KeyRing will be called test and our CryptoKey will be called codelab.


Execute the gcloud command to create the KeyRing. You will specify the location as global, but this can also be a specific region.

gcloud beta kms keyrings create $KEYRING_NAME --location global

Next, using the new KeyRing, create a CryptoKey named codelab.

gcloud beta kms cryptokeys create $CRYPTOKEY_NAME --location global \
      --keyring $KEYRING_NAME \
      --purpose encryption

You've created a KeyRing and CryptoKey! Open the Encryption Keys Web UI and view the resources you've created. This is available under IAM & Admin in the main menu:

The Key Management Web UI allows you to view and manage your CryptoKeys and KeyRings. You will use this UI later when you manage permissions.

Next, let's actually try to encrypt some data! Take the contents of the email we previously looked at and base64 encode it.

PLAINTEXT=$(cat 1. | base64 -w0)

Using the encrypt endpoint, you can send the base64-encoded text you want to encrypt to the specified key. The response will be a JSON payload containing the encrypted text in the attribute ciphertext.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type: application/json"

Now that your data is encrypted, you can save it to a file and upload it to your GCS bucket. To grab the encrypted text from the JSON response and save it to a file, you will use the command-line utility jq. The response from the previous call can be piped into jq, which can parse out the ciphertext property to the file 1.encrypted.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .ciphertext -r > 1.encrypted

To verify the encrypted data can be decrypted, call the decrypt endpoint to verify the decrypted text matches the original email. The encrypted data has information on which CryptoKey version was used to encrypt it, so the specific version is never supplied to the decrypt endpoint.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:decrypt" \
  -d "{\"ciphertext\":\"$(cat 1.encrypted)\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .plaintext -r | base64 -d

Now that you have verified the text has been encrypted successfully, upload the encrypted file to your GCS bucket.

gsutil cp 1.encrypted gs://${BUCKET_NAME}

In KMS, there are two major permissions to focus on. One permissions allows a user or service account to manage KMS resources, the other allows a user or service account to use keys to encrypt and decrypt data.

The permission to manage keys is cloudkms.admin, and allows anyone with the permission to create KeyRings and create, modify, disable, and destroy CryptoKeys. The permission to encrypt and decrypt is cloudkms.cryptoKeyEncrypterDecrypter, and is used to call the encrypt and decrypt API endpoints.

For this exercise, you will use the current authorized user to assign IAM permissions. To get the current authorized user, run the command below.

USER_EMAIL=$(gcloud auth list --limit=1 2>/dev/null | grep '@' | awk '{print $2}') 

Next, you will assign that user the ability to manage KMS resources. Run the following gcloud command to assign the IAM permission to manage the KeyRing you just created:

gcloud beta kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.admin

Since CryptoKeys belong to KeyRings, and KeyRings belong to Projects, a user with a specific role or permission at a higher level in that hierarchy inherits the same permissions on the child resources. That is, a user who has the role of owner on a Project is also an owner on all the KeyRings and CryptoKeys in that project. Similarly, if a user is granted the cloudkms.admin role on a KeyRing, they have the associated permissions on the CryptoKeys in that KeyRing.

Without the cloudkms.cryptoKeyEncrypterDecrypter permission, the authorized user will not be able to use the keys to encrypt or decrypt data. Run the following gcloud command to assign the IAM permission to encrypt and decrypt data for any CryptoKey under the KeyRing you created:

gcloud beta kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Now you can view the assigned permissions in the Key Management Web UI. Click the three vertical dots by the KeyRing and select Edit Permissions.

This will open up a menu where you can see the accounts and permissions for the KeyRing you just added.

Now that you have an understanding of how to encrypt a single file, and have permission to do so, you can run a script to backup all files in a directory. For this example, we will copy all emails for allen-p, encrypt them, and upload them to a Cloud Storage bucket. First, copy all emails for allen-p into your current working directory:

gsutil -m cp -r gs://enron_corpus/allen-p .

Now copy and paste the following into the Cloud Shell terminal to backup and encrypt all the files in the allen-p directory to your Cloud Storage bucket:

FILES=$(find $MYDIR -type f -not -name "*.encrypted")
for file in $FILES; do
  PLAINTEXT=$(cat $file | base64 -w0)
  curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
    -d "{\"plaintext\":\"$PLAINTEXT\"}" \
    -H "Authorization:Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type:application/json" \
  | jq .ciphertext -r > $file.encrypted
gsutil -m cp *.encrypted gs://${BUCKET_NAME}

This script loops over all the files in a given directory, encrypts them using the KMS API, and uploads them to Google Cloud Storage.

After the script completes, you can view the encrypted files in the Google Cloud Storage UI. You should see something like this:

Google Cloud Audit Logging consists of two log streams, Admin Activity and Data Access, which are generated by Google Cloud Platform services to help you answer the question of "who did what, where, and when?" within your Google Cloud Platform projects.

To view the activity for any resource in KMS, Click the three vertical dots by the KeyRing and select View Activity.

This will take you to the Cloud Activity UI, where you should see the creation and all modifications made to the KeyRing.

You've now encrypted and uploaded data using KMS and Cloud Storage!


Let's release compute resources created during the lab. If you no longer need it, go ahead and delete the Cloud Storage bucket you created in the beginning of the lab.

gsutil rm -r gs://${BUCKET_NAME}

As KMS resources (CryptoKeys and KeyRings) are permanent, there is no way to remove them. However, you can destroy the CryptoKey version used in this project so it cannot be used again:

gcloud beta kms cryptokeys versions destroy 1 \
  --location=global \
  --cryptokey=$CRYPTOKEY_NAME \

What we've covered

Next Steps

Learn More

Give us your feedback