Protect against resource exhaustion

Vault is an API driven system that all communication between the clients and Vault are done through Vault API.

As the number of client applications increases, rogue applications can degrade Vault performance. The issue is often caused by applications that are:

• Generating an unbounded number of leases and/or loading too much data into Vault leading to exhausting Consul storage backend's available memory
• Consuming an erroneously large amount of bandwidth leading to internal throttling of the Vault cluster as a whole

Solution

Set Vault resource quotas to protect your Vault environment's stability and network, as well as storage resource consumption from runaway application behavior and distributed denial of service (DDoS) attack.

The Vault operators can control how applications request resources from Vault, and Vault's storage and network infrastructure by setting the following:

To set the rate limit quotas and lease count quotas, use the sys/quotas/<type> endpoint. Each resource quota has a name to identify the quota rule. To manage an individual quota rule, the endpoint becomes sys/quotas/<type>/<name>.

The <type> can be:

• rate-limit: rate limit quota
• lease-count: lease count quota

Prerequisites

To perform the tasks described in this tutorial, you need Vault v1.5 or later. Refer to the Vault install guide to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Lab setup

1. Open a terminal and start a Vault dev server with root as the root token.

   $ vault server -dev -dev-root-token-id root
   

   The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.

   Insecure operation

   Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

2. Export an environment variable for the vault CLI to address the Vault server.

   $ export VAULT_ADDR=http://127.0.0.1:8200
   

3. Export an environment variable for the vault CLI to authenticate with the Vault server.

   $ export VAULT_TOKEN=root
   

The Vault server is ready.

Resource quota configuration

To configure the resource quota, use the sys/quotas/config endpoint.

By default, the requests rejected due to rate limit quota violations are not written to the audit log. Therefore, if you wish to log the rejected requests for traceability, you must set the enable_rate_limit_audit_logging to true. The requests rejected due to reaching the lease count quotas are always logged that you do not need to set any parameter.

Rate limit quotas

Rate limit quotas are designed to protect Vault against external distributed denial of service (DDoS) attacks and are fundamental to Vault's security model. Therefore, it is a part of Vault's core feature set available in both Community and Enterprise editions.

To set rate limit quotas, use the sys/quotas/rate-limit/<name> endpoint.

Parameters

Lease count quotas

Lease count quota is designed to protect Vault from a large volume of leases and tokens persisted in Vault, which can pressure its storage backend. This acts as a guard rail for system stability in large-scale Vault Enterprise deployments where Vault is used as a service.

Parameters

The following examples set lease count quotas on a speific scope. Vault Enterprise 1.16 introduced a default lease count quota that you can set on a global level. The global default is available for new Vault installations. To learn more, go through the global default lease count quota section.

Global default lease count quota

If you are running Vault Enterprise 1.16 or later, consider setting a global lease count quota with a reasonably low default. This would provide an earlier signal to properly tune the lease limits so that unexpected lease explosion issue can be prevented. This approach would reduce the likelihood of customers accidentally imposing a later untenable load.

Vault Enterprise namespaces

When you are working with Vault Enterprise namespaces, you can use the inheritable parameter to apply the resource quota to subsequent child namespaces.

Think of the following namespace hierarchy:

root
   └── parent
      └── child
         └── grand-child

Under the root namespace, you have a parent namespace, and then parent/child and parent/child/grand-child namespaces.

You can set the resource quota on the parent namespace which gets applied to its child namespaces inheritably by setting the inheritable parameter to true. By default, it is set to false.

The us-west namespace and its children are now restricted to 500 requests per minute instead of what is set by the global-quota rule.

To disable the inheritance, set the inheritable parameter to false.

Test to understand the resource quotas

Now that you learned the basic commands, test to see how it behaves.

Rate limit quota test

1. Enable transit secrets engine if it is not enabled.

   $ vault secrets enable transit
   

2. Create a "test" encryption key.

   $ vault write -f transit/keys/test
   

3. For the purpose of demonstration, create a "test-transit" rate limit quota such that you can only make 1 requests per minute.

   $ vault write sys/quotas/rate-limit/rate-test \
      path=transit \
      rate=1 \
      interval=1m
   

4. Create a shortcut script, test-encryption.sh which makes request to encrypt data using "test" key.

   $ tee test-encryption.sh <<EOF
   echo "Request 1"
   vault write transit/encrypt/test plaintext=$(base64 <<< "4111 1111 1111 1111")
   echo "\nRequest 2"
   vault write transit/encrypt/test plaintext=$(base64 <<< "4222 2222 2222 2222")
   echo "\nRequest 3"
   vault write transit/encrypt/test plaintext=$(base64 <<< "4333 3333 3333 3333")
   EOF
   

5. Ensure that the script is executable.

   $ chmod +x test-encryption.sh
   

6. Run the script to see how the quota rule behaves.

   $ ./test-encryption.sh
   

   Output:

   Request 1
   Key            Value
   ---            -----
   ciphertext     vault:v1:6zY2JVtuw3r11ST/JjMkmoNbbcCpQwGbJJ0BJfByk//GLQof8mcxAoerq5Y++BZK
   key_version    1
   
   Request 2
   Error writing data to transit/encrypt/test: Error making API request.
   
   URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
   Code: 429. Errors:
   
   * request path "transit/encrypt/test": rate limit quota exceeded
   
   Request 3
   Error writing data to transit/encrypt/test: Error making API request.
   
   URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
   Code: 429. Errors:
   
   * request path "transit/encrypt/test": rate limit quota exceeded
   

   The second and third requests failed because of the limit.

7. Earlier, you configured the resource quotas to enable audit logging of requests which are rejected due to rate limit quota rule violation. Inspect your audit log for its entry.

   $ more /var/log/vault-audit.log | jq
   

   If your audit log path is not /var/log/vault-audit.log, be sure to set it to the correct path.

   ...snip...
      "request": {
         "id": "4170a181-6a18-589b-f026-a62ad08ae644",
         "operation": "update",
         "namespace": {
            "id": "root"
         },
         "path": "transit/encrypt/test",
         "data": {
            "plaintext": "hmac-sha256:50978c7cb3f0f463491a26152353f485b579d1e8759ec7b7b7c28f3c5b9232cd"
         },
         "remote_address": "127.0.0.1",
         "remote_port": 61028
      },
      "error": "request path \"transit/encrypt/test\": rate limit quota exceeded"
   }
   

   You should find an error message indicating that rate limit quota was exceeded. You can trace the audit log to learn the number of requests which are rejected due to the rate limit quota. It may be working as expected or you may find suspicious activities against a specific path.

8. If you are running Vault 1.12.0 or later, create another rate limit quota to specify the path to transit/encrypt/test where allowed rate is 2 with interval of 1 minute.

   $ vault write sys/quotas/rate-limit/encryption-limit \
      path="transit/encrypt/test" \
      rate=2 \
      interval=1m
   

9. Run the script again to see how the quota rule behaves.

   $ ./test-encryption.sh
   

   Output:

   Request 1
   Key            Value
   ---            -----
   ciphertext     vault:v1:44sOnzePg8c0XaxzGeGzwJ94UZ1uuif8p0nGuze8utbvWVhqNHDiIwo8VZmPrcli
   key_version    1
   
   Request 2
   Key            Value
   ---            -----
   ciphertext     vault:v1:w9tyXqoXH6uXuOWUktVWP/yrzOOZDv+TrRbbRU5GZEVappASIRnh3+6KwkJkjWKE
   key_version    1
   
   Request 3
   Error writing data to transit/encrypt/test: Error making API request.
   
   URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
   Code: 429. Errors:
   
   * request path "transit/encrypt/test": rate limit quota exceeded
   

   Note

   This time, only the last request failed. When a more granular path is set, the rate limit quota rule takes precedence against the path.

   If you have another key or try to decript (transit/decrypt/test), the rate-test rate limit quota will be applied.

Lease count quota test

1. Similarly, create a limited lease count quota named, "lease-test" which applies on the root level. It only allows 3 tokens and leases to be stored.

   $ vault write sys/quotas/lease-count/lease-test \
      path="auth/token" \
      max_leases=3
   

2. Create a shortcut script, test-token-create.sh which attempts to create tokens 4 times.

   $ tee test-token-create.sh <<EOF
   echo "Request #1"
   vault token create -policy=default
   echo "\nRequest #2"
   vault token create -policy=default
   echo "\nRequest #3"
   vault token create -policy=default
   echo "\nRequest #4"
   vault token create -policy=default
   EOF
   

3. Make sure that the script is executable.

   $ chmod +x test-token-create.sh
   

4. Run the script to see how the quota rule behaves.

   $ ./test-token-create.sh
   

   Three tokens were created successfully; however, the fourth request failed due to the lease count quota. Your output should look similar to follow.

   Request #1
   Key                  Value
   ---                  -----
   token                hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
   token_accessor       rhmyha9JjSxh9hkjT2i2i8An
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #2
   Key                  Value
   ---                  -----
   token                hvs.CAESIE0L_nNkTLI3cLayb0NpyEw2gc3x6-pjVIpoYhfyyOBEGh4KHGh2cy5xSjhhNWRaVnp5bzF6V2FqUE44emZUZk8
   token_accessor       oKKZViZet2tTl8TKqnbYMplX
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #3
   Key                  Value
   ---                  -----
   token                hvs.CAESIP53xCUOcY0aJplpxagixzxZskdry6z1YUrYx5Urzk6eGh4KHGh2cy5UWGl0RTR2WDhLTUFFMGtLRnJJTjlBcHc
   token_accessor       gtM8SoUEsaPkf9TvBLk5dswU
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #4
   Error creating token: Error making API request.
   
   URL: POST http://127.0.0.1:8200/v1/auth/token/create
   Code: 429. Errors:
   
   * 1 error occurred:
      * request path "auth/token/create": lease count quota exceeded
   

   Note

   If your Vault server already has client tokens, the lease count quota may be exceeded sooner.

   Also, you can find the lease count quota exceeded error in the audit log. (Be sure to set it to the correct audit log path for your environment.)

   $ tail -f /var/log/vault-audit.log | jq
   
   ...snip...
     "response": {
       "mount_type": "token"
     },
     "error": "1 error occurred:\n\t* request path \"auth/token/create\": lease count quota exceeded\n\n"
   }
   

5. If you revoke one of the tokens, you should be able to request a new one.

   Example:

   $ vault token revoke hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
   Success! Revoked token (if it existed)
   

   Now, request a new one.

   $ vault token create -policy=default
   

   The best practice is to set the tokens and leases' time-to-live (TTL) to be short and don't let them hang around longer than necessary. The lease count quotas allow you to set the upper limit to protect your Vault environment from running into an issue due to a lack of token and lease governance.

6. Delete the lease-test quota rule.

   $ vault delete sys/quotas/lease-count/lease-test
   

Vault 1.12.0 or later

If you are running Vault 1.12.0 or later, you can set more granular lease count quota.

1. To demonstrate, enable approle auth method if it is not already enabled.

   $ vault auth enable approle
   

2. Create a role named "test-role".

   $ vault write auth/approle/role/test-role \
      token_policies="default" \
      token_ttl=1h \
      token_max_ttl=4h
   

3. Create a lease count quota to limit the max number of leases for the test-role to 2.

   $ vault write sys/quotas/lease-count/test-role-limit \
      max_leases=2 \
      path="auth/approle" \
      role="test-role"
   

4. Retrieve the role ID of the test-role and store it in role_id.txt.

   $ vault read -field=role_id auth/approle/role/test-role/role-id > role_id.txt
   

5. Generate a secret ID of the test-role and store it in secret_id.txt.

   $ vault write -force -field=secret_id auth/approle/role/test-role/secret-id > secret_id.txt
   

6. Create a test script.

   $ tee test-approle-login.sh <<EOF
   echo "Login attempt #1"
   vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
   echo "\nLogin attempt #2"
   vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
   echo "\nLogin attempt #3"
   vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
   EOF
   

7. Ensure that the script is executable.

   $ chmod +x test-approle-login.sh
   

8. Run the script to see how the quota rule behaves.

   $ ./test-approle-login.sh
   

   Output:

   Login attempt #1
   Key                     Value
   ---                     -----
   token                   hvs.CAESICjr1nqxXo4UDJyh9MkMwvecOyEFlE69Pgs7ftdmUvF4Gh4KHGh2cy4yN3Q1WXRWU0tpSlBsQWpkZGJOdXpMZHY
   token_accessor          1P7KhrLmdrWRFIPR6IJnR6Ls
   token_duration          1h
   token_renewable         true
   token_policies          ["default"]
   identity_policies       []
   policies                ["default"]
   token_meta_role_name    test-role
   
   Login attempt #2
   Key                     Value
   ---                     -----
   token                   hvs.CAESIGm6g-GUUPCJq3zvs_L1f1RIB4-7RD7a-yQxUaphFi0qGh4KHGh2cy5IVHluNVlEclZrSHNXb3lOZGVraWdmN20
   token_accessor          ThLLvtE42QE0wMbAkTEvOA8N
   token_duration          1h
   token_renewable         true
   token_policies          ["default"]
   identity_policies       []
   policies                ["default"]
   token_meta_role_name    test-role
   
   Login attempt #3
   Error writing data to auth/approle/login: Error making API request.
   
   URL: PUT http://127.0.0.1:8200/v1/auth/approle/login
   Code: 429. Errors:
   
   * 1 error occurred:
      * request path "auth/approle/login": lease count quota exceeded
   

   The last attempt failed. The test-role-limit lease count quota should not affect other AppRole roles.

Vault Enterprise namespaces test

To test, create namespace hierarchy: us-west, us-west/california, and us-west/california/san-francisco.

root
└── us-west
   └── california
      └── san-francisco

1. Create california namespace as a child of the us-west namespace.

   $ vault namespace create -namespace="us-west" california
   Key                Value
   ---                -----
   custom_metadata    map[]
   id                 dUR8y
   path               us-west/california/
   

2. Create san-francisco namespace as a child of the us-west/california namespace.

   $ vault namespace create -namespace="us-west/california" "san-francisco"
   Key                Value
   ---                -----
   custom_metadata    map[]
   id                 InIeR
   path               us-west/california/san-francisco/
   

3. Create a rate limit quota named, "us-west" which limits 3 requests per minute on the us-west namespace and make it inheritable.

   $ vault write sys/quotas/rate-limit/us-west \
      path="us-west" \
      rate=3 \
      interval=1m \
      inheritable=true
   

4. Set the target namespace to us-west/california.

   $ export VAULT_NAMESPACE="us-west/california"
   

5. Run the test-token-create.sh script to test the quota.

   $ ./test-token-create.sh
   

   The rate limit placed onto the us-west namespace is inherited by its child namespaces; therefore, you can make 3 requests per minute on the us-west/california namespace as well.

   Request #1
   Key                  Value
   ---                  -----
   token                hvs.CAESIOw0mxz1DkoQvEdb77rUkEXFvlaFELTZ8WoEJognD9IxGiQKImh2cy5MRFZWa2N3azB0bTVDZVJTVXd0Z1EzTEEuV2xFZ3I
   token_accessor       CTomum1fi0XaQgaXDXA91aKF.WlEgr
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #2
   Key                  Value
   ---                  -----
   token                hvs.CAESIENZqKVA8TEE6wA_Vr9e6I-vlR-lBFDtMSL7zrOmyrqwGiQKImh2cy5WQmxaUkdZbnB0STJYQVUzUzFJMHJQT3ouV2xFZ3I
   token_accessor       O8WnsqrLwnDumWndb0VUKyQs.WlEgr
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #3
   Key                  Value
   ---                  -----
   token                hvs.CAESIFH-IjHLpeM6FT_FgVkxDZXJqocGSBhYoJRSboUSaT-xGiQKImh2cy5OWjlodGMwRUZRa3NOcW04OEJZMTd4TXAuV2xFZ3I
   token_accessor       b9s5wrRNuvKQa6FhOmcENOuA.WlEgr
   token_duration       768h
   token_renewable      true
   token_policies       ["default"]
   identity_policies    []
   policies             ["default"]
   
   Request #4
   Error creating token: Error making API request.
   
   Namespace: us-west/california/
   URL: POST http://127.0.0.1:8200/v1/auth/token/create
   Code: 429. Errors:
   
   * request path "auth/token/create": rate limit quota exceeded
   

6. To test against the us-west/california/san-francisco namespace, update the VAULT_NAMESPACE environment variable.

   $ export VAULT_NAMESPACE="us-west/california/san-francisco"
   

7. Run the script to verify that the quota rule is inherited.

   $ ./test-token-create.sh
   

   The forth request fails due to quota.

   ...snip...
   
   Request #4
   Error creating token: Error making API request.
   
   Namespace: us-west/california/san-francisco/
   URL: POST http://127.0.0.1:8200/v1/auth/token/create
   Code: 429. Errors:
   
   * request path "auth/token/create": rate limit quota exceeded
   

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

1. Unset the VAULT_TOKEN, VAULT_ADDR, and VAULT_NAMESPACE environment variables.

   $ unset VAULT_TOKEN VAULT_ADDR VAULT_NAMESPACE
   

2. Delete files created during the test.

   $ rm test-encryption.sh test-token-create.sh test-approle-login.sh role_id.txt secret_id.txt
   

3. If you are running Vault locally in -dev mode, you can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.

   $ pgrep -f vault | xargs kill
   

Next steps

In this tutorial, you learned the basic commands to set resource quotas to protect your Vault environment. To leverage this feature, you need Vault 1.5 or later.

Rate limit quotas allow Vault operators to set inbound request rate limits which can be set on the root level or a specific path. This is available in both Vault Community Edition and Vault Enterprise.

Lease count quotas require Vault Enterprise Platform and allow operators to set the maximum number of tokens and leases to be persisted at any given time. This can prevent Vault from exhausting the resource on the storage backend.

You also learned that audit logging can be enabled to trace the number of requests that were rejected due to the rate limit quota.

Help and Reference

• Resource Quotas Concepts doc
• Rate limit quota API doc
• Lease count quota API doc