API caching strategies and best practices

What is API caching?

API caching is a process that places commonly requested objects in a secondary data store to avoid continuous calls to a primary database or any other type of data store. A cache's primary advantage is processing speed, as it enables an application to fetch commonly requested objects from sources that it can access efficiently.

Choosing between a primary data store and a cache comes down to speed vs. size. Data in a primary database might have more structure and searchability, but it can still be harder to access than data in a dedicated cache.

Why API caching is important

API caching can often be an inexpensive way to improve performance. By using tools that store the most commonly requested data from an application, developers can reduce the load on the application and speed up requests for most users.

Caching can also be a rudimentary stopgap to manage scaling issues, since most requests will ideally be answered by the cache and a large amount of traffic gets offloaded from directly hitting application servers. In the event of an underlying application failure, a well-implemented cache can also help avoid downtime by at least serving stale data, rather than serving nothing at all.

Types of API caching strategies

There are several well-known API caching strategies, including cache-aside, read-through, write-through, write-back and write-around.

Cache-aside

The most common caching strategy, cache-aside, uses the application itself to manage the cache. If there is a cache miss, the application retrieves the value from the database and inserts it into the cache. On writes of new data, the application stores the data in the database and simply invalidates any cache entry for that data. This strategy is also known as lazy loading because nothing saves to the cache until it's requested from the application. Lazy loading ensures that the cache contains the most requested data.

Read-through

In a read-through cache setup, the cache sits between the application and the database. When data is requested from the application, the application makes a request to the cache. If there is no entry for the data, the cache fetches the value from the database and returns it to the application, saving it in the cache along the way.

Write-through

The write-through strategy is similar to read-through. When data is written to the application, the cache handles writing the data to the database, caching the value that is written to the database.

Write-back

In the write-back strategy, the application writes data to the cache and then the cache asynchronously writes to the database. The idea behind this caching strategy is to prioritize write performance by caching writes, since the cache is responsible for writing new data to the database. Data is read by the application through the cache, so data is guaranteed to be up to date even if the cache hasn't yet updated the database.

Write-around

Write-around caching refers to the application writing directly to the database rather than through the cache. The idea behind the write-around strategy is that data written to a cache on writes might simply go unused. Depending on the context of the data and the way the application works, writing to the cache only on reading data might be much more efficient.

API caching best practices to know

Development teams should keep these best practices in mind when implementing API caching.

1) Establish a time-to-live (TTL)

TTL is an incredibly important setting for caching tools. Defining a TTL is simply choosing a duration of time for how long a cache entry is valid before it expires and must be re-read from the database. The duration of the TTL depends on the application and the data involved. For example, for product inventory, it would be better to have a shorter TTL that would expire more often and ensure the cache is updated. For data that isn't often updated, like pricing or user profile information, a longer TTL can be used.

2) Determine baseline performance

When integrating API caching into an application, developers should understand performance benchmarks, specifically to compare an application's performance with caching against its performance without caching enabled.

To begin, developers can create load tests targeted at API requests using tools such as Apache JMeter or Locust. These two open source tools enable developers to scale the number of API requests to simulate various request loads from different types of users. The results of these early load tests can provide an upfront benchmark of the application's performance.

A computer's network bandwidth, latency and processing power can have a significant effect on the amount of request loads generated. Developers must keep this in mind when comparing load test results, as the results from one run to another might not prove a valid comparison. To avoid these discrepancies, consider adding a cloud-based load testing tool that uses stable, isolated servers that offer consistent network bandwidth and latency. BlazeMeter or CloudTest are some examples of tools that can do this.

3) Run test scenarios for requests

After getting a baseline benchmark, developers can implement caching and reassess the application's performance. Ideally, the application's ability to handle load under stress should improve -- and, hopefully, its overall performance will too. Regardless of performance, however, teams should also validate the responses that requests return to ensure the cache is behaving properly.

One way to confirm this is to create test scenarios that check for updated values, which developers can run in just a few steps. For example:

1. Configure a group of requests to specifically use the application's cache exclusively.
2. Make an update to a value located within the application's primary database.
3. Send a request to the application after the cache's expected expiration time to validate that the updated value was returned.

Depending on cache implementation, developers can also run test scenarios to validate certain features.

4) Use key-value stores

Many open source caching tools -- like Memcached -- use a key-value approach to fill the cache in memory as requests come through. Before a value exists in the cache, an application checks the cache for the specified key, which identifies the objects to return as part of the response.

If no key is present in the cache, the tool will query a database and provide a response along with the expected key for the cache to use. Subsequent requests for that same key won't require a query to the database, as they are now stored in the cache.

5) Avoid the thundering herd problem

Imagine there are 10 servers, each serving the same webpage application. The webpage is stored in a cache, with the cache set to expire every five minutes to ensure users consistently see the most recent version of the page. The cache could expire while those 10 servers are under heavy load, leading each server to simultaneously query the cache, find no webpage and attempt to directly access the primary database.

Caching under a heavy load like this -- particularly in a distributed system -- can lead to the so-called thundering herd problem. Allowing 10 servers to query the database at once creates a heavy load, and a computationally intense query could easily cause a cascading number of requests to time out as the database continues to struggle. Furthermore, when those failed requests retry, they'll continue to put even more load on the database and potentially render the application useless.

Fortunately, there are a few ways to avoid a thundering herd scenario. For one, lock the cache to ensure only one process can update the cache at a time. With the lock in place, applications trying to update the cache can use previously stored values until the update is complete. Developers can also use an external process to update the cache rather than relying on the application itself.

Another useful way to avoid a thundering herd is to update the cache expiration to a predicted value as the cache's expiration time nears. In a case like this, applications that rely on the cache can also calculate expected expiration times and provide a better way to ensure they don't all expire at once.

6) Design with security in mind

Because caching is storing data, it's important to consider the security implications when using caching tools. Any type of personally identifiable information shouldn't be cached as a general rule. Access to the cache should also be restricted, as it would be a vulnerability to a possible attack.

If an attacker can access the cache, they could insert malicious data that could then be used to compromise user accounts or other sensitive data. There should be specific controls in place to validate any data inserted into the cache to prevent these types of attacks. The cache should only be accessible by the application or database required for it to function.

Matt Grasberger is a DevOps engineer with experience in test automation, software development and designing automated processes to reduce work.