The basics of polyglot persistence for microservices data

What is polyglot persistence?

When an application runs, it consumes existing data, creates new data and saves data for later use. If data isn't needed immediately, it can be temporarily stored in a cache layer, or in memory. The data may expire when the application completes the session or if it shuts down. However, in many cases, it's necessary to store the data for later use. In this case, data can be stored in a database or disk that is independent of the application.

Microservices lets you handle data storage as a separate service. Also, you can store data in multiple formats, which helps in certain business cases where not all storage options are practical. For example, if you run an e-commerce website and need to show real-time search suggestions, this is a feature that requires a purpose-built, full-text search engine, and it requires a different storage type. Additionally, this search engine should coexist alongside other routine relational databases. This is polyglot persistence: the coexistence of multiple storage types.

Traditionally, enterprises have designed their monolithic apps to rely exclusively on relational databases. This meant that all data received the same level of performance regardless of its unique need. The rise of the NoSQL movement and the big data revolution challenged this point of view.

As large web companies discovered that their relational databases could no longer support huge data volumes, companies like Google turned to Hadoop, Hbase and other big data processing engines for petabyte-scale data processing. The NoSQL movement spawned many types of niche databases, such as document stores, columnar databases and full-text databases. As the number of database options exploded and microservices gained momentum, so rose the profile of polyglot persistence.

The advantages of polyglot persistence

There are many reasons to adopt a polyglot database approach. First, it provides the flexibility to choose a different database for a different need, such as sub-second latency or big data analysis. It also adds reliability to your applications. CRUD operations can be difficult in relational databases because it can be cumbersome to constantly place new entries over old ones. Modern NoSQL databases focus on the 'create' and 'read' parts of data analysis, and are efficient when they store new data, which can ensure really fast data analysis.

While there are many advantages to polyglot persistence, it takes a keen eye to identify where this approach works best.

Polyglot persistence also commoditizes data management. Rather than scale up with additional expensive servers, modern databases scale out with cheap hardware and make up for it with an intelligent and powerful data processing layer. These databases expect hardware instances to fail, and so it ensures data is always available regardless of the underlying instances' condition.

Microservices are distributed by default. The supporting data architecture should handle data flow between disparate parts of the system and have well-defined rules that resolve conflicts and inconsistencies when they arise.

Applying polyglot persistence in the real world

While there are many advantages to polyglot persistence, it takes a keen eye to identify where this approach works best. Consider these suggestions when you choose a database.

• MySQL, Oracle Database - Relational databases are still useful for data that changes infrequently and is critical to the business. They can be hard to store and scale, and are expensive to maintain. However, it's easy to find management talent as it's the oldest of the data storage types.
• Redis, Aerospike - A key-value database works best for large, and static, quantities of data. It's great for archival storage of things like log data, and it scales easily.
• MongoDB, CouchDB - This is the most popular type of NoSQL database and is a document store. Use this for data that changes frequently and doesn't need strict schema. Think product descriptions and categories on an e-commerce site.
• Elasticsearch - This is a full-text search engine database that often powers search boxes on e-commerce sites. Use this to analyze primarily text-based log data.
• InfluxDB, Prometheus - Time-series databases are built to ingest streaming data in real time and render for easy data analysis. They are used in monitoring apps and stock ticker apps.
• Neo4j - This is a graph database than can analyze the connection between various data points. A typical use is for fraud detection as it easily spots suspicious activity across large data samples.
• MariaDB, Apache Hbase - A columnar database focuses on speedy reads and writes of data with efficient data storage.

Relational storage still has a place

A word of caution about polyglot persistence - relational databases may seem old school but sometimes it's all you need. In some cases, scale is not important, or you can tolerate some latency, so a simple relational database management system (RDBMS) may be enough. Relational databases are great for applications that handle company financial data, employee data, health records and insurance data. But look for ways to introduce other databases. For most modern cloud-native and mobile-first applications, it's best to use microservices with polyglot persistent storage.