Browse Definitions :
Definition

streaming data architecture

A streaming data architecture is an information technology framework that puts the focus on processing data in motion and treats extract-transform-load (ETL) batch processing as just one more event in a continuous stream of events. This type of architecture has three basic components -- an aggregator that gathers event streams and batch files from a variety of data sources, a broker that makes data available for consumption and an analytics engine that analyzes the data, correlates values and blends streams together.

The system that receives and sends data streams and executes the application and real-time analytics logic is called the stream processor. Because a streaming data architecture supports the concept of event sourcing, it reduces the need for developers to create and maintain shared databases. Instead, all changes to an application’s state are stored as a sequence of event-driven processing (ESP) triggers that can be reconstructed or queried when necessary. Upon receiving an event, the stream processor reacts in real- or near real-time and triggers an action, such as remembering the event for future reference.

The growing popularity of streaming data architectures reflects a shift in the development of services and products from a monolithic architecture to a decentralized one built with microservices. This type of architecture is usually more flexible and scalable than a classic database-centric application architecture because it co-locates data processing with storage to lower application response times (latency) and improve throughput. Another advantage of using a streaming data architecture is that it factors the time an event occurs into account, which makes it easier for an application’s state and processing to be partitioned and distributed across many instances.

Streaming data architectures enable developers to develop applications that use both bound and unbound data in new ways. For example, Alibaba’s search infrastructure team uses a streaming data architecture powered by Apache Flink to update product detail and inventory information in real-time. Netflix also uses Flink to support its recommendation engines and ING, the global bank based in The Netherlands, uses the architecture to prevent identity theft and provide better fraud protection. Other platforms that can accommodate both stream and batch processing include Apache Spark, Apache Storm, Google Cloud Dataflow and AWS Kinesis.

This was last updated in October 2018

Continue Reading About streaming data architecture

Networking
  • Multiprotocol Label Switching (MPLS)

    Multiprotocol Label Switching (MPLS) is a switching mechanism used in wide area networks (WANs).

  • computer network

    A computer network is a group of interconnected nodes or computing devices that exchange data and resources with each other.

  • WAN optimization (WAN acceleration)

    WAN optimization -- also known as WAN acceleration -- is a collection of technologies and techniques used to improve the ...

Security
  • three-factor authentication (3FA)

    Three-factor authentication (3FA) is the use of identity-confirming credentials from three separate categories of authentication ...

  • cyber espionage

    Cyber espionage (cyberespionage) is a type of cyber attack that malicious hackers carry out against a business or government ...

  • role-based access control (RBAC)

    Role-based access control (RBAC) is a method of restricting network access based on the roles of individual users within an ...

CIO
  • knowledge-based systems (KBSes)

    Knowledge-based systems (KBSes) are computer programs that use a centralized repository of data known as a knowledge base to ...

  • Sarbanes-Oxley Act

    The Sarbanes-Oxley Act of 2002 is a federal law that established sweeping auditing and financial regulations for public companies.

  • project charter

    A project charter is a formal short document that states a project exists and provides project managers with written authority to...

HRSoftware
  • employee engagement

    Employee engagement is the emotional and professional connection an employee feels toward their organization, colleagues and work.

  • talent pool

    A talent pool is a database of job candidates who have the potential to meet an organization's immediate and long-term needs.

  • diversity, equity and inclusion (DEI)

    Diversity, equity and inclusion is a term used to describe policies and programs that promote the representation and ...

Customer Experience
  • sales development representative (SDR)

    A sales development representative (SDR) is an individual who focuses on prospecting, moving and qualifying leads through the ...

  • service level indicator

    A service level indicator (SLI) is a metric that indicates what measure of performance a customer is receiving at a given time.

  • customer data platform (CDP)

    A customer data platform (CDP) is a type of software application that provides a unified platform of customer information that ...

Close