Modern Big Data Analytics Architecture

Modernize Your Big Data Architecture for Real-Time Analytics
Share on LinkedIn Tweet about this on Twitter Share on Facebook

With recent advances in cloud computing, virtualization, containerization, continuous integration, and the DevOps movement, deploying software solutions today is very different from even just a few years ago. Today’s software adheres to a set of principles that deliver modern distributed applications

Logi Composer’s architecture is adaptable to work with a wide variety of modern data platforms and architectures. It is purpose-built for visual exploration, responsiveness, and user independence. Logi Composer is about delivering an interactive and visual data experience for the broadest range of users in any organization, not just the BI analysts, data jockeys, SQL experts, and Python programmers.

Deliver Analytics at Scale with Microservices

Logi Composer is a modern microservices based platform that scales and offers a whole host of other benefits.

What are microservices?

Microservices are loosely-coupled mini application servers that self-register with each other. Logi Composer microservices are written in Java.

What are the benefits of using microservices for a modern BI platform?

  • The architecture scales, period. Add more computing nodes as needed to meet demand.
  • It scales efficiently and modularly. Scale and optimize memory allocation and runtime attributes for each microservice based on need. For example, you can scale for more users, or apply more nodes and computing power to a big data Smart Data Connector relative to other connectors.
  • You maximize uptime and recover faster in failure scenarios. Logi Composer microservices can be deployed or restarted on-the-fly while the rest of the application is running. You don’t have to redeploy the whole application if a new data source comes online, or interrupt users if a single connector fails.
  • Modularity eliminates system conflicts. Since each microservice is self-contained, you don’t have to worry about system conflicts with different versions of Java or any other dependency.
  • Modern distributed applications also provide flexibility:
    • Flexibility of choice for deployment — deploy the same set of microservices on-premise, in the cloud, or across hybrid infrastructures.
    • Flexibility for future evolution — swap out or upgrade components without affecting other components of the system.
    • Flexibility for integration — embed and extend components, such as the Query Engine and Smart Data Connectors, with other services to build integrated applications.

What about high availability and load balancing?

Glad you asked! When done right, a platform built using microservices can be highly fault tolerant to maximize uptime and performance for mission-critical applications.

A Streaming Analytics Architecture for Real-Time and Big Data

Treating data as streams, rather than batches or sets, is fundamentally different from how traditional BI works. Logi Composer’s unique point of view that data can be represented in streams is baked into the web application, Query Engine, and Smart Data Connectors.

Logi Composer’s streaming data analytics architecture uniquely supports:

  • Real-time data analytics through Live Mode
  • Big data analytics through our patented Data Sharpening™ functionality
  • Historical data playback using the Data DVR
  • High-touch interactive data visualization through dynamic query cancellation, time windowing, and other optimizations

Real-Time Communication with WebSockets

One of the technologies that enable a streaming data analytics platform is the WebSockets bidirectional communication protocol. The Logi Composer web application and any other client application that uses the Logi Composer JavaScript  APIs establishes a two-way WebSocket communication channel between itself and the Logi Composer Query Engine. WebSockets are what allows data to stream in one direction while user requests stream in the other, allowing a dynamic and interactive end-user experience.

Stream Real-Time Data in Live Mode

Empower your whole organization to visualize and analyze streaming data for powerful real-time insights.

Big Data Visualization with Data Sharpening™

Work with data that’s too big move using Logi Composer’s unique Data Sharpening functionality.

Stream Data to Custom Applications

Application developers can use simple JavaScript commands to embed charts and populate them with fresh streaming data — direct from almost any source.

Data Sharpening™ and Microqueries

Microqueries and Data Sharpening™ are patented technologies that work together to provide the most sophisticated end user experience for analyzing big data. There is literally nothing else like it available on the market today.

If enabled for a data source, the Query Engine determines whether to invoke microqueries. Microqueries repeatedly sample data across partitions and return the sample data to the Query Engine for processing.

Data Sharpening is the complementary process of analyzing sample data returned by microqueries, and streaming estimated results to the user’s browser (or other client). Data Sharpening’s estimated results may fluctuate a bit up or down until the final query resolves. However, the relative values of each group usually remain consistent as the data is sharpened. For example, the tallest bar in the chart at 10% completion will almost always remain the tallest bar at 100% completion. This means that users can be confident exploring data even as it streams live to the dashboard.

When the user zooms in, filters, changes metrics or groupings, or any other action that would change data values, Logi Composer cancels any active queries. Canceling active queries, however, is not trivial, and many JDBC drivers do not support it. In these cases, Logi Composer’s smart data connectors issue native API calls to complete the task.


No big data solution is complete without security. Our data architects have built security in to Logi Composer’s architecture. Regardless of whether Logi Composer is used as a standalone BI platform or used to visually analyze data in an embedded application, it ensures adherence to the “three As” of security — proper authentication, authorization, and auditing of the visual analytics environment.

Modern Microservices Architecture for Scale and Speed

  1. Logi Composer Application Server
  2. Modern Query Engine
  3. Smart Data Connectors
  4. JavaScript SDK and RESTful APIs
  5. Microservices architecture — you are here
  6. Adaptable security model

Originally published January 15, 2020; updated on March 19th, 2021