Smart Data Connectors for the Modern World

Share on LinkedIn Tweet about this on Twitter Share on Facebook

Cloud data warehouses, big data, streaming data, search engine databases, NoSQL, and more

While traditional BI and analytics tools offer a broad set of connectors for primarily SQL data sources, Logi Composer offers the widest set of connectors for modern data stores such as cloud data warehouses, Hadoop, NoSQL document store databases, streaming, search engines, as well as for traditional SQL relational database and modern data warehouses.

Data Sources

Smart Data Connectors are ready to connect to a wide variety of traditional and modern data sources

Convert User Requests into Native Queries

Logi Composer Smart Data Connectors let you directly query all your data, regardless of where it’s stored or how it’s managed. All data requests start with the same simple Logi Composer query API for Elasticsearch, Impala, MongoDB, Snowflake, SQL Server, and almost anything else you use. Our library of Smart Data Connectors and scalable data access framework convert those dynamic user requests into native pushdown queries.

No SQL jockeying or XML coding necessary, and your data stays securely in place. No need to move or prep the data in a middle tier BI server or cube.

What is a Smart Data Connector?

A Smart Data Connector is a Logi Composer microservice that retrieves data. Each Smart Data Connector is dedicated to translating abstract query requests from the Logi Composer Query Engine into native pushdown queries.

Smart Data Connectors are different from traditional connectors because they:

  • Simplify working with big data, streaming data, and modern data platforms that are not SQL-native, like Elasticsearch and MongoDB.
  • Enrich the analysis experience with partition awareness for query optimizations, search engine awareness for qualitative analysis, and date/time field awareness for streaming analytics
  • Scale and run independently to optimize overall platform performance and availability

The data access framework is extensible to anticipate even more changes as data platforms continue to evolve. The framework is also adaptable. For traditional RDBMS data sources, for example, the Smart Data Connector will use SQL when available and native API calls when necessary. For example, a data platform may support query cancellation, but its JDBC driver may not. Logi Composer data connectors are smart enough to adapt its query protocol to fulfill user requests.

Leverage Investments in Modern Data Platforms

Video Thumbnail

This episode of The Briefing Room with Eric Kavanagh covers a lot of ground, including a technical discussion of Smart Data Connectors starting at 47:55

Since all data sources are not the same, Logi Composer doesn’t treat them with lowest common denominator connectors. Instead, Logi Composer Smart Data Connectors leverage the unique capabilities of each data source in order to optimize performance and take advantage of the their query expressiveness.

For example, the Logi Composer data access framework accommodates many modern data source innovations, including:

  • Big data: identifies partitions to optimize query performance
  • Document store databases: supports nested and multivalue (array) field aggregation and drill to detail
  • Search engine databases: leverages inverted indices for qualitative analysis
  • Streaming: identifies date and time fields to stream data at various levels of detail

Smart Data Connectors also perform the important job of reporting data types and other supported functionality. The Query Engine uses this information to plan, optimize, and execute data requests.

Adaptable and Extensible Data Access Framework

Logi Composer implementations typically have more than one Smart Data Connector to allow users to connect to different data platforms, often at the same time.

Each Smart Data Connector is a miniature server or microservice that runs in its own protected processing space. This protects the rest of the Logi Composer deployment from connectivity failures or potential system conflicts, such as dependencies on different version of Java. It also allows organizations to easily deploy different versions of the same connector, such as Elasticsearch version 5 and 6.

Benefits of deploying on a microservices architecture

  • Deploy while the server is running — no restart
  • Recover faster in unexpected failure scenarios
  • Scale vertically, horizontally and modularly for heavy loads
  • Optimize memory allocation and runtime attributes
  • Eliminate system conflicts
  • Minimize compatibility concerns

Pushdown Processing for Speed and Scale

Logi Composer pushes query processing to the source as much as possible. For sources like Impala that speak SQL, Logi Composer generates SQL queries and sends them to the source. For sources like Elasticsearch, Cloudera Search, and Solr that speak search, Logi Composer generates search queries and, again, sends them to the source. The “heavy lifting” involved with resolving a query, such as aggregation, filtering and calculations, is performed by the system where the data resides. Only the final result set is transferred from the source to the Logi Composer server. Avoiding unnecessary data movement is essential for big data scalability.


query pushdown bad example
query pushdown

Custom Data Source Connectors

Logi Composer supports the industry’s broadest range of modern data sources designed for big, streaming and unstructured data, as well as traditional SQL relational databases. We keep up to date with new sources as they gain adoption within organizations for their BI and visual analytics requirements.

However, on occasion organizations may find they need to connect to a source not currently supported by Logi Composer, for example a homegrown application, or a new or emerging data platform.

Logi Composer has you covered. The Logi Composer data access framework is extensible to allow the creation of custom Smart Data Connectors. This is an advanced data preparation capability that incorporates Logi Composer “smarts” for functionality and optimizations, such as streaming, partition awareness, and more. Please contact us to discuss your needs and pricing.

Custom Data Source Connectors

Smart Data Connectors scale horizontally and independently

Smart Data Connectors
  1. Logi Composer Application Server
  2. Modern Query Engine
  3. Smart Data Connectors  — you are here
  4. JavaScript SDK and RESTful APIs
  5. Microservices architecture
  6. Adaptable security model

Originally published January 15, 2020; updated on March 19th, 2021