See how you can create, deploy and maintain analytic applications that engage users and drive revenue. See a Logi demo

Development Tips

How to Leverage Your Existing Data Infrastructure with Push-Down Processing

By Steve Murfitt
Share on LinkedIn Tweet about this on Twitter Share on Facebook

When thinking of data architectures and how tools—particularly analytics tools—interact with the data, you have two primary options for how you process your data: extraction and push-down processing. Extraction is a process of extracting the data and then manipulating it in its extracted form. With push-down processing, you attempt to leave the majority of the data in place, extract the smallest set of results possible, and then—if needed—manipulate it.

What Are the Advantages of Push-Down Processing?

Most application teams prefer to start with push-down technology, which comes with several advantages:

  • You can leverage your existing infrastructure
  • You don’t need to worry about moving lots of data as well as storing the data to be used by the tools
  • You won’t need to introduce additional security, assuming you can leverage the security as part of the request

With push-down processing, you can leverage the investment you’ve already made in the technology within the databases and the underlying data architecture that you have. This is particularly important as data volumes grow and moving data can, at times, be the slowest part of a process. In addition, it helps limit building processes to understand when the data needs to be extracted or updated periodically (or on a rules-based basis). This doesn’t mean that you can’t make changes in your data architecture over time; it just gives you a way to get started.

Push-down processing can also make it easier for you to address changing user needs, which is always an iterative process. For instance, you may think you know what your users are asking for and then extract the data. If you missed something, however, you have to go back and add the new information to extract the process. With push-down processing, all of the information is available on the source system so you have access to it.

Why is Push-Down Processing Important?

When you store data in your application, you might need access to that data without moving it—sometimes you want your data available for analytics as soon as a transaction is processed.

At other times, you may be doing a lot of aggregation because of specific requirements. You’ve invested in a database platform and you want to leverage that investment by using push-down processing. In other words, you’re pushing as much of your processing into the database as possible rather than trying to process your analytic solution locally. By minimizing the amount of data that’s being transferred over the network, you’ll have a more performant solution that leverages your existing security and infrastructure.

 Key Takeaways

  • In data architectures, you have two primary options for how you process your data: extraction and push-down processing.
  • A push-down process allows you to leverage your existing infrastructure, and not worry about moving your data or introducing additional security.
  • Most application teams prefer push-down processing as it utilizes existing security and infrastructure and will keep your current data architecture in place.

 

Originally published March 4, 2020

About the Author

Steve Murfitt is a Technical Account Manager at Logi Analytics. He has more than 20 years of experience in the analytics space, helping partners develop solutions to meet and exceed their customers' expectations with embedded analytics. He focuses on reviewing and advising on data and data structures to help present relevant information in a secure, usable, and performant manner.

Subscribe to the latest articles, videos, and webinars from Logi.