Home:XQuery Engine:Our Customers:RLPT Chooses XQuery

RLPTechnologies: Supporting Large XML Files

Customer Profile

RLPTechnologies, Inc., (RLPT) is a wholly owned subsidiary of R. L. Polk & Co., the premier provider of automotive information and marketing solutions.

RLPT’s flagship solution, OneView360°, is an automated data management and database production application. This comprehensive data factory software application is designed to automate the collection of data from disparate sources, create standardized data feeds, orchestrate the enrichment of data to improve its accuracy, create a “Single Source of Truth” for all of an organization’s information assets, and distribute the data for use in analytical and operational applications. The solution is both an engine that enables the timely processing of data, and a single integrated application that provides IT, data and business analysts’ one user interface in which to work to manage their data.

Some of the key benefits attained from implementing OneView360° are to provide the most stable, predictable, flexible and cost-effective data production environment. With an open, service-oriented architecture (SOA), OneView360° can seamlessly integrate investments in legacy, commercial off-the-shelf applications and web services.

Understanding the Business Challenge

RLPT’s goal is to provide their customers a competitive advantage through accurate and timely business information. The ability to quickly analyze and leverage all facets of data within the business and make the appropriate decision faster than your competitors can put you ahead of the game.

Information for these business decisions is only a good as the data. The quality of the data and the ability to process it quickly and cost-effectively are important factors driving the information supplied to customers.

OneView360° is a data processing engine capable of moving very large volumes of data around in XML. For example, in one implementation of OneView360°, source data files containing automotive industry customer records can exceed 30GB, and can exceed 50GB after the source data is enriched.

A Data Quality Inspection module is a critical component of OneView360°, which incorporates checkpoints throughout the entire application. The early identification of data anomalies and quality issues is a critical success factor. WithDataDirect XQuery®, OneView360° is able to perform data profiling functions such as complex frequency distributions, counts and pattern analysis data directly on the large XML data files.

Choosing the Right Solution – DataDirect XQuery®

A major requirement of OneView360° was to support the XML documents in their native format and to support them regardless of size. Other solutions require intermediate processing of the XML data files. This can add to the time required for both development and processing, with the potential for scalability issues.

RLPT found that the XQuery approach to handling and processing XML was one of the best approaches to dealing with the extraction and manipulation of information in native XML format. XQuery provided the means for collecting statistics about the XML content by enabling sophisticated querying capabilities which allowed OneView360° to evaluate measurements taken on the file against history and preset quality thresholds.

The issue of large XML file support caused RLPT to test and analyze various solutions. Ultimately, DataDirect XQuery® provided the fastest and most scalable XQuery implementation for RLPT’s current requirements and projected future growth. During testing, DataDirect XQuery® processed data at least seven times faster than other solutions. Other XQuery products showed performance degradation and high memory consumption when processing much smaller XML files.

“To meet the product requirements for OneView360° we needed to process XML files of up to 50GB successfully 100% of the time in an efficient manner. DataDirect’s optimization techniques including Document Projection and Document Streaming allowed us to meet our requirements and clearly positioned DataDirect XQuery® ahead of its competitors”, said Clara Sagan, senior product engineer, RLPT.

Realizing the Benefits of DataDirect XQuery®

“DataDirect XQuery® provided us a commercial, well-supported implementation of the XQuery specification while delivering the performance, scalability and reliability necessary for our large scale XML profiling needs,” said Joe LaFeir, vice president of product development, RLPTechnologies. “Using the DataDirect product has helped us significantly reduce our development effort, which helped bring our OneView360° Data Quality Inspection module to market faster.”

After testing alternative solutions and products, RLPT chose DataDirect XQuery® for these reasons:

• DataDirect XQuery® processed the XML documents with outstanding performance. OneView360° is able to process the XML documents directly, which is the quickest way to deliver business-critical information.
• DataDirect XQuery® was a proven, reliable XQuery product. Other products and solutions couldn’t meet RLPT’s XML file size requirements. DataDirect XQuery®’s optimizations of memory usage and consumption gave OneView360° the ability to meet and exceed required file sizes.
• RLPT developers found DataDirect's technical support to be a valuable asset. The technical support team worked closely with RLPT to provide best practice strategies when working with large XML files and quickly resolved their questions and issues in a timely manner.

DataDirect XQuery® enabled RLPTechnologies to process very large XML data files natively to efficiently measure the quality of data, while allowing them to reduce development time.