Overview

The whole platform in seven modules, each doing one job and handing its result to the next.

Data Wrangler v4 — overview

Data Wrangler Solutions LLC

Data Wrangler 4.0 is a petroleum data management platform. It is built from seven modules, each of which does one job and hands its result to the next.

1 · DataView — the database

A customized database schema derived from PPDM 3.9, made more adaptable and flexible for today’s messy data environment. DataView is much easier to load data into and to customize than a conformant PPDM implementation.

It carries the provenance of every record, so the file a row was loaded from is identified and can be opened.

It handles multiple schemas — a modified schema for the Gulf of Mexico, and the Federation, where each state and federal agency’s data was loaded into its own schema and the common fields were then federated so a single map can display all of it.

2 · File Catalog

An extension of the DataView schema with tables for the inventory of files on a shared drive, and for the cataloguing and extraction of the tables of relevant data those files contain.

Structured files: LAS, DLIS, LIS, SEG-Y, P190, Esri shapefile. Unstructured files: Word, Excel and PDF.

The extracted data can optionally be loaded into the main DataView schema, or into another database.

3 · Data Assistant

Where flat data files are bulk loaded into DataView. The data is loaded to staging tables; the mapping of tables and columns is presented for review; the data is normalized and validated; foreign keys are presented for resolution wherever one has been violated. Once the mapping and the foreign keys are resolved, the data is promoted into the DataView tables.

The mapping is remembered. Each file’s column shape is kept as a fingerprint, and a synonym store is built alongside it. The result is that once a file format has been seen, a file of that format moves through the system in a few clicks — and the same applies to a whole directory of CSV or Excel files.

Headers, stratigraphy, log summaries, directional surveys, completions and production can be bulk loaded at the same time, provided those formats have been loaded before. Updating the database with data arriving from the field or from a provider becomes virtually automatic.

4 · Mapping

Data Wrangler was designed to be fully integrated with a mapping front end.

The map carries a drop-down query tool and an AI query tool for asking more complex questions. It provides access to scout tickets for wells, and to every document related to a well — final well report, LAS files and the rest — for the wells selected on the map. No SQL required.

The export tool writes the selection out in a range of formats: scout tickets, CSV, multi-page Excel, Petrel, GeoGraphix and OSDU.

The mapping tool also has viewers for log data (LAS, LIS, DLIS), seismic data (SEG-Y) and the standard desktop formats.

5 · Document Assistant

A tool for training the system to recognise tables in a document by their shape.

When the recogniser extracts a table it has seen the shape of before, the File Catalog passes it straight through to the database. The user can inspect a set of files to check that the recogniser identified the table correctly and mapped every column; where it did not, the user corrects the mapping or claims the columns that were missed.

Those corrections go into the knowledge base and are used from then on. So the system grows as more documents are read and the vocabulary accumulates E&P terminology.

The technology is not specific to petroleum — it works on any kind of document with tables in it.

6 · DB Explorer

To be written.

7 · Reference Tables

To be written.

Where the detail lives

Each module has its own component document: