Oil and Gas Data Management Options
Data management involves handling data throughout its life cycle in addition to the infrastructure upon which it resides. The amount of data that oil and gas companies generate is continually growing, as is the infrastructure upon which that data resides.
This is primarily due to the increase in data-collection capability, although business reorganizations such as acquisitions, mergers and joint ventures also contribute to the volume of data in this industry. Changes in storage methods such as cloud computing further complicate data management for oil and gas enterprises.
The best practices in data management often include an open strategy that allows multiple solutions to inter-operate effectively.
Volume
Modern 3D seismic surveys are one reason for larger data sets in the oil and gas industry. NetApp reports that this surveying method contains 16 times more data than a traditional 2D survey, and a 4D survey that monitors changes over time requires an additional 32-fold increase in data over a 3D survey.
New algorithms also increase data volume by using up to 40 well attributes such as dip azimuth, event continuity, instantaneous frequency and instantaneous phase. The increase in oil prices contributes towards data volume by increasing the number of fields that are now cost effective to mine.
Technological advances such as gravity magnetics, ground-penetrating radar and pore-pressure prediction also increase data sets by requiring surveyors to digitize more data. All of these factors result in an annual increase in data volume of about 30 to 70 percent, along with a corresponding increase in data management costs.
Classification
The classification of data in the oil and gas industry is a challenging data management task. The growth of the data sets and changes in infrastructure increase the difficulty of locating a specific piece of data, which often results in data duplication.
The turnover of personnel and changes in their responsibility also leads to problems in identifying the owner of a particular data set. These factors contribute towards the large amount of data that isn’t stored in a structured database, which some analysts estimate at 70 to 80 percent of the total data.
This large quantity of unstructured data complicates the process of planning storage expenditures and archiving data.
Reorganizations
The increasing frequency of business reorganizations further complicates data management in the oil and gas sector. The participants in these reorganizations typically perform an inventory of the data set that each party possesses to identify duplicate data.
They must also analyze these data sets to determine the ones that should be archived or deleted. This process allows the business to avoid purchasing data that it already has such as lease block sales.
Data Storage
The proliferation of storage techniques for big data also makes data management more difficult. The traditional solution to sharing data is the 
Network File System, a file-sharing protocol that’s available in the UNIX operating system.
However, data storage in oil and gas is currently trending towards the Common Internet File System, which is only available in Windows operating systems. A transition from one file-sharing protocol to another involves the creation of duplicate data, which increases the cost of data storage.
Managing infrastructure and data
Effective data management requires managing the infrastructure as well as the data itself. The current challenges in data management include:
- the increasing size of the data set
- frequent business reorganizations
- migrating between file storage systems
Additional tasks in modern data management include migrating to another storage tier, identifying data suitable for deletion and planning for disaster recovery. Challenges in infrastructure management include achieving the optimum balance between hardware and software to meet business needs such as cost reduction, flexibility, scalability and regulatory compliance.
Improved Analytics are Making Oil & Gas Data Management Easier
With today’s exponential growth in data production, using oil and gas data management to wisely leverage assets is essential to helping industry experts make the wise decisions that are driving great reductions in estimated ultimate recovery (EUR).
Our support systems are intelligent and in some cases self-learning. The result is higher productivity. The business intelligence derived from big data analytics is a function of successful data management.
Mining data integration
Integration and mining of data generated in the hydrocarbon discovery and production process provides answers to some of the big questions facing the oil industry. Where is more oil? Where are the most environmentally safe areas for extraction?
Identifying and exploiting data that will provide answers to these questions is but one of the many goals for business intelligence software based on big data analytics.
When I look into the near future, I glimpse automated analysis systems that learn while the user goes about interpreting seismic data. Just as a Google translator expertly translates German to English today by learning from countless human translations, oil and gas expert systems will improve by understanding the user’s selection methods.
There is already proof of concept for this industry specific self-learning software. In the next development stage other types of data will be integrated in a search to weed out non-productive paths and keep those that are productive.
Improving Oil and Gas Drilling Efficiency
While Oil and Gas is in the early stages of employing big data analytics, data manipulation tools, new software and new middleware will be harnessed to locate and process oil and gas resources, as well as to optimize and control operations.
Transform specializes in Analytic Interpretation and Modeling, with a focus on solutions that maximize GG&E productivity and optimize operational decision making. In a recent Transform project, it became possible to reduce the number of wells drilled by 20% without any impact to the EUR.
Transform’s sophisticated geophysical interpretation tools have enabled companies to integrate geological and geophysical data and interpretive products into a data model that can quantify geological drilling impacts on production, and thus promote better field design. Their recently released hydraulic fracturing-induced micro seismic imaging and modeling module permits companies to gain insight into induced fracturing measurements and mapping.
Multivariate statistical analysis functionality enables improved well placement decision making. Wave modeling is optimized for real-time in preparation for managing frack stages.
Transform’s cross-disciplinary platform combines geological, geophysical and engineering tools with browser-based controls that provide clients with the option to select an area and open an analytical workbench on a desktop populated with DI data.
US Seismic Systems designs and manufactures fiber optic monitoring solutions for the oil & gas industry. Their sensing technology solutions are widely used worldwide. As replacements for electronic sensor systems, these are a superior and more economical alternative for 4D and micro seismic monitoring.
A look at software/data implementation
Oracle supports Big Data functionality with Oil and Gas software. Seismic, logs, surveying and real time measurement data are collected and analyzed. NOSQL databases store unstructured data:
- Hadoop – rationalizing huge datasets
- HDFS – massively parallel storage and analysis
- Extreme analytics – R convenient with petabyte datasets
- Complex Event Processing – real-time database analysis
- Bayesian inference engine and decision management perform in real time
- Information Discovery – find relationships regardless of source data structure
The big data cycle can be described as Acquire -> organize (using massive parallelism) -> analyze all data at one time -> process. Solutions can be relational or NOSQL (unstructured and schemaless). The Oracle data integrator takes data from both structured and unstructured sources. R statistical programming and graphics language is used for statistics, graphics and data mining.
The Future for Oil and Gas
Big data analytics for oil and gas are relatively new, but already gaining in productivity due to patterns and intelligence derived by enhanced information analysis. In addition,we know that machine learning techniques are being applied so that software is trained by analysts and will ultimately be able to learn as a by-product of processing.
For more on solving problems and improving on data for the oil and gas industry, check out this post on starting with problems, not solutions…