Data Management Options
Data management involves handling data throughout its life cycle in addition to the infrastructure upon which it resides. The amount of data that oil and gas companies generate is continually growing, as is the infrastructure upon which that data resides.
This is primarily due to the increase in data-collection capability, although business reorganizations such as acquisitions, mergers and joint ventures also contribute to the volume of data in this industry. Changes in storage methods such as cloud computing further complicate data management for oil and gas enterprises.
The best practices in data management often include an open strategy that allows multiple solutions to inter-operate effectively.
Volume
Modern 3D seismic surveys are one reason for larger data sets in the oil and gas industry. NetApp reports that this surveying method contains 16 times more data than a traditional 2D survey, and a 4D survey that monitors changes over time requires an additional 32-fold increase in data over a 3D survey.
New algorithms also increase data volume by using up to 40 well attributes such as dip azimuth, event continuity, instantaneous frequency and instantaneous phase. The increase in oil prices contributes towards data volume by increasing the number of fields that are now cost effective to mine.
Technological advances such as gravity magnetics, ground-penetrating radar and pore-pressure prediction also increase data sets by requiring surveyors to digitize more data. All of these factors result in an annual increase in data volume of about 30 to 70 percent, along with a corresponding increase in data management costs.
Classification
The classification of data in the oil and gas industry is a challenging data management task. The growth of the data sets and changes in infrastructure increase the difficulty of locating a specific piece of data, which often results in data duplication.
The turnover of personnel and changes in their responsibility also leads to problems in identifying the owner of a particular data set. These factors contribute towards the large amount of data that isn’t stored in a structured database, which some analysts estimate at 70 to 80 percent of the total data.
This large quantity of unstructured data complicates the process of planning storage expenditures and archiving data.
Reorganizations
The increasing frequency of business reorganizations further complicates data management in the oil and gas sector. The participants in these reorganizations typically perform an inventory of the data set that each party possesses to identify duplicate data.
They must also analyze these data sets to determine the ones that should be archived or deleted. This process allows the business to avoid purchasing data that it already has such as lease block sales.
Data Storage
The proliferation of storage techniques for big data also makes data management more difficult. The traditional solution to sharing data is the 
Network File System, a file-sharing protocol that’s available in the UNIX operating system.
However, data storage in oil and gas is currently trending towards the Common Internet File System, which is only available in Windows operating systems. A transition from one file-sharing protocol to another involves the creation of duplicate data, which increases the cost of data storage.
Managing infrastructure and data
Effective data management requires managing the infrastructure as well as the data itself. The current challenges in data management include:
- the increasing size of the data set
- frequent business reorganizations
- migrating between file storage systems
Additional tasks in modern data management include migrating to another storage tier, identifying data suitable for deletion and planning for disaster recovery. Challenges in infrastructure management include achieving the optimum balance between hardware and software to meet business needs such as cost reduction, flexibility, scalability and regulatory compliance.
You shouldn’t let these shifting concerns overwhelm your team! For more on getting control, check out this post on turning data into actionable information.
