Data management is a crucial component of scientific research. Multiple terrabytes of collected data can easily get out of hand if steps are not taken to "tame the beast." 

Shumla is committed to maintaining the security, accessibility, and integrity of the data we collect. We have developed rigorous procedures for the long-term management and storage of our vast data sets. 

With increasing use of sophisticated digital technology, the demand on storage space is enormous. Recently, in just three weeks of field work, Shumla accumulated almost 750 gigabytes (GB) of raw data. This includes figure photography, Global Positioning System (GPS) data, Total Data Station (TDS) data, portable x-ray fluorescence (pXRF) elemental data, microphotography, figure attribute data, and so much more. How do we manage it all? Servers, RAID Systems, Back Ups, File structures, and Naming Conventions...oh my!

Servers

Shumla utilizes three separate servers. One is dedicated to the administrative files of the organization. One holds exclusively research data. The third as a fail safe in the event either of the other servers goes down. 

RAID Systems

Disk drives, just like any piece of hardware, will eventually fail. To protect us from the inevitable, our servers incorporate variations of redundant arrays of independent disks, or RAID systems. These systems provide protection from drive failures through disk mirroring and offer a performance boost by dividing all data among multiple disks, thus splitting the workload. 

Back Ups

Utilizing RAID is not a substitute for regular data backups. RAID can help minimize data loss, but ultimately backups are required to protect collected data. Shumla backs up data on a daily, weekly, and monthly basis. Daily backups are stored on the server itself, but weekly and monthly backups are stored on an offsite server away from headquarters. 

File Structure

Within our research server we've developed a particular file structure that keeps our data organized. We create site folders named with unique site trinomials. Within these site folders data is separated into categories:

• 3D Models − 3D modeling data
• Conservation − conservation and site or art degradation data
• Crew and Site Photography − photographs not directly related to figure data
• Databases − individual site and photo databases
• Figures − data pertaining to specific identified figures, including photographs
• Mapping (GIS) − geospatial information
• pXRF − elemental analyses for figures and the site as a whole
• Site Documents − site forms and the SHPO site forms (TexSite)
• Stratigraphic Analysis − relationships of paint layers within and across figures. 

Naming Conventions

Shumla takes thousands of photographs. Original RAW photos are saved in a folder marked "Originals." They are converted into .DNG (Digital Negative) format -- a "universal" file format for uncompressed photos that allows metadata to be written directly onto the file and deals effectively with the color management. Duplicate .JPEGs are then created from the .DNG files. We rename the .JPEG duplicates according to a set naming converntion: the site trinomial, photo type, figure number, subject, and the last four digits of the original RAW photo number. 

Yes, it may seem mundane, but data management is a big deal to the Shumla staff. The data we collect will preserve the information the ancient murals hold long after they are gone...but only if it is managed properly. That's why we take "taming the beast" very seriously.