OpenCGA Storage Overview

Overview

OpenCGA Storage is the submodule in charge of ingest and provide a query language over biological BigData. This storage-engine handles the most common file formats for NGS.

The aim of this module is provide a query engine that will be the source of data for the analysis.

Bioformats

There are an increasing number of biological formats supported by OpenCGA related with the most common steps on a common NGS pipeline. Within this formats, we focus on Genomic Variants due to the complexity and analysis capabilities.

Variants

• Study oriented
• Cohort definition
• Variant annotation

More information at Variant Storage Engine

Alignments

• Coverage calculation
• Alignment stats

More information at Alignment Storage Engine

Other

• Sequence (fastA)
• Feature formats (GFF, BED, BigWIG...)

Multiple Storage Engine Implementations

Based on a common definition of the Storage Engines, depending on the requirements and the resources of the study, there are multiple implementations using different technologies. This document specifies the core functionality that all the implementations must share. Technical details or customization parameters are explained in the plugin specific section.

• OpenCGA Storage MongoDB
• OpenCGA Storage Hadoop
• Creating a new Storage Engine

Pipeline

📄 -(transform)-> intermediate -(load)-> DB -(query)-> result

Configuration file

All the configuration needed to work with is centralized in the storage-configuration.yml file, usually located in the configuration folder.

This file contains the configuration for all the storage engines, the database connections and other cellbase and server information.

In the Storage Configuration you can find an extended explanation of the file structure and all the parameters.

REST/gRPC Servers

OpenCGA Storage command line interface

There is a direct command line to interact with OpenCGA storage. This command line is not intended to be used in a full featured installation because is not connected to OpenCGA Catalog.

Use only for development or testing purposes.

See more information at Storage Command Line