Architecture

This section describes the internal architecture of OpenCGA and the main ideas about how it has been implemented. If you want to collaborate with the project, this is the place to start reading! :)

Data model

We believe that it is important to keep the databases mostly unaware in which format the data was originally stored. A reference to this format will only be stored for specific purposes involving file transfers.

A data model for variants and alignments has been designed and implemented in Java. It explicitly specifies the most commonly used fields, and at the same time provides mechanisms for preserving all the information of a certain format. For a variant, the specified fields would be (among others) chromosome, position, reference and alternatives; if a VCF file is being stored then columns such as INFO are also saved in a key-value data structure.

Data storage

As many organizations involved in Big Data projects have done, we decided to use technologies such as Hadoop and MongoDB. Relatively small datasets can be stored in a Mongo-only backend, while those expecting a really high load of information could opt for a combination of both technologies, which we decided to name Monbase :)

Monbase (Mongo + HBase) backend

This schema uses HBase for storing raw data and Mongo as an index for the HBase database.

In HBase, variants are stored in a table with a chr:position in each row. There are 2 column families, one (data) for storing the variant and samples information, and another one (info) for storing variant statistics in a certain study.

Row Key	Column Family: Data (d)	Column Family: Info (i)
chromosome:position	columns in the input file	study statistics
1:123456	d:st1_ref = { A } d:st1_alt = { C } d:st1_NA001 = { GT : A/C, DP = 5 }	i:st1_stats = { MAF : 0.05, MGF : 0.20, miss : 10 } i:st2_stats = { MAF : 0.04, MGF : 0.15, miss : 2 }

Storing pre-calculated statistics allows to retrieve global information for a study very fast. Statistics for a subset of the samples in the study or for combinations of samples from multiple studies must be calculated on demand and won't be stored afterwards; saving all of them would not be affordable.

Note: At this moment, only one VCF per study is accepted. If multiple VCF files are to be stored, they must be merged before.

Mongo-only backend