Big Data Challenges in Genome Informatics

03/20/2018 ∙ by Ka-Chun Wong, et al. ∙ 0

In recent years, we have witnessed a dramatic data explosion in genomics, thanks to the improvement in sequencing technologies and the drastically decreasing costs. We are entering the era of millions of available genomes. Notably, each genome can be composed of billions of nucleotides stored as plain text files in GigaBytes (GBs). It is undeniable that those genome data impose unprecedented data challenges for us. In this article, we briefly discuss the big data challenges associated with genomics in recent years.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Since 1990s, the whole genomes of different species have been sequenced by their corresponding genome sequencing projects. In 1995, the first free-living organism Haemophilus influenzae was sequenced by the Institute for Genomic Research. In 1996, the first eukaryotic genome (Saccharomyces cerevisiase) was completely sequenced. In 2000, the first plant genome, Arabidopsis thaliana, was also sequenced by Arabidopsis Genome Initiative. In 2003, the Human Genome Project (HGP) announced its completion. Following the HGP, the Encyclopedia of DNA Elements (ENCODE) project was started, revealing massive functional elements on the human genome in 2011 [9]. The drastically decreasing cost of sequencing also enables the 1000 Genomes Project and Roadmap Epigenomics Project to be carried out. Their results have been published in 2012 and 2015 respectively [8, 15]. Nonetheless, the massive genomic data generated by those projects impose an unforeseen challenge for big data analysis at the scale of GigaBytes (GBs) or even TeraBytes (TBs).

In particular, Next-Generation Sequencing (NGS) technologies have enabled massive data generation for different genomes [30, 20]; for instance, DNA sequencing, protein-DNA binding occupancy [28] (e.g. ChIP-seq [26], ChIP-exo [23], and ChIA-PET [12]), bisulfite sequencing [5], transcriptome sequencing (e.g. RNA-seq [22]), and chromatin interaction sequencing (e.g. Hi-C [18]

). Thanks to the relatively low costs, those NGS technologies have been readily applied to human genomes nowadays. The international projects aforementioned have been successfully launched, leading to massive NGS data accumulation at an unprecedentedly fast pace. Nonetheless, current integrative analyses are usually limited to traditional machine learning and data mining methods such as pair-wise correlation analysis, statistical tests, classification, and feature extraction

[29]. Those methods are intentionally designed to generally fit into different types of data. However, the data from NGS is unique and different from the traditional data; for instance, the ChIP-seq data is sparse, noisy, and discontinuous. Special care has to be taken to alleviate and transform those challenges to be taken advantages of [27]. In addition, the NGS data is huge (in GigaBytes per each dataset) which imposes a difficulty in applying some of the existing statistical/computational methods.

Therefore, different genome-scale problems been defined and being tackled to harness those genomic data. In this review, we aim at providing a concise literature review on those challenges.


In this section, we provide a concise review on the big data challenges which are individually described in the following subsections. Especially, the challenges have been visualized and depicted in Figure 1 for reader-friendliness.

Figure 1: Big Data Challenges in Genome Informatics. The challenges are listed from top to bottom; namely, genome assembly, signal profile analysis, genome-wide association study (GWAS), and 3D genome structure reconstruction.

De Novo Genome Assembly

The advancement in DNA sequencing technologies has enabled the assembly of whole genome in an economical and fairly accurate way [19]. Nonetheless, a genome cannot be easily identified in one piece from wet-labs. Limited by our current DNA sequencing technologies, each genome has to be shattered into different small pieces (short DNA sequences or reads) before their DNA nucleotides can get sequenced and identified as shown in Figure 1. Therefore, we come to the de novo genome assembly problem: to sequence and identify a genome, we have to ’stitch’ those short DNA sequences into a single and consistent DNA genome. There are different benchmark measurements such as N50, total length, and number of missing nucleotides. If we already have a reference genome, then the measurements can be more solid than the previous measures such as NG50 and genome fraction. If reference genome annotation is available, the number of genes covered can be a good measurement. More details can be found in [13]. To solve this kind of genome assembly problems (in GBs or TBs), there are many computational methods proposed in the past. Nonetheless, most of them depend on the construction of de brujin graph which is memory-consuming and computationally intensive. According to the recent benchmark study, different genome assembly methods show result disagreement with each other [7]. In addition, the sequencing errors incurred by wet-lab experimental conditions are unavoidable, making the genome assembly problem even more complicated than we have imagined [19]. Therefore, the genome assembly problem remains as a big data challenge to be solved.

Genome Signal Profile Analysis

In addition to genome assembly, there are different genome-wide signals such as gene regulation (e.g. protein-DNA binding interactions) and epigenetic interactions (e.g. DNA methylation) as shown in Figure 1. Therefore, it is essential for us to look into those information. To this end, different genome-wide biotechnologies have been developed such as ChIP-seq, DNase-seq, RNA-seq, CLIP-seq, DNA methylation array assay, Bisulphite sequencing, Repli-seq, and CAGE. To gains insights into those data, tremendous efforts have been made to pre-process the data such as read trimming [6], sequencing error correction [31], sequencing replicates [24], and read mapping [10]. After the data has been processed, downstream analysis methods can be applied to reveal genome-wide signals from it; for instance, multiple signal profile integrative analysis [27, 29] and signal profile peak calling [32]. In particular, the multiple signal profile analysis is very important for us to understand the complex behavior of the genome-wide signals [27]

. Unfortunately, each signal profile is proportional to genome size since it has a genome-wide coverage (usually in GBs). Therefore, if we have multiple signal profiles (e.g. hundreds from the ENCODE consortium), the computational scalability issue has to be taken into serious account. Another issue is that the past wet-lab studies are very limited to fine-scale knowledge (e.g. single gene study). Therefore, the genome-wide result verification is very difficult to be carried out. At the current stage, we heavily rely on null hypothesis testing to ascertain the results’ statistical significances. Therefore, we can foresee that the genome signal profile analysis will still be a big data challenge in genome informatics.

Multiple Genomes: GWAS

Single Nucleotide Polymorphisms (SNPs) are believed to be play major roles in different diseases [8]. To call for a SNP, since each SNP stems from natural selection which is acting and ’fixing’ the SNP to favor genetic adaption [3], we need multiple genomes for comparison (as shown in Figure 1) and satisfy certain statistical significance criteria [1]

; for instance, the simplest criteria is how often an allele occurs across a given genome population. However, there is not any single golden measurement for such a task. Specifically, one of the major challenges in SNP calling is to handle the massive genome-wide data incurred from high-throughput sequencing. To compare multiple genomes for SNP calling, we have to align the genomes first. Although there exists an algorithm for mathematically optimal genome alignment (i.e. Smith-Waterman Alignment), its computational complexity is exponential to the genome size, and thus practically infeasible. Therefore, different heuristic genome alignment methods have been proposed. The most successful example is MUMmer which proposes an anchor-based alignment methodology

[16]. VISTA has also been proposed for multiple genome alignment visualization [11]. However, most of the previous methods were initially designed and targeted for short genomes such as prokaryotic genomes; new computational methods have to be developed, given that we are going to have thousands of eukaryotic genomes available; for instance, the 1000 human genomes from the 1000 Genomes Project [8].

Another challenge is that, given a multiple genome alignment, we have to scan each position for SNP calling as shown in Figure 1. Especially, it is usually the case that we wish to find the SNPs which are associated with certain phenotypes such as human diseases. Therefore, it is named genome-wide association study (GWAS). Given the existing genome sizes measured in GBs, GWAS studies are famous for its computational intensive nature. If we take into account the combinatorial nature of the association, the possible search space could be exponential to genome sizes. Therefore, GWAS study is also considered as a big data problem in genomics.

3D Genome Structure Reconstruction

In recent years, Hi-C technology has been developed and applied to reveal the three dimensional organizations of different cell lines by the chromosome conformation capture method [4]. In particular, there is increasing evidence that long-range chromatin interactions are related to gene co-expression [2, 14] as well as protein-DNA interactions [17, 21]. Therefore, it is essential to comprehensively identify and reconstruct the three-dimensional (3D) genome shape from those long-range chromatin interactions for understanding genomes in the three-dimensional space. Given the GB data size of genome as well as its three-dimensional nature, such a 3D genome reconstruction is doomed to be another big data challenge.


In this study, we have discussed several big data challenges in genome informatics. Especially, we envision that those challenges will be even intense in the near future, given the maturing and cost-effective sequencing technologies. Several future directions are deemed promising: (1) Third-generation sequencing technologies [25] have been developed and being refined to be of practical uses. Although its sequencing error rate is still high, we believe that those third-generation sequencing technologies would enable another wave of big data challenges in genome informatics. (2) Single cell sequencing is another promising direction. For now, we usually just focus on specific cell types or tissue types. In the future, our sequencing technologies would enable us to look at each of the individual cells which holds great potential to trigger the next levels of big data challenges. (3) Given the genome data in GBs or even TBs, high-performance computing frameworks such as MapReduce are definitely needed to handle the exponentially growing genome data in a scalable but still accurate manner.


  • [1] André Altmann, Peter Weber, Daniel Bader, Michael Preuß, Elisabeth B Binder, and Bertram Müller-Myhsok. A beginners guide to snp calling from high-throughput dna-sequencing data. Human genetics, 131(10):1541–1554, 2012.
  • [2] S. Babaei, A. Mahfouz, M. Hulsman, B. P. Lelieveldt, J. de Ridder, and M. Reinders. Hi-C Chromatin Interaction Networks Predict Co-expression in the Mouse Cortex. PLoS Comput. Biol., 11(5):e1004221, May 2015.
  • [3] Luis B Barreiro, Guillaume Laval, Hélene Quach, Etienne Patin, and Lluís Quintana-Murci. Natural selection has driven population differentiation in modern humans. Nature genetics, 40(3):340–345, 2008.
  • [4] J. M. Belton, R. P. McCord, J. H. Gibcus, N. Naumova, Y. Zhan, and J. Dekker. Hi-C: a comprehensive technique to capture the conformation of genomes. Methods, 58(3):268–276, Nov 2012.
  • [5] Christoph Bock, Sabine Reither, Thomas Mikeska, Martina Paulsen, Jörn Walter, and Thomas Lengauer. Biq analyzer: visualization and quality control for dna methylation data from bisulfite sequencing. Bioinformatics, 21(21):4067–4068, 2005.
  • [6] A. M. Bolger, M. Lohse, and B. Usadel. Trimmomatic: a flexible trimmer for Illumina sequence data. Bioinformatics, 30(15):2114–2120, Aug 2014.
  • [7] K. R. Bradnam, J. N. Fass, A. Alexandrov, P. Baranay, M. Bechner, I. Birol, S. Boisvert, J. A. Chapman, G. Chapuis, R. Chikhi, H. Chitsaz, W. C. Chou, J. Corbeil, C. Del Fabbro, T. R. Docking, R. Durbin, D. Earl, S. Emrich, P. Fedotov, N. A. Fonseca, G. Ganapathy, R. A. Gibbs, S. Gnerre, E. Godzaridis, S. Goldstein, M. Haimel, G. Hall, D. Haussler, J. B. Hiatt, I. Y. Ho, J. Howard, M. Hunt, S. D. Jackman, D. B. Jaffe, E. D. Jarvis, H. Jiang, S. Kazakov, P. J. Kersey, J. O. Kitzman, J. R. Knight, S. Koren, T. W. Lam, D. Lavenier, F. Laviolette, Y. Li, Z. Li, B. Liu, Y. Liu, R. Luo, I. Maccallum, M. D. Macmanes, N. Maillet, S. Melnikov, D. Naquin, Z. Ning, T. D. Otto, B. Paten, O. S. Paulo, A. M. Phillippy, F. Pina-Martins, M. Place, D. Przybylski, X. Qin, C. Qu, F. J. Ribeiro, S. Richards, D. S. Rokhsar, J. G. Ruby, S. Scalabrin, M. C. Schatz, D. C. Schwartz, A. Sergushichev, T. Sharpe, T. I. Shaw, J. Shendure, Y. Shi, J. T. Simpson, H. Song, F. Tsarev, F. Vezzi, R. Vicedomini, B. M. Vieira, J. Wang, K. C. Worley, S. Yin, S. M. Yiu, J. Yuan, G. Zhang, H. Zhang, S. Zhou, and I. F. Korf. Assemblathon 2: evaluating de novo methods of genome assembly in three vertebrate species. Gigascience, 2(1):10, 2013.
  • [8] 1000 Genomes Project Consortium et al. A map of human genome variation from population-scale sequencing. Nature, 467(7319):1061–1073, 2010.
  • [9] ENCODE Project Consortium et al. The encode (encyclopedia of dna elements) project. Science, 306(5696):636–640, 2004.
  • [10] M. David, M. Dzamba, D. Lister, L. Ilie, and M. Brudno. SHRiMP2: sensitive yet practical SHort Read Mapping. Bioinformatics, 27(7):1011–1012, Apr 2011.
  • [11] Kelly A Frazer, Lior Pachter, Alexander Poliakov, Edward M Rubin, and Inna Dubchak. Vista: computational tools for comparative genomics. Nucleic acids research, 32(suppl 2):W273–W279, 2004.
  • [12] Melissa J Fullwood, Mei Hui Liu, You Fu Pan, Jun Liu, Han Xu, Yusoff Bin Mohamed, Yuriy L Orlov, Stoyan Velkov, Andrea Ho, Poh Huay Mei, et al. An oestrogen-receptor-&agr;-bound human chromatin interactome. Nature, 462(7269):58–64, 2009.
  • [13] A. Gurevich, V. Saveliev, N. Vyahhi, and G. Tesler. QUAST: quality assessment tool for genome assemblies. Bioinformatics, 29(8):1072–1075, Apr 2013.
  • [14] F. Jin, Y. Li, J. R. Dixon, S. Selvaraj, Z. Ye, A. Y. Lee, C. A. Yen, A. D. Schmitt, C. A. Espinoza, and B. Ren. A high-resolution map of the three-dimensional chromatin interactome in human cells. Nature, 503(7475):290–294, Nov 2013.
  • [15] Anshul Kundaje, Wouter Meuleman, Jason Ernst, Misha Bilenky, Angela Yen, Alireza Heravi-Moussavi, Pouya Kheradpour, Zhizhuo Zhang, Jianrong Wang, Michael J Ziller, et al. Integrative analysis of 111 reference human epigenomes. Nature, 518(7539):317–330, 2015.
  • [16] Stefan Kurtz, Adam Phillippy, Arthur L Delcher, Michael Smoot, Martin Shumway, Corina Antonescu, and Steven L Salzberg. Versatile and open software for comparing large genomes. Genome biology, 5(2):R12, 2004.
  • [17] X. Lan, H. Witt, K. Katsumura, Z. Ye, Q. Wang, E. H. Bresnick, P. J. Farnham, and V. X. Jin. Integration of Hi-C and ChIP-seq data reveals distinct types of chromatin linkages. Nucleic Acids Res., 40(16):7690–7704, Sep 2012.
  • [18] Erez Lieberman-Aiden, Nynke L van Berkum, Louise Williams, Maxim Imakaev, Tobias Ragoczy, Agnes Telling, Ido Amit, Bryan R Lajoie, Peter J Sabo, Michael O Dorschner, et al. Comprehensive mapping of long-range interactions reveals folding principles of the human genome. science, 326(5950):289–293, 2009.
  • [19] E. R. Mardis. A decade’s perspective on DNA sequencing technology. Nature, 470(7333):198–203, Feb 2011.
  • [20] Elaine R Mardis. The impact of next-generation sequencing technology on genetics. Trends in genetics, 24(3):133–141, 2008.
  • [21] B. Mifsud, F. Tavares-Cadete, A. N. Young, R. Sugar, S. Schoenfelder, L. Ferreira, S. W. Wingett, S. Andrews, W. Grey, P. A. Ewels, B. Herman, S. Happe, A. Higgs, E. LeProust, G. A. Follows, P. Fraser, N. M. Luscombe, and C. S. Osborne. Mapping long-range promoter contacts in human cells with high-resolution capture Hi-C. Nat. Genet., 47(6):598–606, Jun 2015.
  • [22] Ali Mortazavi, Brian A Williams, Kenneth McCue, Lorian Schaeffer, and Barbara Wold. Mapping and quantifying mammalian transcriptomes by rna-seq. Nature methods, 5(7):621–628, 2008.
  • [23] Ho Sung Rhee and B Franklin Pugh. Comprehensive genome-wide protein-dna interactions detected at single-nucleotide resolution. Cell, 147(6):1408–1419, 2011.
  • [24] K. Robasky, N. E. Lewis, and G. M. Church. The role of replicates for error mitigation in next-generation sequencing. Nat. Rev. Genet., 15(1):56–62, Jan 2014.
  • [25] Eric E Schadt, Steve Turner, and Andrew Kasarskis. A window into third-generation sequencing. Human molecular genetics, 19(R2):R227–R240, 2010.
  • [26] Axel Visel, Matthew J Blow, Zirong Li, Tao Zhang, Jennifer A Akiyama, Amy Holt, Ingrid Plajzer-Frick, Malak Shoukry, Crystal Wright, Feng Chen, et al. Chip-seq accurately predicts tissue-specific activity of enhancers. Nature, 457(7231):854–858, 2009.
  • [27] K. C. Wong, Y. Li, C. Peng, and Z. Zhang. SignalSpider: probabilistic pattern discovery on multiple normalized ChIP-Seq signal profiles. Bioinformatics, 31(1):17–24, Jan 2015.
  • [28] Ka-Chun Wong, Tak-Ming Chan, Chengbin Peng, Yue Li, and Zhaolei Zhang. Dna motif elucidation using belief propagation. Nucleic acids research, 41(16):e153–e153, 2013.
  • [29] Ka-Chun Wong, Chengbin Peng, and Yue Li. Probabilistic inference on multiple normalized signal profiles from next generation sequencing: Transcription factor binding sites. IEEE/ACM Transactions on Computational Biology and Bioinformatics, (1):1–1.
  • [30] Ka-Chun Wong and Zhaolei Zhang. Snpdryad: predicting deleterious non-synonymous human snps using only orthologous protein sequences. Bioinformatics, page btt769, 2014.
  • [31] X. Yang, S. P. Chockalingam, and S. Aluru. A survey of error-correction methods for next-generation sequencing. Brief. Bioinformatics, 14(1):56–66, Jan 2013.
  • [32] Yong Zhang, Tao Liu, Clifford A Meyer, Jérôme Eeckhoute, David S Johnson, Bradley E Bernstein, Chad Nusbaum, Richard M Myers, Myles Brown, Wei Li, et al. Model-based analysis of chip-seq (macs). Genome biology, 9(9):R137, 2008.