SeqPig: Simple and Scalable Scripting for Large Sequencing Data Sets in Hadoop

TitleSeqPig: Simple and Scalable Scripting for Large Sequencing Data Sets in Hadoop
Publication TypeJournal Article
Year of Publication2014
AuthorsSchumacher A, Pireddu L, Niemenmaa M, Kallio A, Korpelainen E, Zanetti G, Heljanko K
Published inBioinformatics
Volume30
Issue1
Page(s)119-120
Other Numbers3622
Abstract

Hadoop MapReduce-based approaches have become increasingly popular due to their scalability in processing large sequencing datasets. However, as these methods typically require in-depth expertise in Hadoop and Java, they are still out of reach of many bioinformaticians. To solve this problem, we have created SeqPig, a library and a collection of tools to manipulate, analyze and query sequencing datasets in a scalable and simple manner. SeqPigscripts use the Hadoop-based distributed scripting engine Apache Pig, which automatically parallelizes and distributes data processing tasks. We demonstrate SeqPig's scalability over many computing nodes and illustrate its use with example scripts.Availability and Implementation: Available under the open source MIT license at http://sourceforge.net/projects/seqpig/ CONTACT: andre.schumacher@yahoo.com SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Acknowledgment

This work was partially funded by the Deutscher Akademischer Austausch Dienst (DAAD) through a postdoctoral fellowship.

URLhttps://www.icsi.berkeley.edu/pubs/vision/seqpig14.pdf
Bibliographic Notes

Bioinformatics, Vol. 30, Issue 1, pp. 119-120. DOI: 10.1093/bioinformatics/btt601

Abbreviated Authors

A. Schumacher, L. Pireddu, M. Niemenmaa, A. Kallio, E. Korpelainen, G. Zanetti, and K. Heljanko

ICSI Research Group

Networking and Security

ICSI Publication Type

Article in journal or magazine