1. For the Impatient

# Download bwakit (or from <http://sourceforge.net/projects/bio-bwa/files/bwakit/> manually)

wget -O- http://sourceforge.net/projects/bio-bwa/files/bwakit/bwakit-0.7.15_x64-linux.tar.bz2/download \

  | gzip -dc | tar xf -

# Generate the GRCh38+ALT+decoy+HLA and create the BWA index

bwa.kit/run-gen-ref hs38DH   # download GRCh38 and write hs38DH.fa

bwa.kit/bwa index hs38DH.fa  # create BWA index

# mapping

bwa.kit/run-bwamem -o out -H hs38DH.fa read1.fq read2.fq | sh  # skip "|sh" to show command lines

This generates out.aln.bam as the final alignment, out.hla.top for best HLA genotypes on each gene and out.hla.all for other possible HLA genotypes. Please check out bwa/bwakit/README.md for details.

1.1 Introduction

Bwakit is a self-consistent installation-free package of scripts and precompiled binaries, providing an end-to-end solution to read mapping. In addition to the basic mapping functionality implemented完成 in bwa, bwakit is able to generate proper human reference genome and to take advantage of ALT contigs, if present, to improve read mapping and to perform HLA typing for high-coverage human data. It can remap name- or coordinate-sorted BAM with read group and barcode information retained. Bwakit also optionally trims adapters (via trimadap), marks duplicates (via samblaster) and sorts the final alignment (via samtools).

Bwakit has two entry scripts: run-gen-ref which downloads and generates human reference genomes, and run-bwamemwhich prints mapping command lines on the standard output that can be piped to（输送到） sh to execute. The two scripts will call other programs or use data in bwa.kit. The following shows an example about how to use bwakit:

# Download the bwa-0.7.11 binary package (download link may change)

wget -O- http://sourceforge.net/projects/bio-bwa/files/bwakit/bwakit-0.7.15_x64-linux.tar.bz2/download \

  | gzip -dc | tar xf -

# Generate the GRCh38+ALT+decoy+HLA and create the BWA index

bwa.kit/run-gen-ref hs38DH   # download GRCh38 and write hs38DH.fa

bwa.kit/bwa index hs38DH.fa  # create BWA index

# mapping

bwa.kit/run-bwamem -o out -H hs38DH.fa read1.fq read2.fq | sh

The last mapping command line will generate the following files:

• out.aln.bam: unsorted alignments with ALT-aware mapping quality. In this file, one read may be placed on multiple overlapping ALT contigs at the same time even if the read is mapped better to some contigs than others. This makes it possible to analyze each contig independent of others.

• out.hla.top: best genotypes for HLA-A, -B, -C, -DQA1, -DQB1 and -DRB1 genes.

• out.hla.all: other possible genotypes on the six HLA genes.

• out.log.*: bwa-mem, samblaster and HLA typing log files.

Bwakit can be downloaded here. It is only available to x86_64-linux. The scripts in the package are available in the bwa/bwakitdirectory. Packaging is done manually for now.

1.2 Limitations

• HLA typing only works for high-coverage human data. The typing accuracy can still be improved. We encourage researchers to develop better HLA typing tools based on the intermediate output of bwakit (for each HLA gene included in the index, bwakit writes all reads matching it in a separate file).

• Duplicate marking only works when all reads from a single paired-end library are provided as the input. This limitation is the necessary tradeoff of fast MarkDuplicate provided by samblaster.

• The adapter trimmer is chosen as it is fast, pipe friendly and does not discard reads. However, it is conservative and suboptimal. If this is a concern, it is recommended to preprocess input reads with a more sophisticated adapter trimmer. We also hope existing trimmers can be modified to operate on an interleaved FASTQ stream. We will replace trimadap once a better trimmer meets our needs.

• Bwakit can be memory demanding depends on the functionality invoked. For 30X human data, bwa-mem takes about 11GB RAM with 32 threads, samblaster uses close to 10GB and BAM shuffling (if the input is sorted BAM) uses several GB. In the current setting, sorting uses about 10GB.

1.3 Package Contents

bwa.kit

|-- README.md                  This README file.

|-- run-bwamem                 *Entry script* for the entire mapping pipeline.

|-- bwa                        *BWA binary*

|-- k8                         Interpretor for *.js scripts.

|-- bwa-postalt.js             Post-process alignments to ALT contigs/decoys/HLA genes.

|-- htsbox                     Used by run-bwamem for shuffling BAMs and BAM=>FASTQ.

|-- samblaster                 MarkDuplicates for reads from the same library. v0.1.20

|-- samtools                   SAMtools for sorting and SAM=>BAM conversion. v1.1

|-- seqtk                      For FASTQ manipulation.

|-- trimadap                   Trim Illumina PE sequencing adapters.

|

|-- run-gen-ref                *Entry script* for generating human reference genomes.

|-- resource-GRCh38            Resources for generating GRCh38

|   |-- hs38DH-extra.fa        Decoy and HLA gene sequences. Used by run-gen-ref.

|   `-- hs38DH.fa.alt          ALT-to-GRCh38 alignment. Used by run-gen-ref.

|

|-- run-HLA                    HLA typing for sequences extracted by bwa-postalt.js.

|-- typeHLA.sh                 Type one HLA-gene. Called by run-HLA.

|-- typeHLA.js                 HLA typing from exon-to-contig alignment. Used by typeHLA.sh.

|-- typeHLA-selctg.js          Select contigs overlapping HLA exons. Used by typeHLA.sh.

|-- fermi2.pl                  Fermi2 wrapper. Used by typeHLA.sh for de novo assembly.

|-- fermi2                     Fermi2 binary. Used by fermi2.pl.

|-- ropebwt2                   RopeBWT2 binary. Used by fermi2.pl.

|-- resource-human-HLA         Resources for HLA typing

|   |-- HLA-ALT-exons.bed      Exonic regions of HLA ALT contigs. Used by typeHLA.sh.

|   |-- HLA-CDS.fa             CDS of HLA-{A,B,C,DQA1,DQB1,DRB1} genes from IMGT/HLA-3.18.0.

|   |-- HLA-ALT-type.txt       HLA types for each HLA ALT contig. Not used.

|   `-- HLA-ALT-idx            BWA indices of each HLA ALT contig. Used by typeHLA.sh

|       `-- (...)

|

`-- doc                        BWA documentations

    |-- bwa.1                  Manpage

    |-- NEWS.md                Release Notes

    |-- README.md              GitHub README page

    `-- README-alt.md          Documentation for ALT mapping

2. Background

GRCh38 consists of several components: chromosomal assembly, unlocalized contigs (chromosome known but location unknown), unplaced contigs (chromosome unknown) and ALT contigs (long clustered variations). The combination of the first three components is called the primary assembly. It is recommended to use the complete primary assembly for all analyses. Using ALT contigs in read mapping is tricky.

GRCh38 ALT contigs are totaled 109Mb in length, spanning 60Mbp of the primary assembly. However, sequences that are highly diverged from the primary assembly only contribute a few million bp. Most subsequences of ALT contigs are nearly identical to the primary assembly. If we align sequence reads to GRCh38+ALT blindly, we will get many additional reads with zero mapping quality and miss variants on them. It is crucial to make mappers aware of ALTs.

BWA-MEM is ALT-aware. It essentially computes mapping quality across the non-redundant content of the primary assembly plus the ALT contigs and is free of the problem above.

3. Methods

Sequence alignment

As of now, ALT mapping is done in two separate steps: BWA-MEM mapping and postprocessing. The bwa.kit/run-bwamemscript performs the two steps when ALT contigs are present. The following picture shows an example about how BWA-MEM infers mapping quality and reports alignment after step 2:

Step 1: BWA-MEM mapping

At this step, BWA-MEM reads the ALT contig names from "idxbase.alt", ignoring the ALT-to-ref alignment, and labels a potential hit as ALT or non-ALT, depending on whether the hit lands on an ALT contig or not. BWA-MEM then reports alignments and assigns mapQ following these two rules:

1. The mapQ of a non-ALT hit is computed across non-ALT hits only. The mapQ of an ALT hit is computed across all hits.

2. If there are no non-ALT hits, the best ALT hit is outputted as the primary alignment. If there are both ALT and non-ALT hits, non-ALT hits will be primary and ALT hits be supplementary (SAM flag 0x800).

In theory, non-ALT alignments from step 1 should be identical to alignments against the reference genome with ALT contigs. In practice, the two types of alignments may differ in rare cases due to seeding heuristics. When an ALT hit is significantly better than non-ALT hits, BWA-MEM may miss seeds on the non-ALT hits.

If we don't care about ALT hits, we may skip postprocessing (step 2). Nonetheless, postprocessing is recommended as it improves mapQ and gives more information about ALT hits.

Step 2: Postprocessing

Postprocessing is done with a separate script bwa-postalt.js. It reads all potential hits reported in the XA tag, lifts ALT hits to the chromosomal positions using the ALT-to-ref alignment, groups them based on overlaps between their lifted positions, and then re-estimates mapQ across the best scoring hit in each group. Being aware of the ALT-to-ref alignment, this script can greatly improve mapQ of ALT hits and occasionally improve mapQ of non-ALT hits. It also writes each hit overlapping the reported hit into a separate SAM line. This enables variant calling on each ALT contig independent of others.

On the completeness of GRCh38+ALT

While GRCh38 is much more complete than GRCh37, it is still missing some true human sequences. To make sure every piece of sequence in the reference assembly is correct, the Genome Reference Consortium (GRC) require each ALT contig to have enough support from multiple sources before considering to add it to the reference assembly. This careful and sophisticated procedure has left out some sequences, one of which is this example, a 10kb contig assembled from CHM1 short reads and present also in NA12878. You can try BLAT or BLAST to see where it maps.

For a more complete reference genome, we compiled a new set of decoy sequences from GenBank clones and the de novo assembly of 254 public SGDP samples. The sequences are included in hs38DH-extra.fa from the BWA binary package.

In addition to decoy, we also put multiple alleles of HLA genes in hs38DH-extra.fa. These genomic sequences were acquired from IMGT/HLA, version 3.18.0 and are used to collect reads sequenced from these genes.

HLA typing

HLA genes are known to be associated with many autoimmune diseases, infectious diseases and drug responses. They are among the most important genes but are rarely studied by WGS projects due to the high sequence divergence between HLA genes and the reference genome in these regions.

By including the HLA gene regions in the reference assembly as ALT contigs, we are able to effectively identify reads coming from these genes. We also provide a pipeline, which is included in the BWA binary package, to type the several classic HLA genes. The pipeline is conceptually simple. It de novo assembles sequence reads mapped to each gene, aligns exon sequences of each allele to the assembled contigs and then finds the pairs of alleles that best explain the contigs. In practice, however, the completeness of IMGT/HLA and copy-number changes related to these genes are not so straightforward to resolve. HLA typing may not always be successful. Users may also consider to use other programs for typing such as Warren et al (2012), Liu et al (2013), Bai et al (2014) and Dilthey et al (2014), though most of them are distributed under restrictive licenses.

4. Preliminary Evaluation

To check whether GRCh38 is better than GRCh37, we mapped the CHM1 and NA12878 unitigs to GRCh37 primary (hs37), GRCh38 primary (hs38) and GRCh38+ALT+decoy (hs38DH), and called small variants from the alignment. CHM1 is haploid. Ideally, heterozygous calls are false positives (FP). NA12878 is diploid. The true positive (TP) heterozygous calls from NA12878 are approximately equal to the difference between NA12878 and CHM1 heterozygous calls. A better assembly should yield higher TP and lower FP. The following table shows the numbers for these assemblies:

With this measurement, hs38 is clearly better than hs37. Genome hs38DH reduces FP by ~25k but also reduces TP by ~12k. We manually inspected variants called from hs38 only and found the majority of them are associated with excessive read depth, clustered variants or weak alignment. We believe most hs38-only calls are problematic. In addition, if we compare two NA12878 replicates from HiSeq X10 with nearly identical library construction, the difference is ~140k, an order of magnitude higher than the difference between hs38 and hs38DH. ALT contigs, decoy and HLA genes in hs38DH improve variant calling and enable the analyses of ALT contigs and HLA typing at little cost.

5. Problems and Future Development

There are some uncertainties about ALT mappings - we are not sure whether they help biological discovery and don't know the best way to analyze them. Without clear demand from downstream analyses, it is very difficult to design the optimal mapping strategy. The current BWA-MEM method is just a start. If it turns out to be useful in research, we will probably rewrite bwa-postalt.js in C for performance; if not, we may make changes. It is also possible that we might make breakthrough on the representation of multiple genomes, in which case, we can even get rid of ALT contigs for good.