Development manual for vsearch 2.32.0, built from the dev branch: it describes changes that are not released yet. The manual for the current release is published separately.

NAME

expected error — a quality summary metric for fastq sequences

DESCRIPTION

The expected error (EE) of a fastq read is a single number that summarises the overall quality of that read. It is defined as the sum of the per-base error probabilities across the entire sequence:

EE = P_e(1) + P_e(2) + ... + P_e(L)

where L is the length of the read and P_e(i) is the probability that base i was incorrectly sequenced.

Computing error probabilities from quality scores

Each base quality score Q stored in a fastq file encodes its error probability according to the Phred scale:

P_e = 10^(-Q / 10)

For example:

Q score	Error probability	Accuracy
Q10	0.1	90%
Q20	0.01	99%
Q30	0.001	99.9%
Q40	0.0001	99.99%

Since error probabilities are always positive, the expected error of a sequence is always greater than zero. It is at most equal to the sequence length, which occurs when every base has an error probability of 1.0.

Why expected error is preferred over average quality

Averaging Phred quality scores directly is mathematically incorrect: quality scores are on a logarithmic scale, while error probabilities are linear. The expected error sums all per-base error probabilities, giving each base equal weight and producing a meaningful global quality estimate.

A mathematically sound Phred-scaled average does exist: convert the mean per-base error probability (EE divided by the read length L) back to the Phred scale:

Avg_Q = -10 * log10(EE / L)

vsearch does not write this value directly, but it can be computed downstream from the ;ee=float and ;length=integer header annotations added by --eeout and --lengthout. Filtering on it is equivalent to filtering on the expected error rate: keeping reads with an average quality of at least Q corresponds to --fastq_maxee_rate set to 10^(-Q/10). For example, Avg_Q >= 20 corresponds to --fastq_maxee_rate 0.01.

Poisson interpretation

Because sequencing errors are approximately independent, the expected error can be used as the lambda (λ) parameter of the Poisson distribution to estimate the probability of observing exactly k errors in a read. For a read with EE = 1.0:

36.8% chance of zero errors,
36.8% chance of one error,
18.4% chance of two errors,
6.1% chance of three errors,
1.5% chance of four errors,
0.3% chance of five errors,
etc.

vsearch options using expected error

--fastq_maxee real: Discard reads whose total expected error exceeds real. Applied after any trimming step.
--fastq_maxee_rate real: Discard reads whose average expected error per base (EE divided by read length) exceeds real (values from 0.0 to 1.0).
--fastq_truncee real: Truncate reads at the first position where the cumulative expected error would exceed real.
--fastq_truncee_rate real: Truncate reads at the first position where the average expected error per base (cumulative EE divided by current length) would exceed real.
--fastq_eeout / --eeout: Annotate output sequence headers with the expected error value as ;ee=float. Use --xee to remove this annotation from headers.

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

These manual pages are also published online at https://torognes.github.io/vsearch/.

COPYRIGHT

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available:

vsearch includes code from Google’s CityHash project by Geoff Pike and Jyrki Alakuijala, providing some excellent hash functions available under a MIT license.
vsearch includes code derived from Tatusov and Lipman’s DUST program that is in the public domain.
vsearch includes public domain code written by Alexander Peslyak for the MD5 message digest algorithm.
vsearch includes public domain code written by Steve Reid and others for the SHA1 message digest algorithm.
vsearch binaries may include code from the zlib library, copyright Jean-Loup Gailly and Mark Adler.
vsearch binaries may include code from the bzip2 library, copyright Julian R. Seward.