Development manual for vsearch 2.32.0, built from the dev branch: it describes changes that are not released yet. The manual for the current release is published separately.

NAME

vsearch --fastq_filter — trim and filter fastq sequences

SYNOPSIS

DESCRIPTION

The vsearch command --fastq_filter trims and filters the sequences in a fastq file (see vsearch-fastq(5)). It is similar to --fastx_filter (see vsearch-fastx_filter(1)), but restricted to fastq input only.

Sequences are first trimmed, then filtered based on the remaining bases:

Trimming shortens sequences using --fastq_stripleft, --fastq_stripright, --fastq_truncee, --fastq_truncee_rate, --fastq_trunclen, --fastq_trunclen_keep, and --fastq_truncqual.
Filtering discards sequences that do not satisfy criteria set by --fastq_maxee, --fastq_maxee_rate, --fastq_maxlen, --fastq_maxns, --fastq_minlen (default 1), --fastq_minqual, --fastq_trunclen, --maxsize, and --minsize.

If no trimming or filtering options are given, all sequences are written to the output files, possibly after conversion to fasta format.

For paired-end reads, the file with reverse reads is specified with --reverse, and the corresponding outputs are written to --fastaout_rev, --fastqout_rev, --fastaout_discarded_rev, and --fastqout_discarded_rev. Both reads of a pair must pass all filters for either to be retained; if one fails, both are discarded.

Discarded sequences are written to the *_discarded files in their trimmed form: truncation (e.g. --fastq_truncqual) is applied before the filters, so a read truncated and then rejected appears truncated in the discarded output. With --reverse, the kept/discarded counts in the final report refer to pairs, not to individual reads.

After processing, vsearch reports the number of sequences kept and discarded, and how many of the kept sequences were trimmed. Use --eeout or --fastq_eeout to annotate output headers with the expected error count. See vsearch-expected_error(7) for details on expected errors.

To illustrate the effect of --fastq_trunclen 6 --fastq_maxee 1.0:

Input (3 reads):   --fastq_trunclen 6 --fastq_maxee 1.0:

@r1                kept (--fastqout):
ACGTACGTAC         @r1 (truncated to ACGTAC; expected error < 1.0)
@r2         -->
ACGT               discarded (--fastqout_discarded):
@r3                @r2 (4 bases < 6, discarded by --fastq_trunclen)
NNNNNNNNNN         @r3 (expected error > 1.0 after truncation)

OPTIONS

mandatory options

At least one output option must be specified: --fastaout, --fastaout_discarded, --fastaout_discarded_rev, --fastaout_rev, --fastqout, --fastqout_discarded, --fastqout_discarded_rev, or --fastqout_rev.

core options

trimming

--fastq_stripleft positive integer: Remove the specified number of bases from the 5’ (left) end of each read. Reads reduced to zero length are discarded.
--fastq_stripright positive integer: Remove the specified number of bases from the 3’ (right) end of each read. Reads reduced to zero length are discarded.
--fastq_truncee non-negative real: Truncate reads so that their cumulative expected error does not exceed real. Truncation occurs at the first position where the threshold would be exceeded. Negative arguments are rejected. See vsearch-expected_error(7).
--fastq_truncee_rate non-negative real: Truncate reads so that their average expected error per base does not exceed real. Truncation occurs at the first position where the threshold would be exceeded. The average expected error per base is the total expected error divided by the length of the truncated sequence. Negative arguments are rejected. See vsearch-expected_error(7).
--fastq_trunclen positive integer: Truncate reads to positive integer bases. Reads shorter than positive integer bases are discarded.
--fastq_trunclen_keep positive integer: Truncate reads to positive integer bases. Reads shorter than positive integer bases are kept.
--fastq_truncqual positive integer: Truncate reads starting at the first base whose quality score is at or below positive integer. Accepted values range from 0 to 93.

filtering

--fastq_maxee positive real: Discard sequences with an expected error greater than real. The expected error is the sum of error probabilities for all positions in the sequence, and is strictly positive (zero or negative arguments are rejected, as they would discard every sequence). Applied after trimming. See vsearch-expected_error(7).
--fastq_maxee_rate non-negative real: Discard sequences with an average expected error per base greater than real. The average expected error per base is the total expected error divided by the sequence length, and is always in the [0.0, 1.0] range; negative arguments are rejected, and values greater than 1.0 effectively disable the filter. Applied after trimming. See vsearch-expected_error(7).
--fastq_maxlen positive integer: Discard sequences longer than positive integer bases. Applied after trimming.
--fastq_maxns positive integer: Discard sequences containing more than positive integer ambiguous bases (N).
--fastq_minlen positive integer: Discard sequences shorter than positive integer bases. Applied after trimming. Default is 1.
--fastq_minqual positive integer: Discard sequences containing any base with a quality score below positive integer. Default is 0 (no filtering).
--maxsize positive integer: Discard sequences with an abundance greater than positive integer.
--minsize positive integer: Discard sequences with an abundance smaller than positive integer.

output

--eeout: Add the expected error count to each sequence header in output fasta or fastq files, as the annotation ;ee=float. Synonym of --fastq_eeout. Use --xee to remove this annotation from headers. See vsearch-expected_error(7).
--fastq_eeout: Add the expected error count to each sequence header in output fasta or fastq files, as the annotation ;ee=float. Synonym of --eeout. Use --xee to remove this annotation from headers. See vsearch-expected_error(7).
--fastaout filename: Write sequences passing the filter to filename, in fasta format. When --reverse is used, only forward reads are written here.
--fastaout_discarded filename: Write the sequences not selected to filename, in fasta format.
--fastaout_discarded_rev filename: Write the reverse reads not passing the filter to filename, in fasta format. Requires --reverse.
--fastaout_rev filename: Write the reverse reads passing the filter to filename, in fasta format. Requires --reverse.
--fastqout filename: Write sequences passing the filter to filename, in fastq format (see vsearch-fastq(5)). Requires fastq input. When --reverse is used, only forward reads are written here.
--fastqout_discarded filename: Write the sequences not selected to filename, in fastq format. Requires input in fastq format.
--fastqout_discarded_rev filename: Write the reverse reads not passing the filter to filename, in fastq format. Requires --reverse and fastq input.
--fastqout_rev filename: Write the reverse reads passing the filter to filename, in fastq format. Requires --reverse and fastq input.
--reverse filename: Specify the FASTQ file containing the reverse reads.
--sizein: Use the abundance annotations present in sequence headers when reading fasta or fastq file. Search for the pattern [>@;]size=integer[;]. Entries without abundance annotations are silently assumed to be of size=1.
--sizeout: Add abundance annotations to sequence headers when writing fasta or fastq files. Add the pattern ;size=integer. Existing ;size= annotations are reported unchanged; entries without one receive ;size=1. For this command --sizein is not needed: abundance annotations are always parsed from the input headers.

secondary options

--bzip2_decompress: Specify that the input pipe is streaming data compressed using Huffman coding. See bzip2(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with bzip2, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.
--fasta_width positive integer: Set the maximal width of sequences when writing fasta files. Longer sequences are folded and written on several lines. Default width is 80 nucleotides. Set to zero (0) to suppress folding.
--fastq_ascii 33|64: Specify the offset used as the basis for the fastq quality score when reading fastq files. For example, an offset of 33 means that a quality value of 41 is represented by the 74th ASCII symbol (33 + 41 = 74), which is ‘J’. See ascii(7) for a view of the ASCII character set. The offset value is either 33 or 64, default is 33.

The offset matters even to a command that never decodes a quality score: it is what the reader compares the observed quality symbols against before warning that the file may use the other encoding, it sets the default of --fastq_qmax (the highest score the offset can represent), and the sum rules on --fastq_qmin and --fastq_qmax are stated in terms of it.

--fastq_qmax integer: Specify the maximal quality score accepted when reading fastq sequences. Stop with an error message if a quality score higher than the specified value is read. The offset (see --fastq_ascii) plus the maximal score may not exceed 126, the last printable ASCII character: scores up to 93 with offset 33, up to 62 with offset 64. The default is the highest score the offset can represent (93 with offset 33, 62 with offset 64), so no quality score is rejected unless this option is lowered. Before version 2.32.0 the default was 41, the usual maximum for Sanger/Illumina 1.8+ files, which rejected PacBio HiFi and nanopore files outright.
--fastq_qmin integer: Specify the minimal quality score accepted when reading fastq sequences. Stop with an error message if a quality score lower than the specified value is read. The offset (see --fastq_ascii) plus the minimal score must be at least 33, the first printable ASCII character: scores down to 0 with offset 33, down to -31 with offset 64. The value may therefore be negative, which is what the negative scores of older formats require, but note that those formats are not supported (see vsearch-fastq(5)). The default is 0, which is usual for recent Sanger/Illumina 1.8+ files.
--gzip_decompress: Specify that the input pipe is streaming data compressed using Lempel-Ziv coding. See gzip(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with gzip, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.
--label_suffix string: Add the suffix string to sequence headers when writing fasta or fastq files. For example, with --label_suffix ";status=healthy", sequence header ‘>seq1’ becomes ‘>seq1;status=healthy’.
--lengthout: Add a sequence length annotation (;length=integer) to each sequence header when writing fasta or fastq files.
--log filename: Write messages to filename. Messages include program version, start and finish times, elapsed time, amount of memory available, maximum amount of memory consumed, number of cores and command line options, and if need be, command-specific informational messages, warnings, and errors. Messages are written in addition to the standard error, not instead of it; use --quiet to silence the standard error copy (see the --quiet entry for what it spares).
--no_progress: Suppress the gradually increasing progress indicator normally written to the standard error stderr(3).
--quiet: Suppress messages to the standard output stdout(3) and standard error stderr(3), except for warnings and error messages.
--relabel string: Replace sequence headers with the prefix string and a ticker (1, 2, 3, etc.). For example, with --relabel "cluster:", the first sequence header becomes ‘>cluster:1’, the second sequence header becomes ‘>cluster:2’, and so on. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.
--relabel_keep: Retain old sequence identifiers by including them at the end of the new headers, after a space.
--relabel_md5: Replace each sequence header with the MD5 digest derived from the sequence itself. The sequence is converted to upper case, and each ‘U’ is replaced with a ‘T’ before computation of the digest. The MD5 digest is a 128-bit value (16 bytes), represented using a string of 32 ASCII characters. Each pair of characters encodes an hexadecimal value, ranging from x00 to xff. See md5(3) for more details, and --relabel_sha1 for an alternative hashing algorithm. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.
--relabel_self: Replace each sequence header with the sequence itself. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.
--relabel_sha1: Replace each sequence header with the SHA1 digest derived from the sequence itself. The sequence is converted to upper case, and each ‘U’ is replaced with a ‘T’ before computation of the digest. The SHA1 digest is a 160-bit value (20 bytes), represented using a string of 40 ASCII characters. Each pair of characters encodes an hexadecimal value, ranging from x00 to xff. See sha1(3) for more details, and --relabel_md5 for an alternative hashing algorithm. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.
--sample string: Add the given sample identifier string to sequence headers when writing fasta or fastq files. For instance, if string is ‘ABC’, the text ;sample=ABC will be added to the headers. string is silently truncated at the first ‘;’ or whitespace character (space, tab, newline, carriage return, vertical tab or form feed), so such characters should not be used in string. Other characters (alphabetical, numerical and punctuations) are accepted. When nothing is left after truncation — an empty string, or one starting with ‘;’ or a blank character — vsearch issues a warning and writes a bare ;sample= annotation.
--xee: Strip expected error (ee) annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]ee=float[;]. Expected error annotations are added by the synonymous options --fastq_eeout and --eeout described in vsearch-fastx_filter(1).
--xlength: Strip sequence length annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]length=integer[;]. Sequence length annotations are added by the --lengthout option.
--xsize: Strip abundance annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]size=integer[;]. Abundance annotations are added by the --sizeout option.

ignored options

--threads positive non-null integer: Command is not multithreaded, option has no effect (a warning is printed when more than one thread is requested).

EXAMPLES

Filter a fastq file, discarding reads with an expected error above 1.0, and write kept sequences in fasta format:

vsearch \
    --fastq_filter input.fastq \
    --fastq_maxee 1.0 \
    --fastaout filtered.fasta

Truncate reads to 250 bases, then discard those with an expected error above 0.5, annotating output headers with the expected error:

vsearch \
    --fastq_filter input.fastq \
    --fastq_trunclen 250 \
    --fastq_maxee 0.5 \
    --eeout \
    --fastqout filtered.fastq

Strip a fixed-length primer from the left, then filter by minimum length and maximum expected error rate:

vsearch \
    --fastq_filter input.fastq \
    --fastq_stripleft 20 \
    --fastq_minlen 100 \
    --fastq_maxee_rate 0.01 \
    --fastqout trimmed.fastq \
    --fastqout_discarded discarded.fastq

Filter paired-end reads, keeping only pairs where both reads pass:

vsearch \
    --fastq_filter forward.fastq \
    --reverse reverse.fastq \
    --fastq_maxee 1.0 \
    --fastqout kept_fwd.fastq \
    --fastqout_rev kept_rev.fastq \
    --fastqout_discarded discarded_fwd.fastq \
    --fastqout_discarded_rev discarded_rev.fastq

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

These manual pages are also published online at https://torognes.github.io/vsearch/.

COPYRIGHT

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available:

vsearch includes code from Google’s CityHash project by Geoff Pike and Jyrki Alakuijala, providing some excellent hash functions available under a MIT license.
vsearch includes code derived from Tatusov and Lipman’s DUST program that is in the public domain.
vsearch includes public domain code written by Alexander Peslyak for the MD5 message digest algorithm.
vsearch includes public domain code written by Steve Reid and others for the SHA1 message digest algorithm.
vsearch binaries may include code from the zlib library, copyright Jean-Loup Gailly and Mark Adler.
vsearch binaries may include code from the bzip2 library, copyright Julian R. Seward.