Development manual for vsearch 2.32.0, built from the dev branch: it describes changes that are not released yet. The manual for the current release is published separately.

NAME

vsearch --search_exact — search for exact full-length matches against a database

SYNOPSIS

DESCRIPTION

The vsearch command --search_exact searches the query sequences in a fasta or fastq file against a database of target sequences (--db), reporting only 100% exact full-length matches. It is much faster than --usearch_global for this use case.

Because only exact matches are reported, --id, --maxaccepts, and --maxrejects do not apply and are not accepted. Matching is case-insensitive and treats U and T as equivalent. Both sides are normalized before hashing, so the masking options (--qmask, --dbmask) never change which matches are found: they only affect the case of sequences in the output files — except --hardmask combined with soft masking, which replaces masked residues with N and does prevent matches. By default only the plus strand is searched; use --strand both to also check the reverse complement.

Since masking cannot change the results, it can be switched off with --dbmask none --qmask none, and that is usually worth doing: the whole database is masked before the first query is read, and masking accounts for half to two thirds of the total run time. The outputs that report labels and numbers only — --biomout, --blast6out, --mothur_shared_out, --otutabout, --uc, and --userout when it requests neither the qrow nor the trow field — are then byte-for-byte identical. The outputs that echo sequences differ only in the case of the sequences they print. Note that a masked database cannot be reused from one run to the next: unlike --usearch_global, this command does not accept a UDB database (see vsearch-makeudb_usearch(1)).

At least one output option must be specified. This command is multi-threaded: the queries are distributed over the available threads, so the order of the entries written to --alnout, --blast6out, --fastapairs, --matched, --notmatched, --qsegout, --samout, --tsegout, --uc and --userout may vary from run to run when more than one thread is used. The --biomout, --dbmatched, --dbnotmatched, --mothur_shared_out and --otutabout tables are assembled after the search, or written in database order, and keep a stable order. The results themselves do not depend on the thread count.

OPTIONS

mandatory options

--db must be specified, along with at least one output option.

--db filename: Search query sequences against the target sequences in filename, in fasta or fastq format. --db accepts - to read the database from standard input, as well as an explicit stream path such as /dev/stdin, a named pipe, or a process substitution. The query and the database cannot both be -, however, as they would compete for the same standard input; give at least one of them an explicit path.

core options

--dbmask none|dust|soft: Mask regions in the database sequences using the dust method or the soft method, or none to suppress masking. See vsearch-fastx_mask(1) for more details. Warning, when using soft masking, search commands become case sensitive: masking excludes masked regions from the k-mer index used to select candidate targets (the pairwise alignment itself always ignores case). A database sequence with no unmasked stretch of at least the word length contributes no k-mer to the index and is never selected as a candidate target; it can still be reported when the query itself samples no k-mer, as such a query is compared against every database sequence (see --qmask). vsearch warns when there are such sequences, giving how many. The default is to mask using dust.
--qmask none|dust|soft: Mask regions in query sequences using the dust method or the soft method, or none to suppress masking. Values are case-insensitive, so DUST, Dust, and dust are all accepted. See vsearch-fastx_mask(1) for more details. Warning, when using soft masking, search commands become case sensitive: masking excludes masked regions from the k-mer pre-filter that selects candidate targets (the pairwise alignment itself always ignores case). A query with no unmasked stretch of at least the word length samples no k-mers and is therefore compared against every database sequence. The default is to mask using dust.
--strand plus|both: Check the plus strand only (default), or check both strands when comparing sequences. Keywords are case-insensitive.
--threads positive integer: Set the number of computation threads to use, from 1 to 1024. The number of threads should not exceed the number of available CPU cores. The value 0 is also accepted and, like the default, uses all available cores; decimal values are truncated to their integer part. On Linux, “available” accounts for the CPU affinity mask and the cgroup CPU quota of the running process, so a job confined by taskset(1), Slurm, Docker or Kubernetes launches one thread per core it was actually granted, rather than one per core the machine has.

secondary options

--alnout filename

Write pairwise global alignments to filename in a human-readable format. Use --rowlen to set the alignment line width.

--biomout filename

Write an OTU table to filename in the biom version 1.0 JSON file format. The OTUs are represented by the cluster centroids. Sample identifiers are extracted from sequence headers (;sample=abc123; or ;barcodelabel=abc123; patterns, or the initial part of the header). OTU identifiers are extracted from centroid headers (;otu=def789; pattern, or the initial part of the header, or via relabelling options). Taxonomy information is extracted from centroid headers (;tax=...; pattern) if available. Abundance annotations (;size=integer) present in sequence headers are always used when filling the table, whether or not --sizein is given (unlike the --uc cluster summaries, which count each sequence as 1 without --sizein). OTU identifiers must be unique: two OTUs sharing the same identifier are reported as a single observation, and their abundances are summed. When clustering, a relabelling option (--relabel, --relabel_self, --relabel_md5 or --relabel_sha1) guarantees unique identifiers; when searching, the database itself must have unique headers.

--blast6out filename

Write results to filename using a BLAST-like tab-separated format with twelve fields per query-target match: query label, target label, percentage identity, alignment length, mismatches, gap openings, query start, query end, target start, target end, expectation value (always -1), and bit score (always 0). If --output_no_hits is used, non-matching queries are also written. Note that vsearch uses global pairwise alignments, not BLAST’s seed-and-extend algorithm.

--bzip2_decompress

Specify that the input pipe is streaming data compressed using Huffman coding. See bzip2(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with bzip2, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.

--dbmatched filename

Write database target sequences that match at least one query sequence to filename, in fasta format. If --sizeout is specified, the number of matching queries is annotated in the header (;size=integer).

--dbnotmatched filename

Write database target sequences that do not match any query sequence to filename, in fasta format.

--fasta_width positive integer

Set the maximal width of sequences when writing fasta files. Longer sequences are folded and written on several lines. Default width is 80 nucleotides. Set to zero (0) to suppress folding.

--fastapairs filename

Write pairwise alignments of query and target sequences to filename, in fasta format.

--gzip_decompress

Specify that the input pipe is streaming data compressed using Lempel-Ziv coding. See gzip(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with gzip, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.

--hardmask

Replace masked nucleotides with Ns, rather than lowercasing them.

This is also the only masking option that reaches the alignment. Soft and dust masking (see --qmask and --dbmask) only keep masked words out of the k-mer pre-filter that selects candidates; the masked region is still aligned and scored like any other. An N, on the other hand, scores zero and counts as a matching column, so hard masking makes a masked region match whatever it is aligned against, and the reported identity can only go up. Add --n_mismatch to count those columns as mismatches instead.

--label_suffix string

Add the suffix string to sequence headers when writing fasta or fastq files. For example, with --label_suffix ";status=healthy", sequence header ‘>seq1’ becomes ‘>seq1;status=healthy’.

--lengthout

Add a sequence length annotation (;length=integer) to each sequence header when writing fasta or fastq files.

--log filename

Write messages to filename. Messages include program version, start and finish times, elapsed time, amount of memory available, maximum amount of memory consumed, number of cores and command line options, and if need be, command-specific informational messages, warnings, and errors. Messages are written in addition to the standard error, not instead of it; use --quiet to silence the standard error copy (see the --quiet entry for what it spares).

--matched filename

Write query sequences matching a target sequence to filename, in fasta format.

--maxhits non-negative integer

Set the maximum number of hits to report once the search is terminated for a given query; hits are sorted by decreasing identity. Unlimited by default, or when the argument is zero. When searching both strands, --maxhits controls the total number of hits reported per query across both strands.

--maxqsize positive integer

Reject query sequences with an abundance greater than integer.

--maxqt real

Reject the sequence match if the query/target sequence length ratio is greater than real.

--maxseqlength positive integer

Discard sequences longer than positive integer (50,000 nucleotides by default). The value must not exceed 2,147,481,646 (INT_MAX minus 2,001). When that value is smaller than the effective --minseqlength value (whose default is command-specific), no sequence can pass the length filter: vsearch issues a warning.

--maxsizeratio real

Reject the sequence match if the query/target abundance ratio is greater than real.

--maxsl real

Reject the sequence match if the shorter/longer sequence length ratio is greater than real.

--mincols positive integer

Reject the sequence match if the alignment length is shorter than integer columns.

--minqt real

Reject the sequence match if the query/target sequence length ratio is lower than real.

--minseqlength non-negative integer

Discard sequences shorter than non-negative integer (1 nucleotide by default). A value of 0 retains empty sequences. When that value is larger than the --maxseqlength value, no sequence can pass the length filter: vsearch issues a warning.

--minsizeratio real

Reject the sequence match if the query/target abundance ratio is lower than real.

--minsl real

Reject the sequence match if the shorter/longer sequence length ratio is lower than real.

--mintsize positive integer

Reject target sequences with an abundance lower than integer.

--mothur_shared_out filename

Write an OTU table to filename in the mothur ‘shared’ tab-separated plain text format. The first line starts with label, group and numOtus, followed by all OTU identifiers. Each subsequent line starts with vsearch, the sample identifier, the total number of OTUs, and the abundance of each OTU in that sample. Sample and OTU identifiers are extracted from FASTA headers. OTUs are represented by the cluster centroids. Abundance annotations (;size=integer) present in sequence headers are always used when filling the table, whether or not --sizein is given (unlike the --uc cluster summaries, which count each sequence as 1 without --sizein). OTU identifiers must be unique: two OTUs sharing the same identifier are reported in a single column, numOtus counts that column only once, and their abundances are summed. When clustering, a relabelling option (--relabel, --relabel_self, --relabel_md5 or --relabel_sha1) guarantees unique identifiers; when searching, the database itself must have unique headers.

--no_progress

Suppress the gradually increasing progress indicator normally written to the standard error stderr(3).

--notmatched filename

Write the sequences that were not extracted to filename, in fasta format.

--notrunclabels

Retain whole sequence headers in output files. By default, vsearch truncates sequence headers at first space or tabulation. This option suppresses truncation.

--otutabout filename

Write an OTU table to filename in a classic tab-separated plain text format. The first line starts with #OTU ID followed by sample identifiers. Each subsequent line starts with the OTU identifier followed by the abundances in each sample. Sample and OTU identifiers are extracted from FASTA headers (see --sample). OTUs are represented by the cluster centroids. A taxonomy column is appended if taxonomy information is available for any OTU. Abundance annotations (;size=integer) present in sequence headers are always used when filling the table, whether or not --sizein is given (unlike the --uc cluster summaries, which count each sequence as 1 without --sizein). Rows and columns are both ordered lexicographically by identifier, whatever order the OTUs were created in or the samples first appeared in, and no option changes that. Sorting the table by decreasing abundance is a job for a downstream tool.

OTU identifiers must be unique: two OTUs sharing the same identifier are reported on a single line, and their abundances are summed. When clustering, a relabelling option (--relabel, --relabel_self, --relabel_md5 or --relabel_sha1) guarantees unique identifiers; when searching, the database itself must have unique headers.

--output_no_hits

Write both matching and non-matching queries to --alnout, --blast6out, --samout, or --userout output files. Non-matching queries are labelled ‘No hits’ in --alnout files.

--qsegout filename

Write the aligned part of each query sequence to filename, in fasta format.

--quiet

Suppress messages to the standard output stdout(3) and standard error stderr(3), except for warnings and error messages.

--relabel string

Replace sequence headers with the prefix string and a ticker (1, 2, 3, etc.). For example, with --relabel "cluster:", the first sequence header becomes ‘>cluster:1’, the second sequence header becomes ‘>cluster:2’, and so on. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.

--relabel_keep

Retain old sequence identifiers by including them at the end of the new headers, after a space.

--relabel_md5

Replace each sequence header with the MD5 digest derived from the sequence itself. The sequence is converted to upper case, and each ‘U’ is replaced with a ‘T’ before computation of the digest. The MD5 digest is a 128-bit value (16 bytes), represented using a string of 32 ASCII characters. Each pair of characters encodes an hexadecimal value, ranging from x00 to xff. See md5(3) for more details, and --relabel_sha1 for an alternative hashing algorithm. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.

--relabel_self

Replace each sequence header with the sequence itself. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.

--relabel_sha1

Replace each sequence header with the SHA1 digest derived from the sequence itself. The sequence is converted to upper case, and each ‘U’ is replaced with a ‘T’ before computation of the digest. The SHA1 digest is a 160-bit value (20 bytes), represented using a string of 40 ASCII characters. Each pair of characters encodes an hexadecimal value, ranging from x00 to xff. See sha1(3) for more details, and --relabel_md5 for an alternative hashing algorithm. To retain annotations, use their corresponding options (--lengthout, --eeout, and --sizeout). Use --relabel_keep to also retain old sequence identifiers.

--rowlen positive integer

Set the width of alignment lines in --alnout output. The default value is 64. Set to 0 to disable line wrapping.

--samheader

Include header lines (@HD, @SQ, @PG) in the SAM file produced by --samout. By default, no header lines are written.

--samout filename

Write alignment results to filename in the SAM format, see vsearch-sam(5). Use --samheader to include header lines. Each non-header line is a SAM record representing either a query-target alignment or the absence of a match. The alignment column of each record uses the CIGAR format, see vsearch-cigar(5).

--sample string

Add the given sample identifier string to sequence headers when writing fasta or fastq files. For instance, if string is ‘ABC’, the text ;sample=ABC will be added to the headers. string is silently truncated at the first ‘;’ or whitespace character (space, tab, newline, carriage return, vertical tab or form feed), so such characters should not be used in string. Other characters (alphabetical, numerical and punctuations) are accepted. When nothing is left after truncation — an empty string, or one starting with ‘;’ or a blank character — vsearch issues a warning and writes a bare ;sample= annotation.

--self

Reject the sequence match if the query and target sequence labels are identical. The label is the header up to the first blank, so two records sharing an identifier but carrying different descriptions still reject each other; with --notrunclabels the whole header is the label and they no longer do. Use --selfid to reject on identical sequences rather than identical labels.

--sizein

Use the abundance annotations present in sequence headers when reading fasta or fastq file. Search for the pattern [>@;]size=integer[;]. Entries without abundance annotations are silently assumed to be of size=1.

--sizeout

Add abundance annotations to sequence headers when writing fasta or fastq files. Add the pattern ;size=integer. Existing ;size= annotations are reported unchanged; entries without one receive ;size=1. For this command --sizein is not needed: abundance annotations are always parsed from the input headers.

--top_hits_only

Report only the hits with the highest pairwise identity for each query.

--tsegout filename

Write the aligned part of each target sequence to filename, in fasta format.

--uc filename

Write search results to filename in a tab-separated uclust-like format with 10 columns. Each query produces a hit (H) or no-hit (N) record. Use --uc_allhits to report all hits per query instead of just the top hit. Columns are:

record type: H (hit) or N (no hit);
ordinal number of the target sequence (zero-based; * for N);
length of the query sequence (* for N);
percentage of identity with the target (* for N);
match orientation + or - (. for N);
not used; 0 (H) or * (N);
not used; 0 (H) or * (N);
CIGAR alignment string (* for N; = if the query and target sequences are strictly identical); see vsearch-cigar(5);
query label;
target label (* for N).

--uc_allhits

When using --uc, report all hits for each query, not just the top hit.

--userfields string

Select and order the fields written to --userout output. Fields are separated by + (e.g. query+target+id). See vsearch-userfields(7) for a complete description of all available fields.

--userout filename

Write user-defined tab-separated output to filename. Select and order the fields with --userfields.

--xee

Strip expected error (ee) annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]ee=float[;]. Expected error annotations are added by the synonymous options --fastq_eeout and --eeout described in vsearch-fastx_filter(1).

--xlength

Strip sequence length annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]length=integer[;]. Sequence length annotations are added by the --lengthout option.

--xsize

Strip abundance annotations from sequence headers when writing fasta or fastq files. Search for the pattern [>@;]size=integer[;]. Abundance annotations are added by the --sizeout option.

pairwise alignment options

These options modify the pairwise alignment scoring model used when writing alignment output (e.g., --alnout). Modify with caution.

--match integer: Set the score assigned to a match (i.e. equivalent nucleotides) in pairwise alignments. The default value is 2. Accepted values range from -32767 to +32767.
--mismatch integer: Set the score assigned to a mismatch (i.e. different nucleotides) in pairwise alignments. The default value is -4. Accepted values range from -32767 to +32767.

EXAMPLES

Find all exact matches in a database and write results in BLAST-like tabular format:

vsearch \
    --search_exact queries.fasta \
    --db reference.fasta \
    --blast6out results.tsv

Search both strands and write matching and non-matching queries to separate fasta files:

vsearch \
    --search_exact queries.fasta \
    --db reference.fasta \
    --strand both \
    --matched exact_hits.fasta \
    --notmatched no_hits.fasta

Build an OTU table by mapping dereplicated reads to OTU centroids using exact matching:

vsearch \
    --search_exact reads.fasta \
    --db otus.fasta \
    --otutabout otu_table.tsv \
    --threads 8

The same run with masking switched off. --otutabout reports labels and counts, so the table is byte-for-byte the same, and the run is roughly twice as fast:

vsearch \
    --search_exact reads.fasta \
    --db otus.fasta \
    --otutabout otu_table.tsv \
    --qmask none \
    --dbmask none \
    --threads 8

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

These manual pages are also published online at https://torognes.github.io/vsearch/.

COPYRIGHT

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available:

vsearch includes code from Google’s CityHash project by Geoff Pike and Jyrki Alakuijala, providing some excellent hash functions available under a MIT license.
vsearch includes code derived from Tatusov and Lipman’s DUST program that is in the public domain.
vsearch includes public domain code written by Alexander Peslyak for the MD5 message digest algorithm.
vsearch includes public domain code written by Steve Reid and others for the SHA1 message digest algorithm.
vsearch binaries may include code from the zlib library, copyright Jean-Loup Gailly and Mark Adler.
vsearch binaries may include code from the bzip2 library, copyright Julian R. Seward.