Development manual for vsearch 2.32.0, built from the dev branch: it describes changes that are not released yet. The manual for the current release is published separately.

NAME

vsearch --makeudb_usearch — create a UDB database file from a fasta or fastq file

SYNOPSIS

vsearch --makeudb_usearch fastxfile --output dbfile [options]

DESCRIPTION

The vsearch command --makeudb_usearch creates a UDB database file from the sequences in fastxfile, in fasta or fastq format (quality values are ignored and are not stored in the database). The UDB file is a binary format that contains the sequences together with a k-mer index, and can be quickly loaded into memory. Using a UDB file avoids re-indexing the database on every run, which is worthwhile when the same database is searched repeatedly with --usearch_global or --sintax.

Both --makeudb_usearch and --output must be specified.

The database must contain at least one sequence. vsearch reports a fatal error and stops if fastxfile is empty, or if all of its sequences are discarded by --minseqlength (32 nucleotides by default for this command), since a UDB file recording no sequence cannot be read back.

The UDB file holds one 32-bit counter per possible k-mer, whether or not the database contains that k-mer. Those 4^wordlength counters are 256 kB at the default word length of 8, but 1 GB at word length 14 and 4 GB at 15, whatever the size of fastxfile, so raising --wordlength grows the output file as well as the memory needed to build it (see --wordlength).

Of the work this command performs, only DUST masking is distributed over several threads. Reading the input, building the k-mer index and writing the UDB file are single-threaded, and so is the letter replacement --hardmask asks for. --threads therefore shortens the masking step and nothing else: it helps with the default --dbmask dust, and has no measurable effect with --dbmask none or --dbmask soft. What it can save is bounded by the share of the run spent masking, which grows with the length of the sequences, and on short ones the gain stops increasing beyond a handful of threads.

See vsearch-udb(5) for a description of the UDB file format.

OPTIONS

mandatory options

--makeudb_usearch fastxfile: Read fasta or fastq sequences from fastxfile and create a UDB database (quality values are ignored).
--output filename: Write the UDB database to filename.

core options

--dbmask none|dust|soft

Mask regions in the database sequences using the dust method or the soft method, or none to suppress masking. See vsearch-fastx_mask(1) for more details. Warning, when using soft masking, search commands become case sensitive: masking excludes masked regions from the k-mer index used to select candidate targets (the pairwise alignment itself always ignores case). A database sequence with no unmasked stretch of at least the word length contributes no k-mer to the index and is never selected as a candidate target; it can still be reported when the query itself samples no k-mer, as such a query is compared against every database sequence (see --qmask). vsearch warns when there are such sequences, giving how many. The default is to mask using dust.

--hardmask

Replace masked nucleotides with Ns, rather than lowercasing them.

This is also the only masking option that reaches the alignment. Soft and dust masking (see --qmask and --dbmask) only keep masked words out of the k-mer pre-filter that selects candidates; the masked region is still aligned and scored like any other. An N, on the other hand, scores zero and counts as a matching column, so hard masking makes a masked region match whatever it is aligned against, and the reported identity can only go up. Add --n_mismatch to count those columns as mismatches instead.

--wordlength positive integer

Set the length of words (i.e. k-mers) used for sequence indexing and comparisons. Valid values range from 3 to 15. The default is 8. Note that the default --minwordmatches is derived from this value, so changing one changes both (see --minwordmatches).

Longer words make the k-mer index more selective, so fewer targets are offered as candidates and the search itself gets faster: from word length 5 to 11 the search phase shrank by a factor of 2.4 on 130-nucleotide amplicons and 5.8 on full-length reference sequences. Working against that, the index has 4^wordlength slots, so the memory it needs and the time spent building it both quadruple with each added nucleotide. On a 400 000-sequence database the whole run needed 0.3 GB at word length 10, 1.3 GB at 12 and 16 GB at 15.

The best setting balances the two, and depends on how many queries are searched against a given database: with few queries the index build dominates and a shorter word is cheaper overall, while with many queries the search dominates and a longer word repays its index. Changing the word length is not output-neutral, so it should be chosen for a workload rather than tuned per run.

secondary options

--bzip2_decompress: Specify that the input pipe is streaming data compressed using Huffman coding. See bzip2(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with bzip2, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.
--gzip_decompress: Specify that the input pipe is streaming data compressed using Lempel-Ziv coding. See gzip(1) for more details. This option is required when compressed data arrives on standard input through a pipe (‘-’), where the format cannot be detected without consuming the stream. It is not needed when reading from a regular file compressed with gzip, nor when such a file is redirected to standard input: compression is then detected automatically, and a contradicting option is ignored (with a warning when the input is standard input). Pipes other than standard input, such as shell process substitutions and named FIFOs, are always read as uncompressed data; compressed data must arrive on standard input or as a named file.
--log filename: Write messages to filename. Messages include program version, start and finish times, elapsed time, amount of memory available, maximum amount of memory consumed, number of cores and command line options, and if need be, command-specific informational messages, warnings, and errors. Messages are written in addition to the standard error, not instead of it; use --quiet to silence the standard error copy (see the --quiet entry for what it spares).
--maxseqlength positive integer: Discard sequences longer than positive integer (50,000 nucleotides by default). The value must not exceed 2,147,481,646 (INT_MAX minus 2,001). When that value is smaller than the effective --minseqlength value (whose default is command-specific), no sequence can pass the length filter: vsearch issues a warning.
--minseqlength non-negative integer: Discard sequences shorter than non-negative integer (32 nucleotides by default). A value of 0 retains empty sequences. When that value is larger than the --maxseqlength value, no sequence can pass the length filter: vsearch issues a warning.
--no_progress: Suppress the gradually increasing progress indicator normally written to the standard error stderr(3).
--notrunclabels: Retain whole sequence headers in output files. By default, vsearch truncates sequence headers at first space or tabulation. This option suppresses truncation.
--quiet: Suppress messages to the standard output stdout(3) and standard error stderr(3), except for warnings and error messages.
--threads positive integer: Set the number of computation threads to use, from 1 to 1024. The number of threads should not exceed the number of available CPU cores. The value 0 is also accepted and, like the default, uses all available cores; decimal values are truncated to their integer part. On Linux, “available” accounts for the CPU affinity mask and the cgroup CPU quota of the running process, so a job confined by taskset(1), Slurm, Docker or Kubernetes launches one thread per core it was actually granted, rather than one per core the machine has.

EXAMPLES

Create a UDB database from a fasta file with default settings:

vsearch \
    --makeudb_usearch db.fasta \
    --output db.udb

Create a UDB database without masking:

vsearch \
    --makeudb_usearch db.fasta \
    --dbmask none \
    --output db.udb

Use the resulting UDB file with --usearch_global:

vsearch \
    --makeudb_usearch db.fasta \
    --output db.udb

vsearch \
    --usearch_global queries.fasta \
    --db db.udb \
    --id 0.97 \
    --blast6out results.b6

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

These manual pages are also published online at https://torognes.github.io/vsearch/.

COPYRIGHT

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available:

vsearch includes code from Google’s CityHash project by Geoff Pike and Jyrki Alakuijala, providing some excellent hash functions available under a MIT license.
vsearch includes code derived from Tatusov and Lipman’s DUST program that is in the public domain.
vsearch includes public domain code written by Alexander Peslyak for the MD5 message digest algorithm.
vsearch includes public domain code written by Steve Reid and others for the SHA1 message digest algorithm.
vsearch binaries may include code from the zlib library, copyright Jean-Loup Gailly and Mark Adler.
vsearch binaries may include code from the bzip2 library, copyright Julian R. Seward.