Development manual for vsearch 2.32.0, built from the dev branch: it describes changes that are not released yet. The manual for the current release is published separately.

NAME

udb — a binary file format containing fasta sequences and a k-mer index

DESCRIPTION

The UDB (USEARCH database) format is a binary file format used to store fasta sequences together with a pre-computed k-mer index. Building a UDB file from a fasta reference database with --makeudb_usearch allows subsequent search commands (such as --usearch_global) to load the index directly, avoiding the cost of recomputing it at each run.

How much that saves scales with the size of the reference database, and depends on how many cores the search can use. Preparing a fasta reference costs roughly 10 to 12 seconds per gigabyte of fasta with every core busy, but 55 to 85 seconds per gigabyte on a single core: masking is multi-threaded, the index build is not. Loading the same database from a UDB costs about 3 to 4 seconds per gigabyte of the original fasta, whatever the core count, since that path reads a finished index instead of building one. The saving is therefore some 6 to 8 seconds per gigabyte when all cores are busy, and 50 to 80 seconds per gigabyte on one core. The single-core figure is the one to use when a query file is split across several concurrent single-threaded vsearch processes, as each of them pays the full preparation.

Building the UDB costs 60 to 90 seconds per gigabyte, once. It therefore repays itself after about ten searches on a many-core machine, or after the second search on a single core. The file is some four times larger than the fasta it was built from (3.7 times for short amplicons, 4.5 times for full-length references), since it carries the k-mer index as well as the sequences.

These figures were measured on a 24-core desktop processor with the database in the page cache, over reference sets from 17 MB to 1.1 GB, and are meant as orders of magnitude: the underlying cost follows the number of nucleotides rather than the size of the file, so a database of long sequences with short headers sits at the top of each range.

A UDB is not always interchangeable with the fasta file it was built from. --makeudb_usearch masks with dust by default and stores the masked sequences, which matches what --usearch_global would have done to a fasta database, but not what --sintax does (see vsearch-sintax(1)).

Multi-byte numeric values are stored in the machine’s native byte order; every platform vsearch currently supports is little-endian (least significant byte first, as on x86 and ARM processors), so in practice UDB files are little-endian and are not portable to big-endian machines. This contrasts with big-endian order (most significant byte first), which is used for example by the SFF format (see vsearch-sff(5)).

UDB files cannot be read from pipes; a seekable file path must be provided.

The file consists of nine sequential sections described below.

Section 1 — Main Header

The main header is exactly 200 bytes long (50 × uint32_t):

 buffer[0]   uint32_t   magic number: 0x55444246 ("UDBF")
 buffer[1]   uint32_t   reserved, set to 0
 buffer[2]   uint32_t   sequence index bits: 32
 buffer[3]   uint32_t   reserved, set to 0
 buffer[4]   uint32_t   word length (k-mer size, default 8)
 buffer[5]   uint32_t   dbstep: 1
 buffer[6]   uint32_t   dbaccelpct: 100
 buffer[7]   uint32_t   reserved, set to 0
 ...
 buffer[11]  uint32_t   slots: 0
 buffer[12]  uint32_t   reserved, set to 0
 buffer[13]  uint32_t   number of sequences
 buffer[14]  uint32_t   reserved, set to 0
 ...
 buffer[17]  uint32_t   alphabet: 0x0000746e ("nt")
 buffer[18]  uint32_t   reserved, set to 0
 ...
 buffer[49]  uint32_t   end marker: 0x55444266 ("UDBf")

All fields not listed explicitly are reserved and set to zero.

  • The magic number field value is 0x55444246. Stored little-endian, its on-disk bytes read “FBDU”; the value spells “UDBF” when written most significant byte first.
  • The word length field stores the k-mer size used to build the index (between 3 and 15, default 8; controllable with --wordlength).
  • The dbstep field is always 1 and the dbaccelpct field is always 100, reflecting that every sequence is indexed with full coverage.
  • The number of sequences field gives the total number of sequences stored in the file. It is never zero: vsearch rejects a UDB file declaring no sequence when reading it, and refuses to write one.
  • The alphabet field value is 0x0000746e, the little-endian encoding of the ASCII string “nt” (nucleotide), indicating that this is a nucleotide database.
  • The end marker field value is 0x55444266 (on-disk bytes “fBDU”; “UDBf” most significant byte first), marking the end of the main header.

Section 2 — Word Match Counts

This section contains 4^wordlength consecutive uint32_t values (one per possible k-mer):

 kmercount[0]                uint32_t
 kmercount[1]                uint32_t
 ...
 kmercount[4^wordlength - 1] uint32_t

Each value kmercount[i] stores the number of sequences in the database that contain k-mer i at least once. With the default word length of 8, this section contains 65,536 values (256 KB).

Section 3 — UDB3 Marker

A single uint32_t sentinel value:

 marker   uint32_t   0x55444233 ("UDB3")

The value 0x55444233 is the little-endian encoding of the ASCII string “UDB3”.

Section 4 — Word Index

This section stores, for each k-mer, the sorted list of 0-based sequence numbers of all sequences containing that k-mer. The lists are concatenated in k-mer order, with no delimiters:

 seqno[0]   uint32_t   first sequence number for k-mer 0
 seqno[1]   uint32_t   second sequence number for k-mer 0
 ...        (kmercount[0] entries for k-mer 0)
 seqno[.]   uint32_t   first sequence number for k-mer 1
 ...        (kmercount[1] entries for k-mer 1)
 ...

The total number of uint32_t values in this section equals the sum of all kmercount values from section 2. A k-mer with a count of zero contributes no bytes to this section. Sequence numbers use 0-based indexing.

Section 5 — UDB4 Header

A fixed-size block of 32 bytes (8 × uint32_t):

 buffer[0]   uint32_t   magic number: 0x55444234 ("UDB4")
 buffer[1]   uint32_t   constant: 0x005e0db3
 buffer[2]   uint32_t   number of sequences
 buffer[3]   uint32_t   total nucleotide count (low 32 bits)
 buffer[4]   uint32_t   total nucleotide count (high 32 bits)
 buffer[5]   uint32_t   total header characters (low 32 bits)
 buffer[6]   uint32_t   total header characters (high 32 bits)
 buffer[7]   uint32_t   constant: 0x005e0db4
  • The magic number field value is 0x55444234, the little-endian encoding of the ASCII string “UDB4”.
  • The number of sequences field duplicates the value from the main header.
  • The total nucleotide count is a uint64_t stored as two consecutive uint32_t values (low word first). It gives the total number of bases across all sequences, which equals the size of section 9.
  • The total header characters is a uint64_t stored as two consecutive uint32_t values (low word first). It gives the total number of bytes in section 7, including one null terminator per sequence.

Section 6 — Header Index

This section contains seqcount consecutive uint32_t values:

 offset[0]            uint32_t   byte offset of sequence 0 header
 offset[1]            uint32_t   byte offset of sequence 1 header
 ...
 offset[seqcount-1]   uint32_t   byte offset of last sequence header

Each value is the byte offset of the corresponding sequence’s header string within section 7. Offsets use 0-based indexing relative to the start of section 7.

Section 7 — Headers

This section contains header_characters bytes: the ASCII sequence headers concatenated in order, each terminated by a null byte (\0). The section is not padded. There are no ‘>’ characters; the header strings correspond to the part of fasta header lines after the leading ‘>’.

Section 8 — Sequence Lengths

This section contains seqcount consecutive uint32_t values:

 length[0]            uint32_t   length of sequence 0
 length[1]            uint32_t   length of sequence 1
 ...
 length[seqcount-1]   uint32_t   length of last sequence

Each value is the number of bases in the corresponding sequence.

Section 9 — Sequences

This section contains ntcount bytes: the ASCII nucleotide sequences concatenated in order. The section is not null-terminated and not padded. Sequences can contain uppercase and lowercase letters (soft masking is preserved). T and U are treated as equivalent by vsearch.

EXAMPLES

Build a UDB file from a fasta reference database:

vsearch \
    --makeudb_usearch db.fasta \
    --output db.udb

Use a UDB file to search query sequences against the indexed database:

vsearch \
    --usearch_global queries.fasta \
    --db db.udb \
    --id 0.97 \
    --blast6out results.tsv

Inspect the content of a UDB file:

vsearch --udbinfo db.udb

Extract the sequences stored in a UDB file back to fasta:

vsearch --udb2fasta db.udb --output db_extracted.fasta

SEE ALSO

vsearch-makeudb_usearch(1), vsearch-udb2fasta(1), vsearch-udbinfo(1), vsearch-udbstats(1), vsearch-fasta(5), vsearch-sff(5)

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

These manual pages are also published online at https://torognes.github.io/vsearch/.

COPYRIGHT

Copyright (C) 2014-2026, Torbjørn Rognes, Frédéric Mahé and Tomás Flouri

All rights reserved.

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available:

  • vsearch includes code from Google’s CityHash project by Geoff Pike and Jyrki Alakuijala, providing some excellent hash functions available under a MIT license.
  • vsearch includes code derived from Tatusov and Lipman’s DUST program that is in the public domain.
  • vsearch includes public domain code written by Alexander Peslyak for the MD5 message digest algorithm.
  • vsearch includes public domain code written by Steve Reid and others for the SHA1 message digest algorithm.
  • vsearch binaries may include code from the zlib library, copyright Jean-Loup Gailly and Mark Adler.
  • vsearch binaries may include code from the bzip2 library, copyright Julian R. Seward.
This site uses Just the Docs, a documentation theme for Jekyll.