vsearch

Versatile open-source tool for microbiome analysis

View on GitHub

NAME

udb — a binary file format containing fasta sequences and a k-mer index

DESCRIPTION

The UDB (USEARCH database) format is a binary file format used to store fasta sequences together with a pre-computed k-mer index. Building a UDB file from a fasta reference database with --makeudb_usearch allows subsequent search commands (such as --usearch_global) to load the index directly, avoiding the cost of recomputing it at each run.

UDB files are not portable across machines with different native byte orders. All multi-byte numeric values are stored in little-endian byte order (the least significant byte first), which is the native byte order of x86 and ARM processors. This contrasts with big-endian order (most significant byte first), which is used for example by the SFF format (see vsearch-sff(5)).

UDB files cannot be read from pipes; a seekable file path must be provided.

The file consists of nine sequential sections described below.

Section 1 — Main Header

The main header is exactly 200 bytes long (50 × uint32_t):

 buffer[0]   uint32_t   magic number: 0x55444246 ("UDBF")
 buffer[1]   uint32_t   reserved, set to 0
 buffer[2]   uint32_t   sequence index bits: 32
 buffer[3]   uint32_t   reserved, set to 0
 buffer[4]   uint32_t   word length (k-mer size, default 8)
 buffer[5]   uint32_t   dbstep: 1
 buffer[6]   uint32_t   dbaccelpct: 100
 buffer[7]   uint32_t   reserved, set to 0
 ...
 buffer[11]  uint32_t   slots: 0
 buffer[12]  uint32_t   reserved, set to 0
 buffer[13]  uint32_t   number of sequences
 buffer[14]  uint32_t   reserved, set to 0
 ...
 buffer[17]  uint32_t   alphabet: 0x0000746e ("nt")
 buffer[18]  uint32_t   reserved, set to 0
 ...
 buffer[49]  uint32_t   end marker: 0x55444266 ("UDBf")

All fields not listed explicitly are reserved and set to zero.

Section 2 — Word Match Counts

This section contains 4^wordlength consecutive uint32_t values (one per possible k-mer):

 kmercount[0]                uint32_t
 kmercount[1]                uint32_t
 ...
 kmercount[4^wordlength - 1] uint32_t

Each value kmercount[i] stores the number of sequences in the database that contain k-mer i at least once. With the default word length of 8, this section contains 65,536 values (256 KB).

Section 3 — UDB3 Marker

A single uint32_t sentinel value:

 marker   uint32_t   0x55444233 ("UDB3")

The value 0x55444233 is the little-endian encoding of the ASCII string “UDB3”.

Section 4 — Word Index

This section stores, for each k-mer, the sorted list of 0-based sequence numbers of all sequences containing that k-mer. The lists are concatenated in k-mer order, with no delimiters:

 seqno[0]   uint32_t   first sequence number for k-mer 0
 seqno[1]   uint32_t   second sequence number for k-mer 0
 ...        (kmercount[0] entries for k-mer 0)
 seqno[.]   uint32_t   first sequence number for k-mer 1
 ...        (kmercount[1] entries for k-mer 1)
 ...

The total number of uint32_t values in this section equals the sum of all kmercount values from section 2. A k-mer with a count of zero contributes no bytes to this section. Sequence numbers use 0-based indexing.

Section 5 — UDB4 Header

A fixed-size block of 32 bytes (8 × uint32_t):

 buffer[0]   uint32_t   magic number: 0x55444234 ("UDB4")
 buffer[1]   uint32_t   constant: 0x005e0db3
 buffer[2]   uint32_t   number of sequences
 buffer[3]   uint32_t   total nucleotide count (low 32 bits)
 buffer[4]   uint32_t   total nucleotide count (high 32 bits)
 buffer[5]   uint32_t   total header characters (low 32 bits)
 buffer[6]   uint32_t   total header characters (high 32 bits)
 buffer[7]   uint32_t   constant: 0x005e0db4

Section 6 — Header Index

This section contains seqcount consecutive uint32_t values:

 offset[0]            uint32_t   byte offset of sequence 0 header
 offset[1]            uint32_t   byte offset of sequence 1 header
 ...
 offset[seqcount-1]   uint32_t   byte offset of last sequence header

Each value is the byte offset of the corresponding sequence’s header string within section 7. Offsets use 0-based indexing relative to the start of section 7.

Section 7 — Headers

This section contains header_characters bytes: the ASCII sequence headers concatenated in order, each terminated by a null byte (\0). The section is not padded. There are no ‘>’ characters; the header strings correspond to the part of fasta header lines after the leading ‘>’.

Section 8 — Sequence Lengths

This section contains seqcount consecutive uint32_t values:

 length[0]            uint32_t   length of sequence 0
 length[1]            uint32_t   length of sequence 1
 ...
 length[seqcount-1]   uint32_t   length of last sequence

Each value is the number of bases in the corresponding sequence.

Section 9 — Sequences

This section contains ntcount bytes: the ASCII nucleotide sequences concatenated in order. The section is not null-terminated and not padded. Sequences can contain uppercase and lowercase letters (soft masking is preserved). T and U are treated as equivalent by vsearch.

EXAMPLES

Build a UDB file from a fasta reference database:

vsearch \
    --makeudb_usearch db.fasta \
    --output db.udb

Use a UDB file to search query sequences against the indexed database:

vsearch \
    --usearch_global queries.fasta \
    --db db.udb \
    --id 0.97 \
    --blast6out results.tsv

Inspect the content of a UDB file:

vsearch --udbinfo db.udb

Extract the sequences stored in a UDB file back to fasta:

vsearch --udb2fasta db.udb --output db_extracted.fasta

SEE ALSO

vsearch-makeudb_usearch(1), vsearch-udb2fasta(1), vsearch-udbinfo(1), vsearch-udbstats(1), vsearch-fasta(5), vsearch-sff(5)

CITATION

Rognes T, Flouri T, Nichols B, Quince C, Mahé F. (2016) VSEARCH: a versatile open source tool for metagenomics. PeerJ 4:e2584 doi: 10.7717/peerj.2584

REPORTING BUGS

Submit suggestions and bug-reports at https://github.com/torognes/vsearch/issues, send a pull request on https://github.com/torognes/vsearch, or compose a friendly or curmudgeont e-mail to Torbjørn Rognes (torognes@ifi.uio.no).

AVAILABILITY

Source code and binaries are available at https://github.com/torognes/vsearch.

COPYRIGHT

Copyright (C) 2014-2026, Torbjørn Rognes, Frédéric Mahé and Tomás Flouri

All rights reserved.

Contact: Torbjørn Rognes torognes@ifi.uio.no, Department of Informatics, University of Oslo, PO Box 1080 Blindern, NO-0316 Oslo, Norway

This software is dual-licensed and available under a choice of one of two licenses, either under the terms of the GNU General Public License version 3 or the BSD 2-Clause License.

GNU General Public License version 3

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

The BSD 2-Clause License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ACKNOWLEDGMENTS

We would like to thank the authors of the following projects for making their source code available: