Definitions
The following terms are used in this document:
- server : A Dqlite node.
- client : Either application code (typically wanting to issue database queries) or a Dqlite node (typically pushing replicational data).
- connection : A TCP or Unix socket connection established by the client against a server.
- word : A sequence of 8 bytes.
- protocol version : A positive number stored in a word using little endian representation.
-
message : A sequence of bytes sent either by the client to the server or by the server to the client. It consists of a header and a body. The header consists of a single word with the following layout:
- byte 0 to 3: Size of the message body, expressed in words and stored using little endian representation. For example a value of [2 1 0 0] means that the message body consists of 258 bytes.
- byte 4: Type code identifying the schema of the message body.
- byte 5: Message schema version. This version is bumped for a particular message type when the expected format of the body for that message type changes. Unless otherwise noted, the message formats described in this document are for schema version 0.
- byte 6 to 7: Currently unused.
The message body is a sequence of fields as described by the associated schema. Message types and their schemas are listed below.
Setup
As soon as a connection is established, the client must send to the server a single word containing the protocol version it wishes to use. The current protocol version is 1 (in little-endian representation).
Conversation
After the setup, communication between client and server happens by message exchange. Typically the client will send to the server a message containing a request and the server will send to the client a message containing a response. Any request may result in a failure response (type 0); otherwise, the correspondence between request and response types is as follows:
| Request | Response |
|---|---|
0 (Get current leader) |
1 (Leader information) |
1 (Client registration) |
2 (Welcome) |
3 (Open a database) |
4 (Database information) |
4 (Prepare a statement) |
5 (Prepared statement information) |
5 (Execute a prepared statement) |
6 (Statement execution result) |
6 (Execute a prepared statement yielding rows) |
7 (Batch of table rows) |
7 (Finalise a prepared statement) |
8 (Acknowledgement) |
8 (Execute a SQL text) |
6 (Statement execution result) |
9 (Execute a SQL text yielding rows) |
7 (Batch of table rows) |
10 (Interrupt the execution of a statement yielding rows) |
8 (Acknowledgement) |
12 (Add a non-voting node to the cluster) |
8 (Acknowledgement) |
13 (Assign a role to a node) |
8 (Acknowledgement) |
14 (Remove a node from the cluster) |
8 (Acknowledgement) |
15 (Dump the content of a database) |
9 (Database files) |
16 (List all nodes of the cluster) |
3 (Cluster information) |
17 (Transfer leadership to another node) |
8 (Acknowledgement) |
18 (Get metadata associated with this node) |
10 (Node metadata) |
19 (Set the weight of this node) |
8 (Acknowledgement) |
Data types
Each field in a message body has a specific data type, as described in the message schema. Available data types are:
uint64
A single word containing an unsigned integer in little endian representation.
int64
A single word containing a two-complement signed integer in little endian representation.
uint32
Four bytes containing an unsigned integer in little endian representation.
text
A sequence of one or more words containing a UTF-8 encoded zero-terminated string. All bytes past the terminating zero byte are zeroed as well.
row-tuple, params-tuple, params32-tuple
A tuple represents a sequence of typed values. Dqlite uses three distinct
formats to code tuples, depending on the kind of message. They are described in detail below, and the differences between them are summarised by this table:
| Type | Length field | Type codes |
|---|---|---|
row-tuple |
implicit | 4 bits each |
params-tuple |
1 byte | 8 bits each |
params32-tuple |
4 bytes | 8 bits each |
Every tuple ends with a sequence of values, each of which is coded according to the corresponding type code. The correspondence between type codes and values is:
| Type code | Value |
|---|---|
1 |
Integer value stored using the int64 encoding |
2 |
An IEEE 754 floating point number stored in a single word (little endian) |
3 |
A string value using the text encoding |
4 |
A binary blob: the first word of the value is the length of the blob (little endian) |
5 |
A SQL NULL value encoded as a zeroed word |
10 |
An ISO-8601 date value using the text encoding |
11 |
A Boolean value using uint64 encoding (0 for false and 1 for true) |
The row-tuple type represents a single database row in the “Batch of table rows” (response message, and is coded as a sequence of 4-bit type codes (two per byte), followed by zero padding if necessary up to a whole number of words, and then a sequence of values corresponding to the type codes. There is no length field: the client is expected to know the length of the tuple based on the database schema.
The params-tuple type represents a sequence of SQL statement parameters in the following request messages, when the message schema version is 0:
- Execute prepared statement (5)
- Execute prepared statement yielding rows (6)
- Execute SQL text (8)
- Execute SQL test yielding rows (9)
It is coded as a single byte indicating the number of values, followed by a
sequence of 8-bit (1-byte) type codes, followed by zero padding if necessary up to a whole number of words (including the “number of values” field), and then a sequence of values corresponding to the type codes. Clients that need to send more than 255 parameters for a statement must use the params32-tuple type.
The params32-tuple type represents a sequence of SQL statement parameters in the four request messages listed above (5, 6, 8, 9), when the message schema version is set to 1. It is coded in the same way as the params-tuple type, except that the initial “number of values” field is a 4-byte little-endian integer instead of a single byte.
node-info0, node-info
Information about a node in the cluster. node-info0 consists of a node ID (in uint64 encoding) followed by a node address (in text encoding). node-info consists of the same two fields, followed by a role code (in uint64 encoding). The role codes are:
| Code | Value |
|---|---|
0 |
Voter |
1 |
Standby |
2 |
Spare |
file
A single database file. It consists of the file name (in text encoding), followed by the file size (in uint64 encoding) and finally a blob with the file content.
Client messages
The client can send to the server messages with the following type codes and associated schemas:
0 - Get current leader
| Type | Value |
|---|---|
uint64 |
Unused field |
1 - Client registration
| Type | Value |
|---|---|
uint64 |
ID of the client |
3 - Open a database
| Type | Value |
|---|---|
text |
The name of the database |
uint64 |
Currently unused |
text |
Currently unused |
4 - Prepare a statement
| Type | Value |
|---|---|
uint64 |
ID of the open database to use |
text |
SQL text of the statement |
5 - Execute a prepared statement
The format for message schema version 0 is:
| Type | Value |
|---|---|
uint32 |
ID of the open database to use |
uint32 |
ID of the prepared statement to execute |
params-tuple |
A tuple of parameters to bind to the prepared statement |
For message schema version 1, params32-tuple is used instead of params-tuple for the last field.
6 - Execute a prepared statement yielding rows
The format for message schema version 0 is:
| Type | Value |
|---|---|
uint32 |
ID of the open database to use |
uint32 |
ID of the prepared statement to execute |
params-tuple |
A tuple of parameters to bind to the prepared statement |
For message schema version 1, params32-tuple is used instead of params-tuple for the last field.
7 - Finalise a prepared statement
| Type | Value |
|---|---|
uint32 |
ID of the open database to use |
uint32 |
ID of the prepared statement to finalise |
8 - Execute a SQL text
The format for message schema version 0 is:
| Type | Value |
|---|---|
uint64 |
ID of the open database to use |
text |
SQL text to execute |
params-tuple |
A tuple of parameters to bind |
For message schema version 1, params32-tuple is used instead of params-tuple for the last field.
The SQL text may consist of multiple statements, but if it does, parameters must not be provided. The server will not crash or otherwise behave dangerously when processing a request that includes both multi-statement text and parameters, but it may respond with a “Failure” message or with a “Statement execution result” message that is not meaningful.
If the SQL text consists of multiple statements, the fields of the “Statement execution result” response pertain to the last statement.
9 - Execute a SQL text yielding rows
The format for message schema version 0 is:
| Type | Value |
|---|---|
uint64 |
ID of the open database to use |
text |
SQL text to execute |
params-tuple |
A tuple of parameters to bind |
For message schema version 1, params32-tuple is used instead of params-tuple for the last field.
10 - Interrupt the execution of a statement yielding rows
| Type | Value |
|---|---|
uint64 |
ID of the open database currently executing the query |
12 - Add a non-voting node to the cluster
| Type | Value |
|---|---|
node-info0 |
ID and address of the node to add |
13 - Assign a role to a node
| Type | Value |
|---|---|
uint64 |
ID of the node to update |
uint64 |
New role |
The “new role” field is interpreted as described for node-info.
14 - Remove a node from the cluster
| Type | Value |
|---|---|
uint64 |
ID of the node to remove |
15 - Dump the content of a database
| Type | Value |
|---|---|
text |
Name of the database to dump |
16 - List all nodes of the cluster
| Type | Value |
|---|---|
uint64 |
Format |
The format field should always be set to 1.
17 - Transfer leadership to another node
| Type | Value |
|---|---|
uint64 |
ID of the new leader |
18 - Get metadata associated with this node
| Type | Value |
|---|---|
uint64 |
Format |
The format field should always be set to 0.
19 - Set the weight of this node
| Type | Value |
|---|---|
uint64 |
New weight |
Server messages
The server can send to the client messages with the following type codes and associated schemas:
0 - Failure response
| Type | Value |
|---|---|
uint64 |
Code identifying the failure type |
text |
Human-readable failure message |
1 - Leader information
| Type | Value |
|---|---|
node-info0 |
Information about the cluster leader |
2 - Welcome
| Type | Value |
|---|---|
uint64 |
Currently unused |
3 - Cluster information
| Type | Value |
|---|---|
uint64 |
Number of nodes in the cluster |
node-info |
First node |
node-info |
Second node (if any) |
| … |
4 - Database information
| Type | Value |
|---|---|
uint32 |
Database ID |
uint32 |
Unused |
5 - Prepared statement information
| Type | Value |
|---|---|
uint32 |
Database ID |
uint32 |
Statement ID |
uint64 |
Number of parameters |
6 - Statement execution result
| Type | Value |
|---|---|
uint64 |
ID of last row inserted, or 0 |
uint64 |
Number of rows affected or 0 |
7 - Batch of table rows
| Type | Value |
|---|---|
uint64 |
Number of columns |
text |
Name of first column |
text |
Name of second column (if any) |
| … | |
row-tuple |
Column values of the first row in the batch |
row-tuple |
Column values of the second row in the batch (if any) |
| … | |
uint64 |
End marker |
The end marker is the value 0xffffffffffffffff if the statement currently yielding rows has completed and there are no more rows, or otherwise 0xeeeeeeeeeeeeeeee if there are more rows and another batch will be sent.
8 - Acknowledgement
| Type | Value |
|---|---|
uint64 |
Unused |
9 - Database files
| Type | Value |
|---|---|
uint64 |
Number of files = 2 |
file |
Main database file |
file |
Write-ahead log file |
10 - Node metadata
| Type | Value |
|---|---|
uint64 |
Failure domain |
uint64 |
Weight |
These properties can be used to inform how leaders manage node roles within the cluster to maximise availability.