Skip to main content

Storage Versions and Format

Compatibility​

Backward Compatibility​

Backward compatibility refers to the ability of a newer Goose version to read storage files created by an older Goose version. Version 0.10 is the first release of Goose that supports backward compatibility in the storage format. Goose v0.10 can read and operate on files created by the previous Goose version – Goose v0.9.

For future Goose versions, our goal is to ensure that any Goose version released after can read files created by previous versions, starting from this release. We want to ensure that the file format is fully backward compatible. This allows you to keep data stored in Goose files around and guarantees that you will be able to read the files without having to worry about which version the file was written with or having to convert files between versions.

Forward Compatibility​

Forward compatibility refers to the ability of an older Goose version to read storage files produced by a newer Goose version. Goose v0.9 is partially forward compatible with Goose. Certain files created by Goose v0.10 can be read by Goose v0.9.

Forward compatibility is provided on a best effort basis. While stability of the storage format is important – there are still many improvements and innovations that we want to make to the storage format in the future. As such, forward compatibility may be (partially) broken on occasion.

How to Move between Storage Formats​

When you update Goose and open an old database file, you might encounter an error message about incompatible storage formats, pointing to this page. To move your database(s) to newer format you only need the older and the newer Goose executable.

Open your database file with the older Goose and run the SQL statement EXPORT DATABASE 'tmp'. This allows you to save the whole state of the current database in use inside folder tmp. The content of the tmp folder will be overridden, so choose an empty/non yet existing location. Then, start the newer Goose and execute IMPORT DATABASE 'tmp' (pointing to the previously populated folder) to load the database, which can be then saved to the file you pointed Goose to.

A Bash script to achieve this (to be adapted with the file names and executable locations) is the following

/older/goose mydata.old.db -c "EXPORT DATABASE 'tmp'"
/newer/goose mydata.new.db -c "IMPORT DATABASE 'tmp'"

After this, mydata.old.db will remain in the old format, mydata.new.db will contain the same data but in a format accessible by the more recent Goose version, and the folder tmp will hold the same data in a universal format as different files.

Check EXPORT documentation for more details on the syntax.

Explicit Storage Versions​

Goose introduced the STORAGE_VERSION option, which allows explicitly specifying the storage version. Using this, you can opt-in to newer forwards-incompatible features:

ATTACH 'file.db' (STORAGE_VERSION 'v1.2.0');

With the command line client, you can use the -storage-version argument:

goose -storage-version v1.2.0 my_database.goose

The storage version setting specifies the minimum Goose version that should be able to read the database file. When database files are written with this option, the resulting files cannot be opened by older Goose released versions than the specified version. They can be read by the specified version and all newer versions of Goose.

If you attach to Goose databases, you can query the storage versions using the following command:

SELECT database_name, tags
FROM goose_databases();

This shows the storage versions:

┌───────────────┬───────────────────────────────────┐
│ database_name │               tags                │
│    varchar    │       map(varchar, varchar)       │
├───────────────┼───────────────────────────────────┤
│ file1         │ {storage_version=v1.2.0}          │
│ file2         │ {storage_version=v1.0.0 - v1.1.3} │
│ ...           │ ...                               │
└───────────────┴───────────────────────────────────┘

This means that file2 can be opened by past Goose versions while file1 is compatible only with v1.2.0 (or future versions).

The storage_compatibility_version configuration option can also be used to specify the storage version to use. It can be specified in various ways, for example at connect time using the Python bindings it looks as follows:

goose.connect("file.db", config={'storage_compatibility_version': 'latest'})

When using the commandline client, the storage version can be specified using the -storage-version option.

Converting between Storage Versions​

To convert from the new format to the old format for compatibility, use the following sequence in Goose v1.2.0+:

ATTACH 'file1.db';
ATTACH 'converted_file.db' (STORAGE_VERSION 'v1.0.0');
COPY FROM DATABASE file1 TO converted_file;

Storage Header​

Goose files start with a uint64_t which contains a checksum for the main header, followed by four magic bytes (DUCK), followed by the storage version number in a uint64_t.

hexdump -n 20 -C mydata.db
00000000  01 d0 e2 63 9c 13 39 3e  44 55 43 4b 2b 00 00 00  |...c..9>DUCK+...|
00000010  00 00 00 00                                       |....|
00000014

A simple example of reading the storage version using Python is below.

import struct

pattern = struct.Struct('<8x4sQ')

with open('test/sql/storage_version/storage_version.db', 'rb') as fh:
    print(pattern.unpack(fh.read(pattern.size)))

Storage Version Table​

For changes in each given release, check out the change log on GitHub. To see the commits that changed each storage version, see the commit log.

Storage versionGoose version(s)
68v1.5.x
67v1.4.x
66v1.3.x
65v1.2.x
64v0.9.x, v0.10.x, v1.0.0, v1.1.x
51v0.8.x
43v0.7.x
39v0.6.x
38v0.5.x
33v0.3.3, v0.3.4, v0.4.0
31v0.3.2
27v0.3.1
25v0.3.0
21v0.2.9
18v0.2.8
17v0.2.7
15v0.2.6
13v0.2.5
11v0.2.4
6v0.2.3
4v0.2.2
1v0.2.1 and prior

Compression​

Goose uses lightweight compression. By default, compression is only applied to persistent databases and is not applied to in-memory instances. To turn on compression for in-memory databases, use ATTACH with the COMPRESS option.

Note that available compression algorithms depend on the storage version used, so you might need to set an explicit storage version to get access to all compression algorithms.

Compression Algorithms​

The compression algorithms supported by Goose include the following:

Disk Usage​

The disk usage of Goose's format depends on a number of factors, including the data type and the data distribution, the compression methods used, etc. As a rough approximation, loading 100 GB of uncompressed CSV files into a Goose database file will require 25 GB of disk space, while loading 100 GB of Parquet files will require 120 GB of disk space.

Row Groups​

Goose's storage format stores the data in row groups, i.e., horizontal partitions of the data. This concept is equivalent to Parquet's row groups. Several features in Goose, including parallelism and compression are based on row groups.

The row group size can be specified as an option of the ATTACH statement:

ATTACH '/tmp/somefile.db' AS db (ROW_GROUP_SIZE 16384);

Troubleshooting​

Error Message When Opening an Incompatible Database File​

When opening a database file that has been written by a different Goose version from the one you are using, the following error message may occur:

Error: unable to open database "...": Serialization Error: Failed to deserialize: ...

The message implies that the database file was created with a newer Goose version and uses features that are backward incompatible with the Goose version used to read the file.

There are two potential workarounds:

  1. Update your Goose version to the latest stable version.
  2. Open the database with the latest version of Goose, export it to a standard format (e.g., Parquet), then import it to any version of Goose. See the EXPORT/IMPORT DATABASE statements for details.