Skip to main content

Fastly Object Storage Import

Prerequisites​

For Fastly Object Storage, the S3 Compatibility API allows you to use Goose's S3 support to read and write from Fastly buckets.

This requires the httpfs extension, which can be installed using the INSTALL SQL command. This only needs to be run once.

Credentials and Configuration​

You will need to generate an S3 auth token and create an S3 secret in Goose:

CREATE SECRET my_secret (
    TYPE s3,
    KEY_ID '⟨AKIAIOSFODNN7EXAMPLE⟩',
    SECRET '⟨wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY⟩',
	URL_STYLE 'path',
    REGION '⟨us-east⟩',
    ENDPOINT '⟨us-east⟩.object.fastlystorage.app' -- see note below
);
  • The ENDPOINT needs to point to the Fastly endpoint for the region you want to use (e.g., eu-central.object.fastlystorage.app).
  • REGION must use the same region mentioned in ENDPOINT.
  • URL_STYLE needs to use path.

Querying​

After setting up the Fastly Object Storage credentials, you can query the data there using Goose's built-in methods, such as read_csv or read_parquet:

SELECT * FROM 's3://⟨fastly-bucket-name⟩/(file).csv';
SELECT * FROM read_parquet('s3://⟨fastly-bucket-name⟩/⟨file⟩.parquet');