Developer Guide¶

All commits to main go through a PR. CI checks have to pass before merging the PR. Keep in mind that CI checks include lints. Before merge, commits are squashed, but GitHub is taking care of this, so don't worry. PR titles should follow Conventional Commits. We encourage small and orthogonal PRs. If you want to work on a bigger feature, please open an issue and discuss it with us first.

If you want to work on something but don't know what, take a look at our issues tagged with help wanted. If you're still unsure, please reach out to us via the Lakekeeper Discord. If you have questions while working on something, please use the GitHub issue or our Discord. We are happy to guide you!

Foundation & CLA¶

We hate red tape. Currently, all committers need to sign the CLA in GitHub. To ensure the future of Lakekeeper, we want to donate the project to a foundation. We are not sure yet if this is going to be Apache, Linux, a Lakekeeper foundation or something else. Currently, we prefer to spend our time on adding cool new features to Lakekeeper, but we will revisit this topic during 2026.

Initial Setup¶

To work on small and self-contained features, it is usually enough to have a Postgres database running while setting a few envs. The code block below should get you started up to running most unit tests as well as clippy.

# start postgres
docker run -d --name postgres-16 -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:17
# set envs
echo 'export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres' > .env
echo 'export ICEBERG_REST__PG_ENCRYPTION_KEY="abc"' >> .env
echo 'export ICEBERG_REST__PG_DATABASE_URL_READ="postgresql://postgres:postgres@localhost/postgres"' >> .env
echo 'export ICEBERG_REST__PG_DATABASE_URL_WRITE="postgresql://postgres:postgres@localhost/postgres"' >> .env
source .env

# Migrate db (make sure you have sqlx installed `cargo install sqlx-cli`).
# sqlx-cli auto-loads `.env` from the workspace root, so DATABASE_URL is picked up.
sqlx database create
sqlx migrate run --source crates/lakekeeper-storage-postgres/migrations

# Run tests (make sure you have cargo nextest installed, `cargo install cargo-nextest`)
cargo nextest run --all-features

# run clippy
just check-clippy
# formatting the code (make sure you have cargo-sort installed, `cargo install cargo-sort`)
# You may have to install nightly rust toolchain
just fix-format

Keep in mind that some tests are excluded by the default-filter in .config/nextest.toml. You can find a list of them in the Testing section below or by searching for modules whose name contains _integration_tests within files ending with .rs. There are a few cargo commands we run on CI. You may install just to run them conveniently. If you made any changes to SQL queries, please follow Working with SQLx before submitting your PR.

Required tools for OpenAPI regeneration¶

The just update-management-openapi and just update-generic-table-openapi recipes — plus several add-*-to-rest-openapi recipes — require Go yq (mikefarah/yq).

The Python yq (kislyuk) shipped via pip install yq is not compatible: it uses different flags (-y -i instead of -i) and its YAML emitter formats lists differently, which produces large whitespace-only diffs.

Install Go yq:

# macOS
brew install yq

# Linux (download the static binary)
curl -L "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64" \
  -o ~/.local/bin/yq && chmod +x ~/.local/bin/yq

# Verify (must say "mikefarah" in the version output)
yq --version

Code structure¶

What is where?¶

We have three crates, lakekeeper, lakekeeper-bin and iceberg-ext. The bulk of the code is in lakekeeper. The lakekeeper-bin crate contains the main entry point for the catalog. The iceberg-ext crate contains extensions to iceberg-rust.

lakekeeper

The lakekeeper crate contains the core of the catalog. It is structured into several modules:

api - contains the implementation of the REST API handlers as well as the axum router instantiation.
catalog - contains the core business logic of the REST catalog
service - contains various function blocks that make up the whole service, e.g., authn, authz and implementations of specific cloud storage backends.
tests - contains integration tests and some common test helpers, see below for more information.
implementations - contains the concrete implementation of the catalog backend, currently there's only a Postgres implementation and an alternative for Postgres as secret-store, kv2.

lakekeeper-bin

The main function branches out into multiple commands, amongst others, there's a health-check, migrations, but also serve which is likely the most relevant to you. In case you are forking us to implement your own AuthZ backend, you'll want to change the serve command to use your own implementation, just follow the call-chain.

Where to put tests?¶

We try to keep unit-tests close to the code they are testing. E.g., all tests for the database module of tables are located in crates/lakekeeper/src/implementations/postgres/tabular/table/mod.rs. While working on more complex features we noticed a lot of repetition within tests and started to put commonly used functions into crates/lakekeeper/src/tests/mod.rs. Within the tests module, there are also some higher-level tests that cannot be easily mapped to a single module or require a non-trivial setup. Depending on what you are working on, you may want to put your tests there.

I need to add an endpoint¶

You'll start at api and add the endpoint function to either management or iceberg depending on whether the endpoint belongs to official iceberg REST specification. The likely next step is to extend the respective Service trait so that there's a function to be called from the REST handler. Within the trait function, depending on your feature, you may need to store or fetch something from the storage backend. Depending on if the functionality already exists, you can do so via the respective function on the C generic and either the state: ApiContext<State<...>> struct or by first getting a transaction via C::Transaction::begin_<write|read>(state.v1_state.catalog.clone()).await?;. If you need to add a new function to the storage backend, extend the Catalog trait and implement it in the respective modules within implementations. Remember to do appropriate AuthZ checks within the function of the respective Service trait.

I need to change the audit log format¶

Audit records — every line with "event_source": "audit" — carry a MAJOR.MINOR version in their audit_format field, declared as AUDIT_FORMAT in crates/lakekeeper/src/service/events/backends/audit/mod.rs. Consumers route on it.

You never pick that number. It is derived from committed state:

AUDIT_FORMAT = the version in audit-format/released.json, raised once by the highest level among audit-format/unreleased/*.md

What a change owes is a fragment: one file recording how badly it affects a consumer, and describing it in that consumer's terms. just update-audit-fixtures reads the fragments and writes the version for you.

This is why a release ships 4.0 rather than 4.4. The baseline is raised by the highest level among the unreleased fragments, not once per fragment, so a release moves the version at most once however many changes it carries, and a major change absorbs every minor change in the same cycle. The release notes still list all of them — that is what the fragments are for. The version says how badly you are affected; the list says what happened.

Terms¶

Everything below turns on field versus value.

Field — a name in the record: action_name, entity_type, warehouse-id. Adding one changes the record's shape. (The two enums that supply them are spelled EntityField and ActionContextKey; "key" in a type name means the same thing this section calls a field.)
Value — the string a field holds: get_metadata in action_name, table in entity_type, denied in decision. Adding one leaves the shape identical, which is why consumers are told to tolerate values they do not recognise.
Shape — which fields exist, their JSON types, their nesting, and the singular/plural arity switch.
Fixture — a committed golden record under audit/fixtures/v<CURRENT MAJOR VERSION>/ (f. ex. v1), generated by the test that asserts against it. Pins shape by example: only the scenarios it covers.
Manifest — a committed wire_values.json. Pins values exhaustively: every value the types can emit, whether or not a fixture exercises it. Each crate that contributes values commits its own.
Fragment — a committed audit-format/unreleased/*.md recording one change: its level, and prose for the release notes. Written in the pull request that makes the change; assembled and deleted at release.
Level — major, minor or none: how badly the change affects a consumer. The only judgement you make.
Baseline — audit-format/released.json, the version the most recent release on this branch shipped. Maintained by the release recipe, and null until a release carries one.

Fixtures pin shape, manifests pin values, fragments record intent, and nothing else in the tree observes the emitted JSON.

Two enums, and only two, decide which fields may exist: EntityField inside an entity object and ActionContextKey inside an action object. Every other field in a record — the top-level ones, the per-decision entries, the grant context — is fixed by the emitter's structure, so changing that set means changing the emitter rather than adding a variant.

Which level. Wire enums reach the log as string values, so renaming a variant changes the payload even though no field moves. Adding one does not: docs/docs/logging.md tells consumers that value sets are open and that an unrecognised value must be treated as opaque, so a new action or entity type is new capability rather than a changed format. A new field is a changed format, because a consumer reading the record's shape sees it.

Change	Level
A field added	`minor`
A new field on an entity or an action	`minor`
A new action, entity type, or other wire value	`none`
A field removed, renamed or retyped	`major`
A wire value renamed — any of the eleven fields the manifest covers	`major`
A fixture edited, renamed, added or dropped, with no change to what the server emits	none

none is a level you may declare and not one you must: a fragment for a new action puts it in the release notes without moving the version. The last row needs no fragment at all — nothing reached a consumer.

The two field rows are the ones nothing enforces on its own: a new or moved field is only seen where a committed fixture carries it, so if you touch a path no fixture exercises, add a fixture — see "Checked by example only" below. Every value row is enforced by the manifest.

Adding a value, or a field¶

A value has to live in an enum the manifest can read. Adding a variant to an existing one costs nothing beyond regenerating: the string derives through strum, and the manifest test fails until you do. A whole new value enum is the case to watch — derive IntoStaticStr, VariantNames and EnumCount, pick the case style with #[strum(serialize_all = "...")], then add the type to the list that builds the manifest (action_name_enums in the audit test module, relation_enums in authz-openfga). Until it is in that list nothing records its values and a later rename is silent.

Actions are the case you will almost always be in — 395 of the 462 values across both manifests are action_name — so the rule is worth stating in their terms: a variant on an existing Catalog*Action, ManagementAction, AuthnAction or an authorizer's relation enum is free; a new action enum must be registered; and never pass a string literal to action_name, because a literal reaches no manifest.

A field is a variant of EntityField or ActionContextKey, and nothing else. The exhaustive as_str match stops the build until you pick the wire name, and the documentation test stops it until docs/docs/logging.md describes it. But no manifest records fields, and a fixture only pins the fields it happens to emit — so add or extend a fixture that carries the new one, or nothing pins its shape.

Steps¶

1. Make the change. Adding a field, an action or a variant is a compile error by design: the build stops where the wire name and its documentation have to be chosen.

EntityField::as_str (events/context.rs) — a field inside an entity.
EntityType::as_str (events/context.rs) — an entity_type value.
ActionContextKey::as_str (events/context.rs) — a field inside an action's context.
impl CatalogAction for Catalog*Action (service/authz/mod.rs) — a catalog action. Decide what audit context it carries; if none, add it to the explicit no-context list in that match, which exists so that this is a decision rather than a default.
determining_factor_tag / policy_effect_tag (audit test module) and failure_reason_tag (audit::contract) — a variant of an enum that reaches the wire through #[derive(Valuable)]. These types have no hand-written visit, so nothing else stops the derive emitting a new variant.
ManagementAction / AuthnAction (events/context.rs) — an action emitted by a handler that builds its ActionDescriptor directly rather than through a Catalog*Action. Add a variant; never a string literal, because only a name the type system can enumerate reaches the manifest.
AuditOperation / AuditOutcome (events/backends/audit/mod.rs) — an operational record this repository emits, or a new outcome for one. action_name has the same exposure and the same rule; FallbackAction is the enum that keeps the no-action row's value manifested. Add a variant and emit Variant::as_str(); never a string literal. audit_operation! takes any expression so that a crate outside this repository can name its own, which means the compiler does not stop a literal here either — and a literal reaches no manifest, so the value is never checked for a rename. A test enforces this in place of the compiler; see below. If the value comes from a helper that also feeds a metric label, have the helper return the enum and call as_str() at each site, so the two cannot drift apart.

Most of these fail a plain cargo build, because they are production code and the match is exhaustive. Two do not, and both are worth knowing:

The tag-helper bullet fails cargo test and just check instead: determining_factor_tag and policy_effect_tag live in the audit test module, and failure_reason_tag lives in audit::contract, gated #[cfg(any(test, feature = "test-utils"))]. CI runs both, so a local cargo build alone will not tell you about that one.
The AuditOperation / AuditOutcome bullet is the one place in the list where the type system is not the backstop. A literal compiles: the emission site is a tracing field accepting any expression, so no match stops you and no manifest records what you wrote. A lexical test stands in for the missing type check — no_production_code_names_a_wire_value_with_a_literal scans this crate's sources and names the offending file and line. It also covers action_name, which has the same exposure. It fails cargo test (just unit-test locally, and CI's unit-test job) — not cargo build, and not just check, which runs clippy, fmt and cargo-sort but no test binary.

Never add a _ => arm to those matches. The missing wildcard is the mechanism: with one, a new entity field silently takes another field's wire name and a new action silently emits no context, and no fixture notices, because a fixture can only exercise a variant that already exists. They carry #[deny(clippy::wildcard_enum_match_arm)], so a wildcard fails just check and CI, though not a bare cargo build. A wildcard alongside the full list is caught by rustc's unreachable_patterns; the dangerous edit is replacing arms with one. The exhaustive lists make some of these functions long — one carries #[allow(clippy::too_many_lines)] — so do not shorten them by collapsing the list. Not every action_descriptor impl carries the deny yet, and CatalogAction is public, so authorizer crates have impls this repository cannot see; add the deny when you next touch an unprotected one.

If you are adding actions from another crate, that crate owns its vocabulary and commits its own wire_values.json at its crate root, verified with audit::contract::assert_wire_values_manifest. crates/authz-openfga/src/relations.rs is the worked example, and the same pattern works for an authorizer maintained outside this repository: Lakekeeper cannot enumerate types it does not know, so the mechanism is shared and the vocabulary is not. The checker finds every committed manifest and compares them all.

2. Write a fragment. Copy audit-format/TEMPLATE.md to audit-format/unreleased/<descriptive-name>.md, set level from the table above, and describe the change in the terms an operator parsing the log thinks in — which field moved, and what a parser has to do about it. The body is copied verbatim into the release notes, so write it for them rather than for a reviewer. One file per pull request; a name nobody else will pick keeps concurrent changes from colliding.

3. Regenerate: just update-audit-fixtures. This computes AUDIT_FORMAT from the baseline and your fragment, renames the fixture directory if the major moved, and regenerates the fixtures and both manifests. Read the diff. It is exactly what a consumer's pipeline will see, so anything in it you did not intend is the bug.

4. Update docs/docs/logging.md — the field tables, and on a major bump the example records, which are the only documentation carrying a version value.

5. Check it: just check-audit-format. CI runs it on every pull request.

Keep exactly one fixture directory. A fixture records what the current code emits, so a retired format can neither be reproduced nor kept passing; it lives in the release notes and in git history instead.

Two things follow from the version being derived rather than bumped. Withdrawing a fragment lowers the required version again — if the change it described is reverted before release, delete the fragment and rerun the recipe, and a major that never shipped disappears along with its fixture directory rename. And a pull request targeting a rel-* branch may not change the format at all: a patch release shipping a different audit_format than main would give one number two meanings, so CI rejects a fragment or a moved constant there. Rework the change or hold it for the next minor.

What the checks cover¶

Change	What fails
A variant on an existing value enum	the manifest test
A new value enum	nothing, until it is registered — then the manifest test
An `operation` or `outcome` named with a string literal	the literal test (lexical, not the compiler)
A variant on `EntityField` or `ActionContextKey`	the `as_str` build, then the documentation test
A field added, moved or retyped where a fixture covers it	the fixture comparison
A field on a path no fixture covers	nothing — add a fixture
A rule that holds for every record	the contract rules
A change with no fragment, or an understated one	`just check-audit-format`
`AUDIT_FORMAT` not equal to what the fragments imply	`just check-audit-format`

The fragments are the claim and the fixtures are the evidence. The fixture tests compare emitted records against committed JSON and classify any difference, but only until you regenerate: afterwards they pass whether you wrote a fragment or not. check-audit-format closes that gap by comparing the committed fixtures and the wire-value manifests either side of the merge base and requiring a fragment that does not understate what it finds. It then checks AUDIT_FORMAT for equality with the value the baseline and the fragments imply — not for having moved, which is what makes a withdrawn fragment correct the version instead of leaving it over-claimed. Its decision table is self-tested with --self-test.

Comparison is on field paths and JSON types rather than values, and containers record their own type, so {}, [] and an absent field stay distinguishable. Two consequences. A changed fixture value is reported but not classified — nothing can tell a renamed wire value from a more realistic test input, so the checker says so and leaves the decision to you. And fixtures present in only one revision are not compared, so renaming or removing one costs the checker its verdict, which it reports rather than reading as "nothing changed". Neither of those two demands a fragment, for the same reason: both fire on changes that did nothing to the format.

Checked exhaustively:

Entity fields, entity_type values, action context fields — from the type system. The enums are closed, so a field cannot exist without a match arm and a documented row.
Variants of the derived audit enums — via the tag functions above.
Wire values, exhaustively, for removals. Every value the log can carry in action_name, entity_type, actor_type, decision, operation, outcome, privilege_source, resource_type, failure_reason, determined_by, effect and the update-kinds, root_level and privilege_scope action contexts comes from an enum, and the whole set is committed as a manifest — audit/wire_values.json for this crate. A test regenerates it from the types and fails when the file drifts; the checker diffs the committed file across the merge base and requires a major fragment for any value that disappeared. It is keyed by field and by owning type, because most verbs are shared: six action enums emit get_metadata, so a comparison over the flattened set would not notice one of them renaming it. Additions are not a format change. Two limits remain. A brand-new enum is invisible until it is added to the list that builds the manifest, since Rust cannot enumerate the types implementing a trait — grep -rn "impl CatalogAction for" crates/ and grep -rn '.action_name(' crates/*/src are the cross-checks. And the operational operation / outcome space stays open by design: audit_operation! is exported and takes any expression, so a crate outside this repository names its own and owns its own manifest. Every wire_values.json committed to this repository is found and compared, wherever in the tree it lives, so an in-repo crate is covered without this crate knowing anything about its types. A crate outside the repository is not — CI here cannot see its files. It gets the same guarantee by running the same two things in its own CI: the Rust test that regenerates its manifest, and this checker against its own merge base.

Checked by example only: record shape — nesting, value types, which optional fields are omitted versus null, and the singular/plural arity switch. The fixtures pin the scenarios they cover and nothing else, so if you change a code path no fixture exercises, add a fixture: write a test that builds the event, run just update-audit-fixtures, and register the name in FIXTURE_NAMES. That also widens the documentation test that walks the fixtures.

A fixture is compared by value, so everything in the record has to be deterministic. Where the emitter generates an id, pin it through a test-only seam rather than normalising the record afterwards — RequestMetadataTestBuilder::request_id and AdmissionRejection::with_error_id are the two that exist, and a new one belongs next to them. Normalising instead would mean the fixture no longer pins what the code actually emits, which is the only thing it is for.

Outside audit_format entirely: the fields the subscriber adds — timestamp, level, message, target, span, spans, filename, line_number — belong to tracing-subscriber and can move on a dependency upgrade. They are stripped before comparison, and docs/docs/logging.md states that as a contract.

Extend the corpus test while you are here. crates/lakekeeper-integration-tests/tests/audit_corpus.rs drives real requests through the service layer and checks rules that hold for any record: every field a variant of the closed enums, a definitive denial never carrying allowed: true, and so on. It catches what a fixture cannot, because a fixture is generated by the test that asserts against it — a wrongly built event yields a fixture that agrees with it. The rules are already general; only the set of requests is narrow. Drive one more call through CatalogServer or ApiServer, raise EXPECTED_RECORDS by however many records it emits, and the existing rules apply to whatever it produced. That count is exact because it is the only thing that notices a request quietly ceasing to emit. Run it with just test-audit-corpus, which needs the local Postgres from the Initial setup. Keep every test in that file on a current-thread runtime: capture is thread-local, so flavor = "multi_thread" would capture nothing.

Debugging complex issues and prototyping using our examples¶

To debug more complex issues, work on prototypes or simply an initial manual test, you can use one of the examples. Unless you are working on AuthN or AuthZ, you'll most likely want to use the minimal example. All examples come with a docker-compose-build.yaml which will build the catalog image from source. The invocation looks like this: docker compose -f docker-compose.yaml -f docker-compose-build.yaml up -d --build. Aside from building the catalog, the docker-compose-build.yaml overlay also exposes the docker services to your host, so you can also use it as a development environment by e.g. pointing your env vars to the docker container to test against its minio instance. If you made changes to SQL queries, you'll have to run just sqlx-prepare before rebuilding the catalog image. This will update the sqlx queries in .sqlx to enable static checking of the queries without a migrated database.

After spinning the example up, you may head to localhost:8888 and use one of the notebooks.

Working with SQLx¶

This crate uses sqlx. For development and compilation a Postgres Database is required. This is part of the Initial setup. If your database credentials used differ, please modify the .env accordingly and run source .env again.

Run:

# Migrate db. Make sure you have sqlx-cli install with `cargo install sqlx-cli`
# Run this locally if you change the db schema via `crates/lakekeeper-storage-postgres/migrations`,
# e.g. after adding a table or dropping a column.
sqlx database create
sqlx migrate run --source crates/lakekeeper-storage-postgres/migrations

# If you changed any of the SQL statements embedded in Rust code, run this before pushing to GitHub.
just sqlx-prepare

This will update the sqlx queries in .sqlx to enable static checking of the queries without a migrated database. Remember to git add .sqlx before committing. If you forget, your PR will fail to build on GitHub. Be careful, if the command failed, .sqlx will be empty. But do not worry, it wouldn't build on GitHub so there's no way of really breaking things.

⚠️ Schema Qualification Warning¶

IMPORTANT: When adding new migrations, do NOT schema qualify references to any database objects. Schema qualification will break deployments that place the application in a schema different than the public one.

❌ Incorrect - Do NOT do this:

-- This will break deployments in non-public schemas
CREATE TABLE public.my_new_table (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255)
);

INSERT INTO public.my_new_table (name) VALUES ('example');

ALTER TABLE public.existing_table ADD COLUMN new_column INTEGER;

✅ Correct - Do this instead:

-- This will work in any schema
CREATE TABLE my_new_table (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255)
);

INSERT INTO my_new_table (name) VALUES ('example');

ALTER TABLE existing_table ADD COLUMN new_column INTEGER;

The migration system will automatically apply the migration in the correct schema context, so explicit schema qualification is unnecessary and will cause issues in deployments where Lakekeeper is deployed to a custom schema.

Operators pick that schema with LAKEKEEPER__PG_SCHEMA. Do not export it in a shell you build or test from: cargo sqlx prepare ignores it and uses DATABASE_URL, and the postgres crate reads LAKEKEEPER_TEST__ in its own tests but LAKEKEEPER__ when compiled as a dependency, so it would apply unevenly.

Inspecting the db¶

The db schema is the result of all migrations applied in order. To inspect it you can:

# Assumes you set up the db as described above

# Get a shell in the db's container
docker exec -it postgres-16 /bin/bash

# Then you can connect to the db
psql "postgresql://postgres:postgres@localhost:5432/postgres"
# And inspect it, for instance by describing views or tables
\d+ active_tabulars

# Or you can dump the entire schema
pg_dump --schema-only "postgresql://postgres:postgres@localhost:5432/postgres" > /home/lakekeeper_schema.sql
# Copy it out of the container and then inspect it or pass it as context to LLMs
docker cp postgres-16:/home/lakekeeper_schema.sql .

Extension tables (`ext_*` prefix)¶

Lakekeeper reserves the ext_* table-name prefix for downstream extensions that need to store their own state in the catalog database. The convention is an operational contract between upstream and any extension:

Rule	What it means
Reserved prefix	Upstream core migrations never create tables matching `ext_*`. Extensions own that namespace. The integration test `test_core_does_not_create_ext_objects` enforces this.
FK direction	Extension tables FK into upstream tables. Upstream never FKs into `ext_*`.
CASCADE required	Every FK from an `ext_*` table to an upstream table should be `ON DELETE CASCADE` or `ON DELETE SET NULL`. Enforce in the extension crate's own CI — upstream cannot inspect downstream migration sets.
Scope of allowed objects	`ext_*` may name tables and objects owned by those tables (indexes, sequences). Triggers, functions, indexes, or views attached to upstream-owned objects are not permitted under the prefix — they would survive extension removal and could brick OSS.
Tracker tables	Each registered extension migration source uses its own SQLx tracker, named `ext_<name>_sqlx_migrations`. Core's `_sqlx_migrations` is untouched.

Registering extension migrations¶

Extensions call upstream's migrate(pool, extensions) with their own migration source. Every registered source runs inside the same outer transaction as core migrations — either every migration commits or the entire upgrade rolls back. Partial state is impossible.

use lakekeeper_storage_postgres::migrations::{ExtensionMigrations, migrate};

// `name` must be 1–40 chars: first [a-z_], remaining [a-z0-9_]; rejected at
// the start of `migrate()` otherwise. Derives `ext_my_extension_sqlx_migrations`.
let extensions = vec![ExtensionMigrations::builder()
    .name("my_extension")
    .migrator(sqlx::migrate!("./migrations")) // embedded at compile time
    .build()];
let server_id = migrate(&pool, extensions).await?;

Optional fields not shown: .data_hooks(map) for Rust-side hooks tied to specific migration versions, and .sha_patches(set) for in-place edits to already-shipped migrations.

The data_hooks field on ExtensionMigrations is a HashMap<i64, Box<dyn MigrationHook>> keyed by the migration's version id. Each entry's MigrationHook runs immediately after the matching extension migration is applied, inside the same transaction — use it for Rust-side data backfills tied to a specific SQL migration. Pass HashMap::new() when no hooks are needed (the common case in the snippet above).

Callers that don't register extensions use the back-compat shim migrate_core_only(pool). Core upstream tooling and tests already do.

Recovery: removing an extension's state¶

Dropping the extension binary and removing its tables restores the database to a working OSS-only state. The SQL below scans every relation kind that the ext_* prefix may name and drops it:

-- Run inside the catalog database. Drops every ext_* table and tracker
-- (CASCADE handles dependent indexes, sequences, and constraints).
DO $$
DECLARE r record;
BEGIN
    -- Tables (covers extension state + per-source `_sqlx_migrations` trackers).
    FOR r IN SELECT c.relname
             FROM pg_class c
             JOIN pg_namespace n ON n.oid = c.relnamespace
             WHERE n.nspname = current_schema()
               AND c.relkind IN ('r', 'p')
               AND c.relname LIKE 'ext\_%' ESCAPE '\'
    LOOP
        EXECUTE format('DROP TABLE %I CASCADE', r.relname);
    END LOOP;

    -- Defensive sweeps for object kinds the convention forbids extensions
    -- from creating on upstream-owned tables, but that may exist if a
    -- non-conforming extension was deployed.
    FOR r IN SELECT t.tgname, c.relname AS tbl
             FROM pg_trigger t
             JOIN pg_class c ON c.oid = t.tgrelid
             WHERE NOT t.tgisinternal
               AND t.tgname LIKE 'ext\_%' ESCAPE '\'
    LOOP
        EXECUTE format('DROP TRIGGER %I ON %I', r.tgname, r.tbl);
    END LOOP;

    FOR r IN SELECT typname FROM pg_type t
             JOIN pg_namespace n ON n.oid = t.typnamespace
             WHERE n.nspname = current_schema()
               AND typname LIKE 'ext\_%' ESCAPE '\'
    LOOP
        EXECUTE format('DROP TYPE %I CASCADE', r.typname);
    END LOOP;

    FOR r IN SELECT p.proname FROM pg_proc p
             JOIN pg_namespace n ON n.oid = p.pronamespace
             WHERE n.nspname = current_schema()
               AND p.proname LIKE 'ext\_%' ESCAPE '\'
    LOOP
        EXECUTE format('DROP FUNCTION %I CASCADE', r.proname);
    END LOOP;
END $$;

After running this, the OSS binary boots cleanly against the remaining catalog state.

KV2 / Vault¶

This catalog supports KV2 as a backend for secrets. Tests for KV2 are disabled by default. To enable them, you need to run the following commands:

docker run -d -p 8200:8200 --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' hashicorp/vault

# append some more env vars to the .env file, it should already have PG related entries defined above.

# the values below configure KV2
echo 'export ICEBERG_REST__KV2__URL="http://localhost:8200"' >> .env
echo 'export ICEBERG_REST__KV2__USER="test"' >> .env
echo 'export ICEBERG_REST__KV2__PASSWORD="test"' >> .env
echo 'export ICEBERG_REST__KV2__SECRET_MOUNT="secret"' >> .env

source .env
# setup vault
./tests/vault-setup.sh http://localhost:8200

# Select kv2 tests
cargo nextest run --all-features --all-targets \
    --ignore-default-filter -E "test(::kv2_integration_tests::)"

Test cloud storage profiles¶

Currently, we're not aware of a good way of testing cloud storage integration against local deployments. That means, to test against AWS S3, GCS and ADLS Gen2, you need to set the following environment variables. For more information, take a look at the Storage Guide. A sample .env could look like this:

export LAKEKEEPER_TEST__AZURE_TENANT_ID=<your tenant id>
export LAKEKEEPER_TEST__AZURE_STORAGE_FILESYSTEM=<your azure adls filesystem name>
export LAKEKEEPER_TEST__AZURE_STORAGE_ACCOUNT_NAME=<your azure storage account name>
# Auth Method 1: Client Credentials
export LAKEKEEPER_TEST__AZURE_CLIENT_ID=<your entra id app registration client id>
export LAKEKEEPER_TEST__AZURE_CLIENT_SECRET=<your entra id app registration client secret>
# Auth Method 2: Shared Key
export LAKEKEEPER_TEST__AZURE_STORAGE_SHARED_KEY=<shared key>

export LAKEKEEPER_TEST__AWS_S3_BUCKET=<your aws s3 bucket>
export LAKEKEEPER_TEST__AWS_S3_REGION=<your aws s3 region>
export LAKEKEEPER_TEST__AWS_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export LAKEKEEPER_TEST__AWS_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export LAKEKEEPER_TEST__AWS_S3_STS_ROLE_ARN=arn:aws:iam::123456789012:role/role-name

# the values below should work with the default minio in our docker-compose
export LAKEKEEPER_TEST__S3_BUCKET=tests
export LAKEKEEPER_TEST__S3_REGION=local
export LAKEKEEPER_TEST__S3_ACCESS_KEY=minio-root-user
export LAKEKEEPER_TEST__S3_SECRET_KEY=minio-root-password
export LAKEKEEPER_TEST__S3_ENDPOINT=http://localhost:9000

export LAKEKEEPER_TEST__GCS_CREDENTIAL='{"type": "service_account","project_id": "..", ...}'
export LAKEKEEPER_TEST__GCS_BUCKET=name-of-gcs-bucket-without-hns
export LAKEKEEPER_TEST__GCS_HNS_BUCKET=name-of-gcs-bucket-with-hns

You may then run tests by ignoring the nextest's default filter and selecting the desired tests:

source .example.env-from-above
cargo nextest run --all-features --ignore-default-filter -E "test(::aws_integration_tests::)"
# see .config/nextest.toml for all filters

To check a new S3-compatible store, point the LAKEKEEPER_TEST__S3_* variables at it and run the s3_compat profile, which selects exactly the tests that use those variables and nothing else:

cargo nextest run --profile s3_compat --all-features --all-targets --workspace

This is what the SeaweedFS workflow runs.

Running integration test¶

Our integration tests are written in Python and use pytest. They are located in the tests folder. The integration tests spin up Lakekeeper and all the dependencies via docker compose. Please check the Integration Test Docs for more information.

Running Authorization unit tests¶

Some authorization unit tests need to be run against an OpenFGA server. They are excluded by our nextest default-filter. The workflow for executing them is:

# Start an OpenFGA server in a docker container
docker rm --force openfga-client && docker run -d --name openfga-client -p 36080:8080 -p 36081:8081 -p 36300:3000 openfga/openfga:v1.14 run

# Set Lakekeeper's OpenFGA endpoint
export LAKEKEEPER_TEST__OPENFGA__ENDPOINT="http://localhost:36081"

# Use a filterset to select the tests
cargo nextest run --all-features --ignore-default-filter -E "test(::openfga_integration_tests::)"

Extending Authz¶

When adding a new endpoint, you may need to extend the authorization model. Please check the Authorization Docs for more information. For OpenFGA, perform the following steps:

Add the new action to the relevant enum in crate::service::authz, e.g. CatalogViewAction::CanUndrop. Actions that must carry request context for policy-based authorizers are parameterized variants — see CatalogProjectAction::CreateWarehouse.
In the lakekeeper-authz-openfga crate (crates/authz-openfga/src/relations.rs), add or reuse a relation on the resource enum (e.g. RoleRelation::CanUndrop) and map the action to it in the ReducedRelation impl (e.g. CatalogViewAction::CanUndrop => ViewRelation::CanUndrop).
Bump the model version by renaming the latest folder — e.g. git mv authz/openfga/v4.7 authz/openfga/v4.8. Do not create a new folder alongside the old one. For a backward-compatible change (adding a type, relation, or action; no rewrite of existing tuples) the rename is all you need: existing stores re-migrate to the new model id on startup and their tuples keep authorizing the same actions. This holds whether or not the previous version was already released — a released store simply re-migrates to the new id. The only exception is a change that rewrites/migrates existing tuples: that one gets a brand-new folder while the old folder is kept for the migration chain (see v4.0, which introduced lakekeeper_table / lakekeeper_view and migrated tuples). Rule of thumb: backward-compatible ⇒ rename the folder; tuple migration ⇒ add a new folder.
Edit the relevant component(s) under authz/openfga/<version>/components/*.fga (e.g. add define can_undrop: modify_effective to lakekeeper_view.fga). An action reads the _effective twin of a privilege; the bare relation holds only what was granted on the object itself, then regenerate and validate:
```
just update-openfga   # fga model transform <latest>/fga.mod > <latest>/schema.json
just test-openfga     # runs the <latest>/store.fga.yaml assertions
```
(Requires the fga CLI — download from the OpenFGA repo.)
In crates/authz-openfga/src/migration.rs bump ACTIVE_MODEL_VERSION to the new version. For backward-compatible changes, repoint the current add_model_*_current call (schema-path include_str! + version). For tuple-migrating changes, add another add_model call carrying the migration fn.
Record the change under the new version heading in authz/openfga/README.md.

Building the docs locally¶

cd site
just serve