# Lakekeeper — full documentation > Lakekeeper is an open source Apache Iceberg REST Catalog written in Rust, with fine-grained access control, multi-engine support and S3, GCS & Azure storage. Source: https://docs.lakekeeper.io/ — generated at build time from the current release. --- # Code of Conduct Source: https://docs.lakekeeper.io/about/code-of-conduct/ --8<-- "../CODE_OF_CONDUCT.md" --- # Lakekeeper+ Source: https://docs.lakekeeper.io/about/enterprise-release-notes/ # Lakekeeper+ Release Notes ## Unreleased ### Features - **Cedar: grants decide access.** Privileges granted through the catalog are now visible to your Cedar policies. Each resource carries its own grants as `direct_privileges` and everything granted above it as `inherited_privileges`, so a policy can simply check `resource.direct_privileges.select`. Grant-administration actions carry the privilege in question as the typed `context.privilege`, so a policy can condition on exactly which privilege is being handed out. - **Cedar: predefined policies turn grants into access.** Lakekeeper Plus now ships ready-made policies that turn recorded grants into access, so permissions can be handed out at runtime without redeploying a policy file. Every project and warehouse chooses which of these policies apply to it, through the new `predefined-policies` endpoints. The feature is on by default and can be switched off for the whole deployment with `LAKEKEEPER__CEDAR__PREDEFINED_POLICIES_ENABLED=false`. Policies added by a future release start out disabled for scopes you have already curated, so an upgrade never widens access on its own. - **Cedar: manage a project's or warehouse's own policies through the API.** Each project and warehouse can now store its own Cedar policies, managed through the new `…/policies` endpoints. Changes are applied as a single transaction that is validated up front and reports back what actually changed. You can preview a change with `dry-run`, protect against concurrent edits with `if-scope-version`, and send your complete desired state with `replace: true`, which removes every policy you did not include. A scope holds up to 1000 policies by default (`LAKEKEEPER__CEDAR__MAX_POLICIES_PER_SCOPE`). - **Cedar: break-glass.** A policy written too broadly can lock out the very people who could fix it. An operator who sends the header `x-break-glass: ` is judged by the server's own policies alone and can then repair a project's or warehouse's policies. This path opens policy administration only, never any data, and every use is recorded as its own audit event together with the given reason. The new `break-glass-status` endpoint tells you in advance whether this path is open to you. - **Cedar: a new `manage_policies` privilege.** Granting it on a project or warehouse lets someone read and write that scope's policies without giving them access to any data. A grant on a project also covers the warehouses inside it. Because whoever writes policies can permit themselves anything on the next request, this right is never part of `manage` and always has to be granted explicitly. - **Cedar: policy sources report their tier.** Every source listed by `server-policy-sources` now carries a `tier` field. This endpoint lists the server's configured policy sources only, so their tier is `server`. - **Cedar: two endpoints have new names.** `policy-sources` and `policy-list` are now called `server-policy-sources` and `server-policy-list`, which tells these server-level listings apart from the new per-scope policy endpoints. The old paths still work but are marked deprecated. ### Bug Fixes - **Okta role provider: people who sign in with an email address now resolve to their groups.** The user identifier was escaped for a query string rather than for a URL path, so the `@` in a login like `alice@example.com` reached Okta as `%40` and every such lookup came back as a bare `400`. Deployments whose OIDC subject is a raw Okta user id were unaffected, which is why service principals kept working while people did not. - **Okta role provider: a retried token request no longer replays its client assertion.** Okta accepts each assertion once, so a DPoP nonce challenge or a throttled token request could fail with `invalid_client`. Every attempt is now signed afresh. ### Breaking Changes - **Cedar: reading the server's policy sources now counts as policy administration.** The actions for listing and evaluating server policy sources moved out of `ServerActions` into the new `ServerCedarPolicyActions` and `CedarPolicyReadActions` groups. A policy that permits `ServerActions` no longer covers them, so name the new groups where you want these reads allowed. - **Admission gate: `idp_id` must name a provider the server authenticates.** The server now refuses to start when `idp_id` matches none of the configured authenticators, and tells you which ones are available. Such an id never enforced anything, so this turns a silent misconfiguration into a clear startup error. - **Admission gate: unknown configuration keys are refused.** A misspelled key under `admission_enforce` used to fall back to its default silently. The server now names the unknown key and stops. - **Admission gate: requests without an authenticated principal are rejected.** A gate that cannot see who is calling cannot enforce anything. - **OpenFGA: revoking a privilege requires `manage_grants`.** Passing on a privilege you hold still needs only `pass_grants`, but taking one back is now considered administration. This applies to the `/grants` endpoint and to the older assignment deletes. - **OpenFGA: only instance admins can manage membership of `system`-provider roles.** These roles also no longer accept other roles as members. Ordinary and `lakekeeper`-provider roles are unaffected. - **Cedar: seeing another principal's access needs its own grant.** The `IntrospectAuthorization` actions moved into the grant-reading groups. Grant `ReadGrants`, or name the action, where you want this available. - **Cedar: a request must name the project its resources are in.** Addressing a warehouse outside the project given in `x-project-id` now returns an error, and one request can no longer span two projects. Single-project deployments are unaffected. - **Cedar: a replaced schema file must declare the privilege records.** With `LAKEKEEPER__CEDAR__SCHEMA_FILE` set, the server only starts when every resource entity declares `direct_privileges` and `inherited_privileges`, and it names what is missing. - **Cedar: external entity files are validated against the schema.** A resource entity without both privilege records fails to load and the server does not start. - **Cedar: with externally managed identity, the entity file must declare every principal a request names.** A request that names a user or role missing from your file is now refused instead of answered. Cedar skips policies about entities it cannot find, which could otherwise turn a `forbid` into an allow. ### Upgrade Notes - **Run `lakekeeper-plus migrate` to completion before rolling any new pod.** This release adds database tables for Cedar policy management, and a new replica refuses to serve while they are missing. Note that `wait-for-db -m` does not check for these tables. Rolling back is safe. - **Grants that already exist begin granting access.** The predefined policies are on by default, so every grant recorded in your deployment starts deciding requests after the upgrade. Review your grants beforehand, or start with `LAKEKEEPER__CEDAR__PREDEFINED_POLICIES_ENABLED=false` and curate each scope before switching it on. - **Multi-project Cedar deployments: check that clients send `x-project-id`.** Before this release a request could be answered using another project's roles. If your clients omitted the header, treat past cross-project decisions as unreliable. - **Cedar with externally managed identity: grant to users, not roles.** A grant held by a user applies. One held by a role does not, because role membership lives in your entity file. ## v0.13.6 (2026-09-18) _Based on Lakekeeper OSS v0.13.5._ ### Bug Fixes - **Okta role provider: people who sign in with an email address now resolve to their groups.** The user identifier was escaped for a query string rather than for a URL path, so the `@` in a login like `alice@example.com` reached Okta as `%40` and every such lookup came back as a bare `400`. Deployments whose OIDC subject is a raw Okta user id were unaffected, which is why service principals kept working while people did not. - **Okta role provider: a retried token request no longer replays its client assertion.** Okta accepts each assertion once, so a DPoP nonce challenge or a throttled token request could fail with `invalid_client`. Every attempt is now signed afresh. - **Security: TLS handshake handling (RUSTSEC-2026-0285).** `rustls` is updated to 0.23.45, which rejects handshake messages spanning a key change instead of accepting them. The handshake transcript stayed authenticated, so this could not be used to alter or complete a handshake. ### Upgrade Notes - **OpenFGA deployments that have renamed a table, view or generic table across namespaces should run `lakekeeper openfga reconcile --mode add-and-delete-drift` once after upgrading.** The upstream fix below stops the leak but cannot remove a permission edge already recorded, and the default `add-missing` mode does not repair it. ### Upstream Lakekeeper changes (bump to v0.13.5) - **GovCloud, China and ISO region support.** Vended-credential policies now carry the correct ARN partition, derived automatically from the region, endpoint or role ARN, so `AssumeRole` succeeds for buckets outside the commercial partition ([lakekeeper#1928](https://github.com/lakekeeper/lakekeeper/pull/1928)). - **S3 request signing for generic tables** ([lakekeeper#1910](https://github.com/lakekeeper/lakekeeper/pull/1910)). - Fixed a permissions leak: a table, view or generic table renamed into another namespace kept inheriting grants from the namespace it left ([lakekeeper#2013](https://github.com/lakekeeper/lakekeeper/pull/2013)). - Renaming a table onto a name that is already taken now returns `409 Conflict` instead of `404 Not Found`, matching the Iceberg REST spec, and renaming onto a soft-deleted name succeeds ([lakekeeper#1955](https://github.com/lakekeeper/lakekeeper/pull/1955)). ## v0.13.5 (2026-08-26) _Based on Lakekeeper OSS v0.13.3._ ### Highlights - **Improved memory behaviour on long-running instances.** Conditions that could, under some circumstances, prevent freed memory from being returned to the OS are addressed. ### Features - **New memory metrics.** `lakekeeper_jemalloc_*` separates live heap from memory the allocator is holding back; `lakekeeper_http_connections` reports open connections. - **The table maintenance cache reports what it holds:** `lakekeeper_cache_weighted_bytes{cache_type="table_metadata"}` and `lakekeeper_cache_instances`, with hits and misses in the shared `lakekeeper_cache_*` series. ### Bug Fixes - **Transparent huge pages could prevent freed memory from being returned to the OS.** On nodes with `THP=always`, the default on common EKS AMIs, resident memory could grow for the life of the process. - **The manifest cache under-counted its entries**, so under some circumstances its 128 MiB budget did not bind during table maintenance. - **The UI asset cache keyed on an unvalidated header**, so variants of `x-forwarded-prefix` could each add an entry to a cache with no expiry. - **Allocator metrics now report on every serving path**, including `LAKEKEEPER__DEBUG__AUTO_SERVE`. Memory behaviour itself was unaffected. ### Upgrade Notes - **Idle HTTP connections now close after 75 seconds** and carry TCP keepalive probes. Standard Iceberg and S3 clients retry; previously a connection whose peer had vanished could be held for the life of the process. - **Request headers above 64 KiB are rejected with 431.** Request bodies are unaffected. - **Shutdown drains for at most 10 seconds** before abandoning connections still open. - **Unmatched request paths report `endpoint="unmatched"` in metrics** instead of each creating a permanent series. Dashboards that group by raw path lose those values. ## v0.13.4 (2026-08-14) _Based on Lakekeeper OSS v0.13.3._ ### Bug Fixes - **Audit log fields are emitted as structured JSON.** `actor`, `action`/`actions`, `entity`/`entities`, `authorizations` and `context` were emitted as strings containing escaped pseudo-JSON, so audit pipelines could not read them as objects without decoding each field first. They are now nested JSON objects, as documented. See _Upgrade Notes_. - **Reduced memory growth from allocator fragmentation.** The server now uses jemalloc as its global allocator. Deployments that saw `container_memory_working_set_bytes` climb steadily without returning to baseline — a glibc malloc fragmentation pattern — should see flatter memory use. The effect depends on workload, and this changes the allocator only. - **JSON logs no longer carry duplicate span data.** Every line included both a `span` object and a `spans` array with the same content; only `spans` is emitted now. Applies to `lakekeeper-plus` and `lakekeeper-maintenance`. - **`LAKEKEEPER__DEBUG__LOG_AUTHORIZATION_HEADER` now applies to UI-server routes.** The setting was silently ignored there, so enabling it produced no `authorization` field on those request spans. ### Upgrade Notes - **Audit log consumers must read objects, not strings.** If your pipeline JSON-decodes the audit fields a second time to get at their contents, that step now fails or double-decodes — read them directly instead. The previous string form parsed only by luck: Rust `Debug` escapes non-ASCII as `\u{1F600}`, which is not valid JSON, so any field carrying such text would have broken the parse outright. Consumers that already tolerate both shapes need no change, and rollout order does not matter for them. ## v0.13.3 (2026-07-24) _Based on Lakekeeper OSS v0.13.3._ ### Highlights - **Admission-gate roles now take effect.** Roles granted by a `role_granting` admission check are finally evaluated by authorization — completing the external admission gate shipped in v0.13.0, whose granted roles previously never reached Cedar. ### Features - **Apply admission-gate roles in Cedar authorization.** A new `AdmissionRoleProvider` (an uncached role-provider-chain leaf, mirroring the token role provider) serves the caller's admission-granted roles under their configured provider id; Cedar materialises them as `Role` entities and evaluates policies against them. Plus wires one provider per role-provider id the gate can mint under, so a gate-only deployment still resolves roles. - **No-Access page for instance-level 403.** An authenticated caller denied access to the instance (e.g. rejected by the admission gate) now lands on a dedicated No-Access page with proper 403 routing, instead of a broken view. (console v0.16.4) ### Upgrade Notes - **Conflicting role-provider ids now fail fast.** A gate-minted `role_provider_id` that collides with a configured or token role-provider id is rejected at startup with `ExtraProviderIdConflict` — give the gate's minted roles a distinct provider id. - **Admission roles apply under Cedar only.** With the OpenFGA authorizer, or Cedar in externally-managed mode, admission-granted roles are not applied; the server logs a warning at startup so a misconfiguration is visible. ## v0.13.2 (2026-07-23) _Based on Lakekeeper OSS v0.13.3._ ### Bug Fixes - **Maintenance page.** Console bump to 0.16.3 (console-components 0.17.2, console-plus-components 0.11.0). Redesigned maintenance date-range filters, suppressed spurious 403 notifications for maintenance tasks, and fixed the Home page chart title. ## v0.13.1 (2026-07-20) _Based on Lakekeeper OSS v0.13.3._ ### Highlights - **Okta role provider.** Resolve a user's Okta group memberships to Lakekeeper roles via the Okta management API — private-key-JWT client auth with DPoP (RFC 9449) proofs enabled by default. - **Provider-synced roles are now protected.** Roles owned by a configured role provider (Okta, Entra, LDAP, or token IdP) can no longer be mutated through the management API, so the next provider sync can't silently clobber manual edits. ### Features - **Okta role provider (with DPoP).** Group memberships resolved via `GET /users/{id}/groups` (Link-header pagination, keyed by immutable group id). OAuth2 client-credentials + private-key-JWT (JWK or PEM key); DPoP on by default with ephemeral P-256 proofs and nonce challenge/replay handling — opt out to Bearer. Requires the `okta.users.read` scope; wrapped in the shared role cache. See the Okta role-provider docs. - **Managed-role write protection.** The Cedar authorizer now reports its configured provider namespaces to the management-API guard, so create / update / delete / source-system rebind / member (un)assignment on a provider-owned role is rejected with `400 ManagedRoleImmutable`. Native `lakekeeper` roles and the reserved `system` namespace are never included, so API-native and catalog-managed roles stay writable. Active only when a role provider is configured. - **Post-logout redirect controls.** Two new UI env vars — `LAKEKEEPER__UI__OPENID_POST_LOGOUT_REDIRECT_URL` and `LAKEKEEPER__UI__OPENID_POST_LOGOUT_REDIRECT_DISABLED`. - **Static-asset caching in the UI server.** Per-class `Cache-Control` plus weak `ETag`/`304` on bundled assets: content-hashed `assets/*` are cached immutably and the DuckDB WASM is no longer re-downloaded on every load, while `index.html` stays uncached so runtime config placeholders remain fresh. ### Bug Fixes - **Maintenance page for warehouse-only permissions.** Console bump to 0.16.1 (console-components 0.17.1) fixes the maintenance view for users who hold only warehouse-level permissions. ### Upgrade Notes - **Provider-role edits now return `400 ManagedRoleImmutable`.** If you previously edited provider-synced roles (Okta/Entra/LDAP/token) through the management API, those calls are now rejected — such edits were overwritten by the next sync anyway. Manage those roles at the source. No migration. - **Building Plus from source:** the Kubernetes client stack moved to k8s-openapi 0.28 / kube 4.0 (pulled in by the upstream limes 0.4.2 bump). Prebuilt binaries and images are unaffected. ### Upstream Lakekeeper changes (up to Lakekeeper v0.13.3) Notable for Plus users: - **Configurable Kubernetes subject source.** `LAKEKEEPER__KUBERNETES_AUTHENTICATION_SUBJECT_SOURCE=username` derives a service account's Lakekeeper user id from `system:serviceaccount::` (stable across clusters) instead of the per-cluster `uid` (default, unchanged), so Kubernetes roles and instance admins can be pre-provisioned ([lakekeeper#1899](https://github.com/lakekeeper/lakekeeper/pull/1899)). - **Reject role writes in provider-managed namespaces** — the upstream API guard behind the managed-role protection above ([lakekeeper#1891](https://github.com/lakekeeper/lakekeeper/pull/1891)). - Stop evaluating a discarded `Select` on target-view load, avoiding a spurious authorization check ([lakekeeper#1886](https://github.com/lakekeeper/lakekeeper/pull/1886)). ## v0.13.0 (2026-07-02) _Based on Lakekeeper OSS v0.13.1._ ### Highlights - **Microsoft Entra ID (Graph) role provider.** Resolve a user's transitive Entra group memberships into Lakekeeper roles — with secret, certificate, managed-identity, and workload-identity credentials, sovereign-cloud support, and built-in throttling/retry. - **External admission gate.** A new post-authentication seam can ask your control plane whether an already-authenticated caller may use this instance — for IdPs that issue broad, non-instance-scoped tokens — and contribute the caller's resolved roles. - **Console overhaul.** The bundled UI jumps to v0.13.2: a Files/storage explorer with in-browser Parquet/Avro/CSV preview, per-entity action menus, datasets as a first-class entity, redesigned view and table-health pages, a Role Members tab, and an enterprise usage-report builder. ### Features - **Entra ID / Microsoft Graph role provider.** Paged `transitiveMemberOf` resolution; credential methods secret / certificate / managed-identity / workload-identity; public, US-gov, and China clouds; retries on 429 (honoring `Retry-After`) and transient 5xx. - **AD range retrieval for LDAP attribute-mode groups.** Active Directory returns >1500 group values under a ranged `memberOf;range=…` key; attribute mode now walks the range windows, so users in many groups are no longer silently truncated. OpenLDAP/389-DS behavior is unchanged. - **External enforce-endpoint admission gate** (`lakekeeper-admission-enforce`). Configurable named checks POST to your endpoint; the HTTP status is the decision (2xx allows and grants the check's role, `403` denies, anything else fails closed with `503` + `Retry-After`). Allow and deny are both cached; caller bearer-token relay is opt-in and never logged. - **Persist OIDC token roles for DEFINER views.** Opt-in via `LAKEKEEPER__ROLE_PROVIDER_CHAIN__PERSIST_TOKEN_ROLES` (default off) — mirrors a user's OIDC-token roles into the catalog so authorization can evaluate them when the user isn't the live caller, e.g. a DEFINER view running as its owner. Write-gated; no migration. - **Generic-table parity in Cedar authorization.** Non-Iceberg generic tables (e.g. Lance, Delta) now resolve and authorize through the Cedar surface exactly like tables and views. - **Destructive-delete context for Cedar policies.** `force` / `purge` / `recursive` and a warehouse `soft_delete_enabled` attribute are now in the Cedar request context, so a policy can forbid hard deletes that would bypass configured soft-deletion. - **Role-membership actions in Cedar.** The new manage/read role-assignment actions map to dedicated fine-grained Cedar actions, so policy authors control their bundling. - **Destination-aware role source-system rebind.** A dedicated `update_source_system` Cedar action exposes the target provider/source, so rebinds can be gated by destination — something the coarse upstream OpenFGA relation cannot express. - **Per-decision policy trace in authorization audit.** Audit events and the `/check` endpoint now record which Cedar policies determined each allow/deny outcome. - **Schedule maintenance directly.** `expire_snapshots` and `remove_orphan_files` can be triggered per table via the task-queue schedule endpoint, without waiting for a commit hook. ### Bug Fixes - **Corrupt-manifest orphan-files task no longer retries forever.** A permanent failure (e.g. a corrupt Avro manifest) is now classified permanent and not requeued, instead of failing silently and re-running every day. Maintenance workers (`remove_orphan_files`, `expire_snapshots`) also persist a readable failure reason, surfaced in the task-details API — no server-log access required. ### Breaking Changes - **Default storage layout is now flat** (inherited from upstream Lakekeeper 0.13): new namespaces use `/` instead of nesting tabulars under the parent-namespace UUID. Not retroactive — existing namespaces and paths are unchanged — so explicitly configure the full-hierarchy layout if you need the old behavior for new namespaces ([lakekeeper#1853](https://github.com/lakekeeper/lakekeeper/pull/1853)). ### Upgrade Notes - **Encrypted tables are skipped by maintenance.** `expire_snapshots` and `remove_orphan_files` now detect Iceberg native encryption (format v3) via the immutable `encryption.key-id` property and skip such tables — Lakekeeper cannot read their encrypted manifests, and processing anyway risked deleting live data. Manually scheduling either task on an encrypted table returns `400`. - **Downgrade protection** (upstream): `serve` refuses to start against a database already migrated by a newer binary. After a rollback, start the older binary with `serve --force-start`, accepting the schema-incompatibility risk ([lakekeeper#1861](https://github.com/lakekeeper/lakekeeper/pull/1861)). - **Docker base images** moved from Debian 12 (bookworm) to Debian 13 (trixie). - **Building Plus from source:** the catalog Postgres backend and the NATS/Kafka event backends are now separate upstream crates (`lakekeeper-storage-postgres`, `lakekeeper-events-nats`, `lakekeeper-events-kafka`). Prebuilt binaries and images are unaffected ([lakekeeper#1812](https://github.com/lakekeeper/lakekeeper/pull/1812), [lakekeeper#1814](https://github.com/lakekeeper/lakekeeper/pull/1814)). ### Upstream Lakekeeper changes (up to Lakekeeper v0.13.1) Rolls up OSS **v0.12.4**, **v0.13.0**, and **v0.13.1** (full list in the [Lakekeeper release notes](https://docs.lakekeeper.io/about/release-notes/)). Notable for Plus users: - **Generic Table API** — register non-Iceberg tables (Lance, Delta) as first-class generic tables with credential vending and full authorization ([lakekeeper#1673](https://github.com/lakekeeper/lakekeeper/pull/1673), [lakekeeper#1813](https://github.com/lakekeeper/lakekeeper/pull/1813)); surfaced in Plus through the Cedar generic-table parity above. - **Operator-owned warehouses.** A `managed_by` marker locks warehouse spec mutations (delete, rename, (de)activate, storage profile, protection, format-version policy) to instance admins ([lakekeeper#1828](https://github.com/lakekeeper/lakekeeper/pull/1828)). - **Authorizer-independent role-membership API** — one management surface to list/add/remove a role's members regardless of the configured authorizer ([lakekeeper#1829](https://github.com/lakekeeper/lakekeeper/pull/1829)). - **Multiple OIDC providers** at once via `LAKEKEEPER__OPENID_PROVIDERS` (e.g. Okta for users + a cloud issuer for service accounts) ([lakekeeper#1760](https://github.com/lakekeeper/lakekeeper/pull/1760)). - **Microsoft OneLake / Fabric storage** profile, including workspace private-link endpoints ([lakekeeper#1852](https://github.com/lakekeeper/lakekeeper/pull/1852)). - **Per-warehouse table format-version policy** — allowed Iceberg format versions and an optional default per warehouse ([lakekeeper#1786](https://github.com/lakekeeper/lakekeeper/pull/1786)). - **Customer-managed KMS encryption** — warehouses with `aws-kms-key-arn` advertise `s3.sse.type=kms`, so vended-credential writes use your KMS key ([lakekeeper#1847](https://github.com/lakekeeper/lakekeeper/pull/1847)). - **Cache hardening for large fleets** — single-flight read-throughs and TTL jitter cut thundering-herd load on the database and on rate-limited STS/SAS endpoints ([lakekeeper#1833](https://github.com/lakekeeper/lakekeeper/pull/1833), [lakekeeper#1837](https://github.com/lakekeeper/lakekeeper/pull/1837)). - `/health` now returns `503` (not `200`) when unhealthy, so Kubernetes HTTP probes detect it ([lakekeeper#1802](https://github.com/lakekeeper/lakekeeper/pull/1802)). - Postgres migration locks are transaction-scoped, so a failed migration can't leak an advisory lock that blocks future migrations ([lakekeeper#1790](https://github.com/lakekeeper/lakekeeper/pull/1790)). ## v0.12.2 (2026-05-26) ### Highlights - Orphan-file cleanup now schedules itself adaptively per table — running more often where files accumulate and backing off where they don't — with a new dry-run mode. - The LDAP role provider can resolve groups via subtree **Search** and conditional **Branching**, not just the `memberOf` attribute. ### Features - **Adaptive orphan-file scheduling.** The remove-orphan-files worker now self-tunes its cadence based on how fast reclaimable data builds up, and adds a dry-run mode that reports what it _would_ delete. Orphan removal is now opt-in via `enable-remove-orphan-files`; default retention raised from 3 to 7 days. See the Table Maintenance docs for full config. - **LDAP group resolution modes.** Resolve group memberships via `Search` (paged subtree) or `Branching` (per-user-DN rules) in addition to the `memberOf` attribute; the resolution mode is recorded in audit logs. - **Build metadata in Server Info.** Server Info now reports Lakekeeper, Enterprise, and Console versions and commit SHAs, so deployed builds are easy to identify. ### Breaking Changes - The orphan-files task-queue API was renamed `remove_orphaned_files` → `remove_orphan_files` (paths, config schemas, and worker/enable fields). ### Upgrade Notes - Update any automation/IaC to the new `remove_orphan_files` task-queue path and schema names. - Orphan removal is now **opt-in** (it ran by default in 0.12.1): set `enable-remove-orphan-files=true` to keep it active. Default retention is now 7 days. ### Upstream Lakekeeper changes (bump to v0.12.3) - Core and extension database migrations now apply atomically — no partial-migration state ([lakekeeper#1768](https://github.com/lakekeeper/lakekeeper/pull/1768)). - Fixed table property removal being lost when no properties remained ([lakekeeper#1767](https://github.com/lakekeeper/lakekeeper/pull/1767)). - New read-only maintenance mode for the server ([lakekeeper#1765](https://github.com/lakekeeper/lakekeeper/pull/1765)). - Users may now share an email address — the unique-email constraint was dropped ([lakekeeper#1755](https://github.com/lakekeeper/lakekeeper/pull/1755)). - New `LAKEKEEPER__UI__ENABLE_SURVEYS` flag to opt out of in-console surveys ([lakekeeper#1750](https://github.com/lakekeeper/lakekeeper/pull/1750)). ## v0.12.1 (2026-05-10) ### Features - **Remove Orphan Files.** New maintenance capability that reclaims storage by deleting data, manifest, and metadata files no longer referenced by any snapshot — available as a server background worker and as a `remove-orphan-files` subcommand (with dry-run). Enabled by default; set `LAKEKEEPER__TASK_REMOVE_ORPHANED_FILES_WORKERS=0` to disable. Respects `gc.enabled` and per-table opt-out properties. See the Table Maintenance docs for full config. - **Bounded orphan-files runtime.** Cap how long a single orphan-files run may take with a configurable max run time. ### Bug Fixes - Warehouse rename no longer leaves a stale name in the UI table preview (bundled UI 0.7.12). ### Upgrade Notes - The orphan-files worker is **enabled by default** (2 workers); set `LAKEKEEPER__TASK_REMOVE_ORPHANED_FILES_WORKERS=0` to disable. By default it only deletes files older than 3 days and honors `gc.enabled` / per-table opt-out. ### Upstream Lakekeeper changes (bump to v0.12.2) - OpenFGA: rebuild/reconcile authorization tuples from the catalog, and support switching an existing server to OpenFGA ([lakekeeper#1731](https://github.com/lakekeeper/lakekeeper/pull/1731), [lakekeeper#1733](https://github.com/lakekeeper/lakekeeper/pull/1733)). - OPA Trino batch authorization gains a broad-access fast path for warehouses/namespaces ([lakekeeper#1727](https://github.com/lakekeeper/lakekeeper/pull/1727)). - Storage: dropped the opendal dependency and now validates vended credentials via `lakekeeper_io` ([lakekeeper#1737](https://github.com/lakekeeper/lakekeeper/pull/1737)). - ADLS fixes: correct SAS-token key removal and `%`-encoding in blob names ([lakekeeper#1746](https://github.com/lakekeeper/lakekeeper/pull/1746)). ## v0.12.0 (2026-04-21) ### Highlights - **Cedar authorization matured** into a configurable, inspectable system: derive user attributes from identity fields, reference roles by global ID, and use a new resolve-entities API + Console tabs to see exactly what drives a decision. - **Role providers** resolve user roles from external sources — including LDAP groups and table properties — with caching, metrics, and audit. - **Console**: a visual Cedar Policy Builder (beta) with a Cedar-aware editor, authorization-inspection tabs, and new statistics dashboards. - **Container images now default to `ubi10`** (breaking — see below). ### Features - **Cedar user identity derivations.** Extract attributes from identity fields with named-capture regex rules (optional `lowercase`/`uppercase` transform) and match policies on the derived values. - **Global role IDs in policies.** Reference provider-scoped global role IDs as Cedar property values, and use short-form roles without a default provider. - **Resolve-entities API.** `POST /management/v1/permissions/cedar/resolve-entities` returns the Cedar entities for any resource — for debugging why a decision was reached. - **SelectView action.** Adds `select` / `SelectView` (and `grant_select`) for views, aligning with the upstream data-plane authorization split. - **Role-provider subsystem.** LDAP Group Provider + token-provider chain, roles parsed from table properties, caching with stale-fallback and metrics, and an opt-in audit event for resolved roles — wired into the Cedar authorizer. Configure via `ROLE_PROVIDER_FILE` (TOML), overridable per-field by env vars. - **Richer permission-introspection audit.** `introspect_permissions` logs now include the inner check tuples and their individual decisions. - **Console: Cedar Policy Builder (beta).** Visual editor/builder with a CodeMirror Cedar editor (highlighting, autocomplete, inline diagnostics, format/validate via cedar-wasm) and live Evaluate. - **Console: authorization inspection + dashboards.** Tabs for entity/policy sources, schema, and resolve-entities; new Home and Warehouse statistics dashboards; storage-layout configuration. ### Bug Fixes - Cedar: correctness fixes around short-form role tags, per-request provider-ID derivation, Role subjects, and resolve-entities server gating. - Maintenance: expire-snapshots now removes statistics / partition-statistics from metadata, avoiding dangling references to deleted files. - TLS: added webpki and native root certs to the S3 client and UBI images, fixing handshake failures in some environments. - Console: correct handling of sub-namespaces containing dots; per-tab 403 keeps the navigation rail visible. ### Breaking Changes - Container images now default to **`ubi10`**; the `ubi9`-based image remains available under a separate tag. ### Upgrade Notes - If you pin the `ubi9` base image (e.g. FIPS/compliance), switch to the dedicated `ubi9` tag — the default is now `ubi10`. - Role-provider config can be supplied via `ROLE_PROVIDER_FILE` (TOML), with env vars overriding per field — review precedence if you set both. ### Upstream Lakekeeper changes - **Instance Admins** — server-wide admin role independent of project membership ([lakekeeper#1716](https://github.com/lakekeeper/lakekeeper/pull/1716)). - **Idempotency keys** for safely retrying mutating requests ([lakekeeper#1671](https://github.com/lakekeeper/lakekeeper/pull/1671)). - **`referenced-by`** to discover views referencing a table/view ([lakekeeper#1627](https://github.com/lakekeeper/lakekeeper/pull/1627)). - Configurable **trusted engines** in request metadata for authorization ([lakekeeper#1629](https://github.com/lakekeeper/lakekeeper/pull/1629)). - **Protect immutable table properties** (e.g. `encryption.key-id`) during commits ([lakekeeper#1700](https://github.com/lakekeeper/lakekeeper/pull/1700)). - Faster list namespaces/tables/views ([lakekeeper#1618](https://github.com/lakekeeper/lakekeeper/pull/1618)). --- # License Source: https://docs.lakekeeper.io/about/license/ # License --- ## Included projects Themes documentation is based on `MkDocs`. * MkDocs - [View license](https://www.mkdocs.org/about/license/). Many thanks to the authors and contributors of `MkDocs`! ## Apache License Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2025 Vakamo Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --- # Logos Source: https://docs.lakekeeper.io/about/logos/ # Logos Below you can find the logos available for download. Click on the links to download the files. ## Available Logos Lakekeeper Bear
Lakekeeper Bear Lakekeeper Bear Lakekeeper Bear Lakekeeper Bear Download All --- # Lakekeeper (OSS) Source: https://docs.lakekeeper.io/about/release-notes/ # Release Notes Highlights for each Lakekeeper release. For the full commit-level changelog, see the [GitHub Releases](https://github.com/lakekeeper/lakekeeper/releases) or [`CHANGELOG.md`](https://github.com/lakekeeper/lakekeeper/blob/main/CHANGELOG.md). For Lakekeeper+ releases, see the [Lakekeeper+ Release Notes](enterprise-release-notes.md). --8<-- "_includes/subscribe-form.html" _[Subscribe by email](subscribe.md) to hear about new releases, or **Watch → Releases** on [GitHub](https://github.com/lakekeeper/lakekeeper/releases)._ ## v0.13.4 (2026-09-10) ### Features - **GovCloud, China and ISO region support.** Vended-credential policies now carry the correct ARN partition (`aws-us-gov`, `aws-cn`, `aws-iso*`, `aws-eusc`), derived automatically from the region, endpoint or role ARN, so `AssumeRole` succeeds for buckets outside the commercial partition — nothing to configure (thanks @123digits) ([#1928](https://github.com/lakekeeper/lakekeeper/pull/1928)). ### Bug Fixes - Fixed a permissions leak: a table, view or generic table renamed into another namespace kept inheriting grants from the namespace it left ([#2013](https://github.com/lakekeeper/lakekeeper/pull/2013)). - Renaming a table onto a name that is already taken now returns `409 Conflict` instead of `404 Not Found`, matching the Iceberg REST spec, and renaming onto a soft-deleted name succeeds — as create already did ([#1955](https://github.com/lakekeeper/lakekeeper/pull/1955)). - Fixed four independent causes of resident memory growing until restart: jemalloc now compiles with `thp:never`, the route table is no longer rebuilt per accepted connection, connections get TCP keepalive and a header-read timeout so vanished peers are dropped, and unmatched request paths no longer become permanent Prometheus series. Settled RSS fell 68% in testing ([#1990](https://github.com/lakekeeper/lakekeeper/pull/1990)). ### Upgrade Notes - Deployments that renamed a table, view or generic table across namespaces should run `lakekeeper openfga reconcile --mode add-and-delete-drift` once after upgrading. The default `add-missing` mode cannot repair it — the stale permission edge is a surplus tuple, not a missing one. ## v0.13.3 (2026-08-16) ### Features - **Stable Kubernetes service-account identities.** `LAKEKEEPER__KUBERNETES_AUTHENTICATION_SUBJECT_SOURCE=username` derives a service account's user ID from `system:serviceaccount::` instead of the token UID, so roles and instance admins can be pre-provisioned (e.g. via the Terraform provider) and survive a cluster rebuild. The default `uid` is unchanged ([#1899](https://github.com/lakekeeper/lakekeeper/pull/1899)). - **Provider-managed roles are protected from drift.** Roles owned by a configured role provider (LDAP, Entra, Okta, token) can no longer be created, updated, deleted, rebound or have their members changed through the management API, since the next provider sync would silently clobber those edits. Nothing changes without a role provider configured ([#1891](https://github.com/lakekeeper/lakekeeper/pull/1891)). ### Bug Fixes - Fixed two denial-of-service advisories (RUSTSEC-2026-0194 and RUSTSEC-2026-0195) reachable through the S3 remote signer, which parses a client-supplied XML body; `quick-xml` is bumped to 0.41 ([#1885](https://github.com/lakekeeper/lakekeeper/pull/1885)). - Loading a table with `referenced-by` no longer runs a discarded authorization check on the target view, removing a wasted authorizer evaluation per request and a misleading `SelectView` entry from audit logs ([#1886](https://github.com/lakekeeper/lakekeeper/pull/1886)). ## v0.13.1 (2026-06-30) ### Bug Fixes - Console updated to v0.19.0; the default OAuth `client_id` no longer carries a leading slash ([ac1094a](https://github.com/lakekeeper/lakekeeper/commit/ac1094a52e51da586c60a27159f9720784eb66e7)). ## v0.13.0 (2026-06-28) ### Highlights - **Generic Table API.** Register non-Iceberg tables (e.g. Lance, Delta) as first-class generic tables and get credential vending, list/load/rename/drop, soft-delete/undrop, protection, and full authorization — without faking Iceberg metadata ([#1673](https://github.com/lakekeeper/lakekeeper/pull/1673), [#1813](https://github.com/lakekeeper/lakekeeper/pull/1813)). - **Operator-owned warehouses.** A `managed_by` marker lets a control plane (operator/IaC) own a warehouse, locking spec mutations (delete, rename, (de)activate, storage profile, protection, format-version policy) to instance admins even when authorization would otherwise allow them ([#1828](https://github.com/lakekeeper/lakekeeper/pull/1828)). - **Cache hardening for large fleets.** Hot read-through caches now coalesce concurrent identical misses (single-flight) and jitter their TTLs, cutting thundering-herd load on the database and on rate-limited cloud STS/SAS endpoints ([#1833](https://github.com/lakekeeper/lakekeeper/pull/1833), [#1837](https://github.com/lakekeeper/lakekeeper/pull/1837)). ### Features - **Per-warehouse table format-version policy.** Set the allowed Iceberg table format versions and an optional default per warehouse, enforced on create/commit/upgrade ([#1786](https://github.com/lakekeeper/lakekeeper/pull/1786)). - **Customer-managed KMS encryption.** Warehouses with `aws-kms-key-arn` now advertise `s3.sse.type=kms` to clients, so writes via vended credentials are encrypted with your KMS key ([#1847](https://github.com/lakekeeper/lakekeeper/pull/1847)). - **Schedule a task directly.** `POST .../task-queue/{queue_name}/schedule` schedules a task for a single table without waiting for a commit hook ([#1783](https://github.com/lakekeeper/lakekeeper/pull/1783)). - **Task failures are debuggable.** Failed tasks now surface the root-cause failure reason in task-detail responses, no server-log access required ([#1873](https://github.com/lakekeeper/lakekeeper/pull/1873)). - **Pluggable admission gates.** A new post-authentication seam lets an external service reject already-authenticated principals that have no entitlement on this instance and contribute their resolved roles ([#1865](https://github.com/lakekeeper/lakekeeper/pull/1865), [#1866](https://github.com/lakekeeper/lakekeeper/pull/1866), [#1869](https://github.com/lakekeeper/lakekeeper/pull/1869)). - **Authorizer-independent role-membership API.** A single management surface lists, adds, and removes a role's members (users or roles) and shows what a role or user belongs to — the same API regardless of the configured authorizer. The membership edges keep a single source of truth: the authorizer's own store when it manages assignments (e.g. OpenFGA), otherwise the catalog's Postgres tables ([#1829](https://github.com/lakekeeper/lakekeeper/pull/1829)). - **Iceberg 1.11 remote signing.** Emits the new `signer.uri`/`signer.endpoint` properties alongside the legacy `s3.signer.*` keys, so clients ≥1.11 stop logging deprecation warnings while older clients keep working ([#1820](https://github.com/lakekeeper/lakekeeper/pull/1820)). - **Per-decision authorization audit.** Authorization audit events now record the contributing policies behind each allow/deny outcome ([#1844](https://github.com/lakekeeper/lakekeeper/pull/1844)). - **More observability.** New client-side Postgres connection-pool metrics and event-listener dispatch timing metrics ([#1838](https://github.com/lakekeeper/lakekeeper/pull/1838), [#1863](https://github.com/lakekeeper/lakekeeper/pull/1863)). - **Console v0.15.1** — generic-tables (Lance/Delta) tab, Iceberg format-version policy editor, OneLake/Fabric backend, and per-queue maintenance summaries ([#1855](https://github.com/lakekeeper/lakekeeper/pull/1855)). ### Bug Fixes - `GET /config` now authorizes only the requested warehouse (fixing timeouts on large projects) and masks hidden/unknown warehouses identically so existence and UUIDs can't leak ([#1788](https://github.com/lakekeeper/lakekeeper/pull/1788)). - Conditional `loadTable` no longer returns `304` once the cached response's vended credentials have expired ([#1862](https://github.com/lakekeeper/lakekeeper/pull/1862)). - S3: keep the STS vended-credential policy within the packed-size limit and emit a reliable table resource ARN for paths with special characters ([#1857](https://github.com/lakekeeper/lakekeeper/pull/1857)). - Azure (ADLS): add a connect timeout and a larger retry budget so transient connect failures are retried; retry storage OAuth token acquisition for ADLS and GCS ([#1815](https://github.com/lakekeeper/lakekeeper/pull/1815), [#1827](https://github.com/lakekeeper/lakekeeper/pull/1827)). - Postgres: migration locks are now transaction-scoped, so a failed migration no longer leaks an advisory lock that could permanently block future migrations ([#1790](https://github.com/lakekeeper/lakekeeper/pull/1790)). ### Breaking Changes - **Default storage layout is now flat** — new namespaces use `/` instead of nesting tabulars under the parent-namespace UUID. Only namespaces created on/after 0.13 are affected; existing namespaces and tabulars keep their persisted paths (not retroactive, no migration), so a warehouse spanning the upgrade can hold a mix. To keep the previous behavior for new namespaces, explicitly configure the full-hierarchy layout ([#1853](https://github.com/lakekeeper/lakekeeper/pull/1853)). - **Event backends are separate crates.** The NATS and Kafka backends moved out of the core `lakekeeper` crate into `lakekeeper-events-nats` and `lakekeeper-events-kafka`. Prebuilt binaries and images are unaffected (same env vars); building from source must depend on the new crates — the `nats`, `kafka`, and `vendored-protoc` features are removed from `lakekeeper` ([#1814](https://github.com/lakekeeper/lakekeeper/pull/1814)). - **Postgres backend is a separate crate.** Catalog Postgres logic moved into `lakekeeper-storage-postgres`, making `lakekeeper` backend-agnostic. Source consumers of `lakekeeper::implementations::*` must now depend on the new crate; prebuilt binary/image users are unaffected ([#1812](https://github.com/lakekeeper/lakekeeper/pull/1812)). - **Role source-system rebind is now permission-gated.** `PUT /role/{id}/source-system` requires membership-control permission (OpenFGA model v4.6→v4.7), closing a privilege-escalation gap ([#1848](https://github.com/lakekeeper/lakekeeper/pull/1848)). - **Custom authorizers.** `are_allowed_*_actions_impl` now return `Vec` instead of `Vec`, and `RoleAction` gained a non-`Copy` `UpdateSourceSystem` variant; out-of-tree authorizer implementations must adapt. Built-in OpenFGA users are unaffected ([#1844](https://github.com/lakekeeper/lakekeeper/pull/1844), [#1848](https://github.com/lakekeeper/lakekeeper/pull/1848)). - **Regenerate action client SDKs.** `GET …/actions` now advertises action _kinds_ (`Lakekeeper*ActionKind`); the wire JSON is unchanged, but generated clients should be regenerated ([#1860](https://github.com/lakekeeper/lakekeeper/pull/1860)). ### Upgrade Notes - **Downgrade protection.** `serve` now refuses to start against a database already migrated by a newer binary and does not retry. After a rollback, start the older binary with `serve --force-start`, accepting the schema-incompatibility risk ([#1861](https://github.com/lakekeeper/lakekeeper/pull/1861)). - Docker base images moved from Debian 12 (bookworm) to Debian 13 (trixie) ([#1794](https://github.com/lakekeeper/lakekeeper/pull/1794)). - The docker-compose examples now use SeaweedFS instead of MinIO for object storage ([#1811](https://github.com/lakekeeper/lakekeeper/pull/1811)). ## v0.12.4 (2026-06-17) ### Features - **Multiple OIDC providers.** Authenticate tokens from several identity providers at once (e.g. Okta for users + a cloud OIDC issuer for service accounts) via `LAKEKEEPER__OPENID_PROVIDERS`; fully backwards-compatible with the existing single-provider config ([#1760](https://github.com/lakekeeper/lakekeeper/pull/1760)). - **Microsoft OneLake / Fabric storage + private endpoints.** New OneLake (ADLS Gen2) storage profile with configurable endpoint modes including workspace private link ([#1852](https://github.com/lakekeeper/lakekeeper/pull/1852)); Console updated to v0.14.3 for OneLake support. ### Bug Fixes - `/health` now returns HTTP `503` (not `200`) when aggregate health is unhealthy or unknown, so Kubernetes HTTP probes correctly detect an unhealthy server; the JSON body is unchanged ([#1802](https://github.com/lakekeeper/lakekeeper/pull/1802)). ## v0.12.3 (2026-05-26) ### Features - **Read-only maintenance mode.** Set `LAKEKEEPER__MAINTENANCE_MODE=read-only` to reject mutating requests with `503` + `Retry-After` during planned maintenance ([#1765](https://github.com/lakekeeper/lakekeeper/pull/1765)). - **Atomic core + extension migrations.** Schema migrations now apply in a single transaction, so an interrupted upgrade can't leave a half-migrated database ([aa734bf](https://github.com/lakekeeper/lakekeeper/commit/aa734bffcacd98566aa670f62341a4455833496c)). - **Users may share an email address.** The unique-email constraint was dropped, so multiple users can have the same email ([#1755](https://github.com/lakekeeper/lakekeeper/pull/1755)). - **Survey opt-out.** New `LAKEKEEPER__UI__ENABLE_SURVEYS` flag disables in-console surveys and their third-party requests ([719150b](https://github.com/lakekeeper/lakekeeper/commit/719150bed3b700308ff5954217cfad8aac5ba9cf)). - **Reserved `system` role provider** for catalog-managed roles ([#1776](https://github.com/lakekeeper/lakekeeper/pull/1776)). - **Console.** New "Export for GitHub" support bundle (server info + UI config, no tokens), a Feedback button, role-provider IDs in the overview, and a two-column Server Settings layout ([719150b](https://github.com/lakekeeper/lakekeeper/commit/719150bed3b700308ff5954217cfad8aac5ba9cf)). ### Bug Fixes - Fixed table property removal being lost when no properties remained ([#1767](https://github.com/lakekeeper/lakekeeper/pull/1767)). - Views now preserve protection (`protected=true`) across commits — previously lost on update (thanks @fallintoplace) ([#1770](https://github.com/lakekeeper/lakekeeper/pull/1770)). - `force=true` is now respected when dropping soft-deletion warehouses that contain views ([#1779](https://github.com/lakekeeper/lakekeeper/pull/1779)). - Fixed a memory leak from stale Vault (KV2) health status (thanks @fallintoplace) ([#1773](https://github.com/lakekeeper/lakekeeper/pull/1773)). - Postgres: rewrote the namespace trigger so `pg_restore` can replay it ([#1781](https://github.com/lakekeeper/lakekeeper/pull/1781)). ### Upgrade Notes - Minimum supported Rust version (MSRV) raised to 1.94 — affects building from source. ## v0.12.2 (2026-05-10) ### Features - Storage locations are canonicalised at parse time to avoid path aliases ([#1743](https://github.com/lakekeeper/lakekeeper/pull/1743)). - Object size is now exposed on `FileInfo` ([#1741](https://github.com/lakekeeper/lakekeeper/pull/1741)). ### Bug Fixes - **Security:** hardened S3 STS/CEL credential-vending policies against path injection ([#1740](https://github.com/lakekeeper/lakekeeper/pull/1740)). - Azure (ADLS): pre-encode `%` in blob names so the SDK no longer collapses distinct paths onto the same alias ([#1746](https://github.com/lakekeeper/lakekeeper/pull/1746)). - Postgres: apply `pg_acquire_timeout` to all connection-pool initialisations ([#1744](https://github.com/lakekeeper/lakekeeper/pull/1744)). ## v0.12.1 (2026-05-04) ### Highlights - **Instance Admins.** Designate break-glass principals that bypass control-plane authorization for management actions via `LAKEKEEPER__INSTANCE_ADMINS` (a list of `~` IDs). The bypass excludes data-plane operations and role-assumed requests ([#1716](https://github.com/lakekeeper/lakekeeper/pull/1716)). - **Safe switch to OpenFGA.** Existing deployments can adopt or rebuild OpenFGA: `openfga reconcile` rebuilds hierarchy tuples from the catalog (with dry-run and drift-deletion), and `reopen-bootstrap` re-enables bootstrap for recovery ([#1731](https://github.com/lakekeeper/lakekeeper/pull/1731), [#1733](https://github.com/lakekeeper/lakekeeper/pull/1733)). - **OpenDAL dropped.** Storage I/O now goes exclusively through the hyperscaler-native backends wrapped by `lakekeeper-io`, including vended-credential validation ([#1737](https://github.com/lakekeeper/lakekeeper/pull/1737)). - **Security.** Upgraded `rustls-webpki` to 0.103.12 for RUSTSEC-2026-0098 ([#1713](https://github.com/lakekeeper/lakekeeper/pull/1713)). ### Features - **Trusted engines for views (`referenced-by`).** Validates the `referenced-by` parameter and resolves view-on-view chains for batch authorization, enabling secure DEFINER-style execution for trusted query engines — configured under `LAKEKEEPER__TRUSTED_ENGINES__` ([#1647](https://github.com/lakekeeper/lakekeeper/pull/1647)). - **Protected security-relevant properties.** Only a matched trusted engine may set or remove view owner / run-as properties (case variants rejected), and commits can no longer overwrite immutable table properties such as `encryption.key-id` ([#1700](https://github.com/lakekeeper/lakekeeper/pull/1700), [#1724](https://github.com/lakekeeper/lakekeeper/pull/1724)). - **Richer audit.** `introspect_permission` events now include the inner check tuples and their decisions, and events record whether access was granted internally, via an instance admin, or by the authorizer ([#1697](https://github.com/lakekeeper/lakekeeper/pull/1697)). - **Data-plane `Select` action for views**, plus a public `resolve_principal` API for downstream API-to-authz `UserOrRole` conversion ([#1721](https://github.com/lakekeeper/lakekeeper/pull/1721), [#1703](https://github.com/lakekeeper/lakekeeper/pull/1703)). - **OPA bridge.** View-on-view queries via `CreateViewWithSelectFromColumns`, the Trino `ADD_FILES` operation, and a warehouse/namespace broad-access fast path for batch authorization ([#1712](https://github.com/lakekeeper/lakekeeper/pull/1712), [#1727](https://github.com/lakekeeper/lakekeeper/pull/1727)). - **Extended Server Info** with console information and commit SHAs ([#1725](https://github.com/lakekeeper/lakekeeper/pull/1725)). ### Bug Fixes - Added `webpki_root_certs` / UBI native certs to the S3 client to fix TLS trust issues ([#1720](https://github.com/lakekeeper/lakekeeper/pull/1720)). - Namespace/table case handling: allow renaming a table to a different case of its own name; lookups return the caller's case, ID lookups the canonical case ([7c26309](https://github.com/lakekeeper/lakekeeper/commit/7c263091f255b75ed5d66024b5bc6b29ef553508)). - Pinned `gcloud-storage` / `gcloud-auth` to `~1.2` to avoid a `reqwest-middleware` conflict ([#1701](https://github.com/lakekeeper/lakekeeper/pull/1701)). - ADLS: remove the actual matched SAS token key rather than its prefix ([76a091b](https://github.com/lakekeeper/lakekeeper/commit/76a091b9b01ba507cc448a56241d54f526c19a14)). - Console: fixed base-URL trailing slash, Vite 8 authentication breakage, and a stale warehouse name after rename ([#1729](https://github.com/lakekeeper/lakekeeper/pull/1729), [#1723](https://github.com/lakekeeper/lakekeeper/pull/1723)). ### Upgrade Notes - **Switching to OpenFGA on an existing instance is now safe:** run `openfga reconcile` (dry-run first; drift-deletion mode to also remove stale tuples), and `reopen-bootstrap` to re-enter bootstrap if needed. The minimum required OpenFGA version was raised. - **OpenDAL removed** — storage now relies solely on the native S3/GCS/ADLS backends in `lakekeeper-io`; re-verify storage and vended-credential config after upgrade. - Deploying into a custom Postgres schema is now supported and documented ([#1714](https://github.com/lakekeeper/lakekeeper/pull/1714)). ## v0.12.0 (2026-04-01) ### Highlights - **Audit Event System.** Authorization decisions and catalog operations emit dedicated audit events with exactly-once-per-call delivery, giving a reliable trail of who did what ([b77c687](https://github.com/lakekeeper/lakekeeper/commit/b77c68740a67221669acaa122742b3912d48aeb5)). - **Idempotency keys for safe retries.** Send an `Idempotency-Key` header on mutating requests and Lakekeeper replays the original response instead of applying the change twice — on by default, scoped per warehouse, with a 30-minute key lifetime ([#1671](https://github.com/lakekeeper/lakekeeper/pull/1671)). - **Customizable storage layouts.** Choose how namespace/table paths are templated on S3/GCS/ADLS using `{uuid}` / `{name}` placeholders ([#1615](https://github.com/lakekeeper/lakekeeper/pull/1615), [#1628](https://github.com/lakekeeper/lakekeeper/pull/1628)). - **Structured JSON logs.** Log output is now structured JSON with objects as field values, ready for log-pipeline ingestion ([b77c687](https://github.com/lakekeeper/lakekeeper/commit/b77c68740a67221669acaa122742b3912d48aeb5)). ### Features - **Configurable STS endpoint.** Set a separate `sts-endpoint` on an S3 storage profile when your S3-compatible storage exposes STS on a different host ([#1653](https://github.com/lakekeeper/lakekeeper/pull/1653)). - **Fallback subject claims.** OpenID subject-claim config accepts a comma-separated list; the first matching claim in the token wins, easing varied-IdP integration ([#1646](https://github.com/lakekeeper/lakekeeper/pull/1646)). - **Request size/time limits.** New `LAKEKEEPER__MAX_REQUEST_BODY_SIZE` (default 2 MB) and `LAKEKEEPER__MAX_REQUEST_TIME` (default `30s`) guard against oversized and slow requests ([#1583](https://github.com/lakekeeper/lakekeeper/pull/1583)). - **Trusted engines configuration.** Declare trusted engines (e.g. Trino) with a selectable Invoker/Definer security model; the engine is auto-detected from token audience and recorded in request metadata ([#1629](https://github.com/lakekeeper/lakekeeper/pull/1629)). - **Role assignment store and cache** with provider-scoped role identifiers, plus a roles cache for faster authorization ([#1638](https://github.com/lakekeeper/lakekeeper/pull/1638), [#1623](https://github.com/lakekeeper/lakekeeper/pull/1623)). - **Iceberg V3 Variant datatype** support, validated against Spark 4 integration tests ([daa7947](https://github.com/lakekeeper/lakekeeper/commit/daa7947333097b25e09a91281a6057d334db599c)). - **Tokio runtime metrics** exported for runtime observability ([#1664](https://github.com/lakekeeper/lakekeeper/pull/1664)). - **Reduced memory footprint** by switching the allocator to jemalloc ([0eaeedc](https://github.com/lakekeeper/lakekeeper/commit/0eaeedc8411120f18ec9229b4dd08c36dd294d23)). - **OPA bridge improvements:** batch-authorization optimization, configurable admin users, system-schema handling, and request-context forwarding ([#1674](https://github.com/lakekeeper/lakekeeper/pull/1674), [#1662](https://github.com/lakekeeper/lakekeeper/pull/1662)). - **Faster listing** of namespaces, tables, and views ([#1618](https://github.com/lakekeeper/lakekeeper/pull/1618)). - **Console.** New Home dashboard with usage statistics and API-call charts; branch operations (create, rename, delete, rollback, fast-forward); a Properties dialog for tables/views/namespaces; storage-layout configuration; and a local query engine with memory management ([#1621](https://github.com/lakekeeper/lakekeeper/pull/1621), [#1634](https://github.com/lakekeeper/lakekeeper/pull/1634)). ### Bug Fixes - Fixed duplicate results when paginating `list_tabulars` ([#1682](https://github.com/lakekeeper/lakekeeper/pull/1682), [#1684](https://github.com/lakekeeper/lakekeeper/pull/1684)). - Fixed a memory leak in the S3 identity cache ([0eaeedc](https://github.com/lakekeeper/lakekeeper/commit/0eaeedc8411120f18ec9229b4dd08c36dd294d23)). - Allowed updating the storage-profile region when an S3 endpoint is set ([#1678](https://github.com/lakekeeper/lakekeeper/pull/1678)). - Patched security advisories in crypto dependencies (`aws-lc-sys` / `rustls-webpki`) ([#1672](https://github.com/lakekeeper/lakekeeper/pull/1672)) and an `lz4_flex` memory-leak advisory ([#1665](https://github.com/lakekeeper/lakekeeper/pull/1665)). ### Breaking Changes - **Cache metrics unified.** Per-cache metric names are replaced by shared names distinguished by a `cache_type` label ([#1641](https://github.com/lakekeeper/lakekeeper/pull/1641)). - **Structured log format.** Logs now emit structured JSON with objects as field values instead of flat text ([b77c687](https://github.com/lakekeeper/lakekeeper/commit/b77c68740a67221669acaa122742b3912d48aeb5)). ### Upgrade Notes - **Cache metrics:** migrate dashboards/alerts from the old per-cache metric names to the unified `lakekeeper_cache_hits_total` / `lakekeeper_cache_misses_total` / `lakekeeper_cache_size`, filtering by the `cache_type` label (`role`, `warehouse`, `namespace`, `secrets`, `stc`). - **Structured logs:** log consumers must parse JSON rather than plain text. Set `LAKEKEEPER__DEBUG__EXTENDED_LOGS=true` to include `filename`/`line_number` fields. - **S3 credential fields** dropped their `aws_` prefix; the old names remain accepted as aliases, but update to the new names ([#1685](https://github.com/lakekeeper/lakekeeper/pull/1685)). --- # Stay Updated Source: https://docs.lakekeeper.io/about/subscribe/ # Stay Updated Want to hear about new Lakekeeper releases? Subscribe below and we'll email you when a notable release ships — new features, breaking changes, and upgrade notes worth knowing about. - **Low volume.** Only occasional release announcements — no marketing, no drip campaigns. - **You're in control.** Confirm your address after signing up, and unsubscribe anytime with one click. - **Prefer a feed?** Every release is also on [GitHub Releases](https://github.com/lakekeeper/lakekeeper/releases) — click **Watch → Custom → Releases** to get GitHub notifications instead. --8<-- "_includes/subscribe-form.html" --- # Admission Gates Source: https://docs.lakekeeper.io/docs/latest/admission/ # Admission Gates {#admission-gates} An **admission gate** makes a coarse allow/deny decision about an *already-authenticated* request **before it reaches any handler** — distinct from the per-resource [Authorizer](./authorization.md). Use one to consult an external control-plane entitlement service, suspend a tenant or principal, or reject revoked tokens. Gates run on every authenticated request, in order, after the actor and instance-admin status are resolved; the first rejection wins. A gate returns either a terminal `403 Forbidden` or — when it fails closed because an upstream it depends on is unreachable — a `503` with a `Retry-After`. The gate seam itself ([`AdmissionGate`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/admission.rs)) is a Rust trait; see [Customize](./customize.md) to implement your own. This page documents the **external enforce-endpoint gate** that ships ready-to-configure with Lakekeeper Plus. ## When to use it This gate is for deployments whose IdP issues **broad, non-instance-scoped tokens**, where a separate service — not the token — is authoritative for whether the caller may use *this* Lakekeeper instance. After authentication, the gate asks that service, per caller, and either lets the request through or rejects it. If your tokens already carry the entitlement (claims, roles, audience), you don't need this — use [authentication](./authentication.md) and [authorization](./authorization.md). ## How it works The gate evaluates one or more named **checks** against a configured enforce endpoint. **Each check carries its own complete request body** (any JSON shape), so the gate imposes no schema on the enforce API — you decide what the endpoint sees. Each check is a single `POST`; the decision is the HTTP status: | Upstream status | `kind = "gating"` | `kind = "role_granting"` | | ------------------------------------------ | --------------------------------------- | ----------------------------- | | `2xx` | admit + grant role | grant role | | `403` (exactly) | **reject request (`403`)** | withhold role, continue | | any other status, timeout, network error | **`503` + `Retry-After`** (fail closed) | same — `503` fail closed | Only an exact `403` is read as an authoritative deny. Every other non-`2xx` status — including other `4xx` (e.g. `400`, `401`, `404`, `429`) and any `5xx` — is treated as the endpoint being unable to give a verdict, so the gate fails closed with a `503`. This is deliberate: a misconfigured or malfunctioning enforce endpoint must never silently admit. If your endpoint signals "denied" with a status other than `403`, map it to `403` on its side. On admit, each passing check contributes its role to the request's admission roles, consumed by authorization downstream. - **Operator-defined body.** A check's `body` is a JSON string of arbitrary shape, parsed and validated once at startup. The only substitutions the gate makes are the request-derived placeholders `{{subject}}` (the token `sub`) and `{{idp_id}}` inside string values; everything else is sent literally. Invalid JSON or an unknown placeholder is rejected at startup. The gate models no "actions"/"resource" concepts — those are just whatever you write in the body. - **IdP-scoped.** The gate only governs tokens from the configured `idp_id`; tokens from any other identity provider are admitted untouched. - **Cached.** Decisions are cached in memory per `(subject, check)` for `cache_ttl_secs`. Both allow *and* deny are cached, so a denied-but-authenticated caller triggers at most one upstream call per TTL and cannot amplify load; transient `5xx`/timeout results are never cached. - **Fail closed.** Anything other than `2xx`/`403` becomes a `503` with `Retry-After`. - **Token relay is opt-in.** The caller's bearer token is forwarded only when `auth` is `forward_caller_token`; otherwise the endpoint is reached with the static `headers` only. A forwarded token goes over the configured URL (use TLS) and is never logged. ## Configuration The gate is **disabled unless an `[admission_enforce]` block is present**. Like [Cedar derivations](./authorization-cedar.md) and [role providers](./configuration.md#role-provider), this is nested config, so it is configured via a TOML file with full environment-variable parity — point `LAKEKEEPER__ADMISSION_ENFORCE_FILE` at a TOML file, and/or set `LAKEKEEPER__ADMISSION_ENFORCE__*` variables on top. ### `[admission_enforce]` | Key | Required | Default | Description | | ------------------------------ | -------- | ------- | ----------------------------------------------------------------- | | `endpoint` | yes | — | Enforce endpoint URL (`POST`). Validated at startup. | | `idp_id` | yes | — | Only govern tokens from this IdP; others are admitted untouched. | | `role_provider_id` | yes | — | Provider namespace for the synthesized admission roles. | | `cache_ttl_secs` | no | `60` | TTL for cached allow/deny decisions. | | `cache_max_entries` | no | `10000` | Max cached decisions. | | `request_timeout_secs` | no | `5` | Per-request timeout. | | `connect_timeout_secs` | no | `2` | Connection timeout. | | `unavailable_retry_after_secs` | no | `5` | `Retry-After` returned on the fail-closed `503`. | | `headers` | no | `{}` | Extra static headers sent on every call (e.g. a service API key). | | `auth` | no | _none_ | How to authenticate (see below). Omit to send no `Authorization`. | | `checks` | yes | — | Named map of checks (at least one). See below. | ### `[admission_enforce.auth]` Omit to forward no token. Currently one scheme — relay the caller's bearer token: ```toml [admission_enforce.auth] type = "forward_caller_token" # sent as `Authorization: Bearer ` ``` When set, the request **must** carry a bearer token; the gate fails closed if it is absent. ### `[admission_enforce.checks.]` Each check is keyed by a name you choose (used in the cache key and logs). Use lowercase letters, digits, and underscores, so the name round-trips through `…__CHECKS____…` environment variables. | Key | Required | Default | Description | | ------------------ | -------- | ---------------- | --------------------------------------------------------------------- | | `kind` | yes | — | `gating` (a `403` rejects the request) or `role_granting` (`403` withholds the role). | | `body` | yes | — | This check's complete request body as a JSON string (any shape; parsed at startup). `{{subject}}`/`{{idp_id}}` are substituted; everything else is literal. | | `role_source_id` | yes | — | Source id of the admission role granted when the check passes. | | `role_provider_id` | no | gate-level value | Override the role provider namespace for this check. | ## Example ```toml [admission_enforce] endpoint = "https://control-plane.internal/v1/authorize" idp_id = "oidc" role_provider_id = "control-plane" cache_ttl_secs = 60 [admission_enforce.auth] type = "forward_caller_token" # A `403` here rejects the request outright. [admission_enforce.checks.instance_access] kind = "gating" role_source_id = "instance-access" body = ''' { "subject": "{{subject}}", "resource": "102befc3-424d-479e-b1f7-bb47c1e1a1a2", "resourceType": "project", "actions": ["workflows.instance.read"] } ''' # A `403` here only withholds the role; the request still proceeds. [admission_enforce.checks.workflow_editor] kind = "role_granting" role_source_id = "workflow-editor" body = ''' { "subject": "{{subject}}", "resource": "102befc3-424d-479e-b1f7-bb47c1e1a1a2", "resourceType": "project", "actions": ["workflows.instance.update", "workflows.instance.delete"] } ''' ``` The body shape is entirely yours — a different enforce API (e.g. OPA-style `{"input": {...}}`) is just a different `body`, with no code change: ```toml [admission_enforce.checks.can_read] kind = "gating" role_source_id = "reader" body = '''{ "input": { "user": "{{subject}}", "tenant": "acme", "verb": "read" } }''' ``` ### Configuring via environment variables Because the body is a single JSON string, the whole config — including bodies with arrays and mixed-case keys — round-trips through environment variables with full parity. The example above is equivalent to: ```bash LAKEKEEPER__ADMISSION_ENFORCE__ENDPOINT='https://control-plane.internal/v1/authorize' LAKEKEEPER__ADMISSION_ENFORCE__IDP_ID='oidc' LAKEKEEPER__ADMISSION_ENFORCE__ROLE_PROVIDER_ID='control-plane' LAKEKEEPER__ADMISSION_ENFORCE__AUTH__TYPE='forward_caller_token' LAKEKEEPER__ADMISSION_ENFORCE__CHECKS__INSTANCE_ACCESS__KIND='gating' LAKEKEEPER__ADMISSION_ENFORCE__CHECKS__INSTANCE_ACCESS__ROLE_SOURCE_ID='instance-access' LAKEKEEPER__ADMISSION_ENFORCE__CHECKS__INSTANCE_ACCESS__BODY='{"subject":"{{subject}}","resource":"102befc3-424d-479e-b1f7-bb47c1e1a1a2","resourceType":"project","actions":["workflows.instance.read"]}' ``` !!! note "Body is one variable" The entire body is a single JSON string, so set it as one `…__BODY` variable rather than per-field keys. This keeps arrays and mixed-case keys (e.g. `resourceType`) intact — a `__`-split native map cannot express either. As with [role providers](./configuration.md#role-provider), the two approaches combine: load non-sensitive config from the file and inject secrets (e.g. a service API key in `headers`) via env vars on top. --- # Overview Source: https://docs.lakekeeper.io/docs/latest/api-overview/ # Lakekeeper APIs Lakekeeper is a rust-native Apache Iceberg REST Catalog implementation. It exposes three distinct HTTP APIs. An interactive Swagger-UI for the exact Lakekeeper version and configuration you are running is available at `/swagger-ui/#/` (by default [http://localhost:8181/swagger-ui/#/](http://localhost:8181/swagger-ui/#/)). ## The three APIs * **Iceberg REST Catalog API** (`/catalog/v1/...`) — the standard [Apache Iceberg REST specification](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml). Query engines (Spark, Trino, PyIceberg, ...) speak this API to create namespaces, commit tables, load metadata, and obtain vended storage credentials. You generally do not call it by hand. Reference: [Catalog API](./api/catalog.md). * **Management API** (`/management/v1/...`) — Lakekeeper-specific administration that has no equivalent in the Iceberg spec: bootstrapping the server, creating and configuring Warehouses (storage profiles, credentials, soft-delete), managing Projects, Users, Roles, Tasks, and — when Authorization is enabled — permissions. This is what the Lakekeeper UI and your platform automation use. Reference: [Management API](./api/management.md). * **Data API** (`/lakekeeper/v1/...`) — Lakekeeper's data-plane API for functionality beyond the Iceberg REST spec. Today it serves [Generic Tables](./generic-tables.md) — non-Iceberg formats such as Lance and Delta — including credential vending; it is the home for further data-related functions over time. Reference: [Generic Table API](./api/generic-table.md). ## Management API endpoint groups All Management endpoints are served under `/management/v1/`. The API is grouped as follows (see Swagger-UI for the exact routes and request/response schemas of your running version): | Group | Purpose | |-------|---------| | `server` | Server info, bootstrapping, and global server configuration. | | `project` | Create and manage Projects (the top-level tenant boundary). | | `warehouse` | Create and manage Warehouses: storage profiles, storage credentials, soft-delete/undrop settings, activation status, and format-version policy. | | `tasks` | List, inspect, and control background tasks (e.g. soft-delete expiration and purge). | | `user` | Manage Users provisioned in the catalog. | | `role` | Manage Roles, which are first-class principals that can be granted permissions and assumed. | | `permissions-openfga` | View and manage fine-grained permissions when the OpenFGA authorizer is enabled. | ## Exploring the APIs * **Swagger-UI** (interactive, version-accurate): `/swagger-ui/#/` — by default [http://localhost:8181/swagger-ui/#/](http://localhost:8181/swagger-ui/#/). * **Reference pages**: [Catalog API](./api/catalog.md), [Management API](./api/management.md), [Generic Table API](./api/generic-table.md). * **OpenAPI documents**: served by the running server and used to generate clients. ## Try it locally ```bash git clone https://github.com/lakekeeper/lakekeeper.git cd lakekeeper/examples/minimal docker compose up ``` Then open your browser at [http://localhost:8181/swagger-ui/#/](http://localhost:8181/swagger-ui/#/). --- # Index Source: https://docs.lakekeeper.io/docs/latest/api/

OpenAPI moved to docs/docs/api Folder

--- # Catalog Source: https://docs.lakekeeper.io/docs/latest/api/catalog/ --- # Generic Tables Source: https://docs.lakekeeper.io/docs/latest/api/generic-table/ --- # Management Source: https://docs.lakekeeper.io/docs/latest/api/management-plus/ --- # Management (Core) Source: https://docs.lakekeeper.io/docs/latest/api/management/ --- # Authentication Source: https://docs.lakekeeper.io/docs/latest/authentication/ # Authentication Authentication is crucial for securing access to Lakekeeper. By enabling authentication, you ensure that only authorized users can access and interact with your data. Lakekeeper supports authentication via any OpenID (or OAuth 2) capable identity provider as well as authentication for Kubernetes service accounts, allowing you to integrate with your existing identity providers. Authentication and Authorization are distinct processes in Lakekeeper. Authentication verifies the identity of users, ensuring that only authorized individuals can access the system. This is performed via an Identity Provider (IdP) such as OpenID or Kubernetes. Authorization, on the other hand, determines what authenticated users are allowed to do within the system. Lakekeeper is extendable and can connect to different authorization systems. By default, Lakekeeper uses OpenFGA to manage and evaluate permissions, providing a robust and flexible authorization model. For more details, see the [Authorization guide](./authorization.md). Lakekeeper does not issue API-Keys or Client-Credentials itself. Instead, it relies on external IdPs for authentication, ensuring a secure and centralized management of user identities. This approach minimizes the risk of credential leakage and simplifies the integration with existing security infrastructures. ## OpenID Provider Lakekeeper can be configured to integrate with all common identity providers. For best performance, tokens are validated locally against the server keys (`jwks_uri`). This requires all incoming tokens to be JWT tokens. If you require support for opaque tokens, please upvote the corresponding [GitHub Issue](https://github.com/lakekeeper/lakekeeper/issues/620). If `LAKEKEEPER__OPENID_PROVIDER_URI` is specified, Lakekeeper will verify access tokens against this provider. The provider must provide the `.well-known/openid-configuration` endpoint and the openid-configuration needs to have `jwks_uri` and `issuer` defined. Optionally, if `LAKEKEEPER__OPENID_AUDIENCE` is specified, Lakekeeper validates the `aud` field of the provided token to match the specified value. We recommend to specify the audience in all deployments, so that tokens leaked for other applications in the same IdP cannot be used to access data in Lakekeeper. Users are automatically added to Lakekeeper after successful Authentication (user provides a valid token with the correct issuer and audience). If a User does not yet exist in Lakekeeper's Database, the provided JWT token is parsed. The following fields are parsed: - `name`: `name` or `given_name`/ `first_name` and `family_name`/ `last_name` or `app_displayname` or `preferred_username` - `subject`: `sub` unless `subject_claim` is set, then it will be the value of the claim. - `claims`: all claims - `email`: `email` or `upn` if it contains an `@` or `preferred_username` if it contains an `@` If the `name` cannot be determined because none of the claims are available, the principal is registered under the name `Nameless App with ID `. Lakekeeper determines the ID of users in the following order: 1. If `LAKEKEEPER__OPENID_SUBJECT_CLAIM` is set, this value (or comma-separated list of values) is tried in order and the first claim present in the token is used. Setting only one claim, that claim must be present. 1. If `oid` is present, it is used. The main motivation to prefer the `oid` over the `sub` is that the `sub` field is not unique across applications, while the `oid` is. (See for example [Entra-ID](https://learn.microsoft.com/en-us/entra/identity-platform/id-token-claims-reference)). Lakekeeper needs to the same IDs as query engines in order to share Permissions. 1. If the `sub` field is present, use it, otherwise fail. IDs from the OIDC provider in Lakekeeper have the form `oidc~`. ### Authenticating Machine Users All common iceberg clients and IdPs support the OAuth2 `Client-Credential` flow. The `Client-Credential` flow requires a `Client-ID` and `Client-Secret` that is provided in a secure way to the client. In the following sections we demonstrate for selected IdPs how applications can be setup for machine users to connect. ### Authenticating Humans Human Authentication flows are interactive by nature and are typically performed directly by the IdP. This enables the use of all security options that the IdP supports, including 2FA, hardware keys, single-sign-on and more. The recommended flows for authentication are Authorization Code Flow [RFC6749#section-4.1](https://datatracker.ietf.org/doc/html/rfc6749#section-4.1) with PKCE and Device Code Flow [RFC8628](https://datatracker.ietf.org/doc/html/rfc8628). At the time of writing all common iceberg clients (spark, trino, starrocks, pyiceberg, ...) do not support any authorization flow that is suitable for human users natively. The iceberg community is working on introducing those flows and we started an [initiative](https://docs.google.com/document/d/1buW9PCNoHPeP7Br5_vZRTU-_3TExwLx6bs075gi94xc/edit?tab=t.0) to standardize and document them as part of the iceberg docs. Until iceberg clients are natively ready for human flows, authentication flows have to be performed outside of iceberg clients. To make this process as easy as possible, the Lakekeeper UI offers the option to get a new token for a human user: ![](../../assets/generate-token.png) The lifetime of this token is specified in the corresponding application in your IdP. We recommend to set the lifetime to no longer than one day. ### Keycloak We are creating two Client: The first client with a "public" profile for the Lakekeeper API & UI and the second client for a machine client (e.g. Spark). Repeat step 2 for each machine client that is needed. ##### Client 1: Lakekeeper 1. Create a new "Client": - **Client Type**: choose "OpenID Connect" - **Client ID**: choose any, for this example we choose `lakekeeper` - **Name**: choose any, for this example we choose `Lakekeeper Catalog` - **Client authentication**: Leave "Off". We need a public client. - **Authentication Flows**: Enable "Standard flow", OAuth 2.0 Device Authorization Grant". - **Valid redirect URIs**: For testing a wildcard "*" can be set. Otherwise the URL where the Lakekeeper UI is reachable for the user suffixed by `/callback`. E.g.: `http://localhost:8181/ui/callback`. 1. When the client is created, click on the "Advanced" tab of this client, scroll down to "Advanced settings" and set "Access Token Lifespan" to "Expires in" - 12 Hours. 1. Create a new "Client scope" in the left side menu: - **Name**: choose any, for this example we choose `lakekeeper` - **Description**: `Client of Lakekeeper` - **Type**: Optional 1. When the scope is created, we need to add a new mapper. This is recommended because Lakekeeper can validate the `audience` (target service) of the token for increased security. In order to add the `lakekeeper` audience to the token every time the `lakekeeper` scope is requested, we create a new mapper. Select the "Mappers" tab of the previously created `lakekeeper` scope. Select "Configure a new mapper" -> "Audience". ![](../../assets/keycloak-audience-mapper.png) - **Name**: choose any, for this example we choose `Add lakekeeper Audience` - **Included Client Audience**: Select the id of the previously created App 1. In our example this is `lakekeeper`. - Make sure `Add to access token` and `Add to token introspection` is enabled. 1. Finally, we need to grant the `spark` client permission to use the `lakekeeper` scope which adds the correct audience to the issued token. Select the "Client scopes" tab of the `lakekeeper` client and select "Add client scope". Select the previously created scope, in our example this is `lakekeeper`. We recommend adding the scope as "Default". We are now ready to deploy Lakekeeper and login via the UI. Set the following environment variables / configurations: ```sh LAKEKEEPER__OPENID_PROVIDER_URI=http://localhost:30080/realms/iceberg (URI of the keycloak realm) LAKEKEEPER__OPENID_AUDIENCE=lakekeeper (ID of Client 1) LAKEKEEPER__UI__OPENID_CLIENT_ID="lakekeeper" (ID of Client 1) # LAKEKEEPER__UI__OPENID_SCOPE="lakekeeper" (Name of the created scope, not required if scope was added as default) ``` ##### Client 2: Machine User Repeat this process for each query engine / machine user that is required: 1. Create a new "Client": - **Client Type**: choose "OpenID Connect" - **Client ID**: choose any, for this example we choose `spark`. - **Name**: choose any, for this example we choose `Spark Client accessing Lakekeeper` - **Client authentication**: Turn "On". Leave "Authorization" turned "Off". - **Authentication Flows**: Enable "Service accounts roles" and "Standard Token Exchange". 1. When the client is created, click on "Credentials", choose "Client Authenticator" as "Client Id and Secret". Copy the `Client Secret` for later use. 1. Finally, we need to grant the `spark` client permission to use the `lakekeeper` scope which adds the correct audience to the issued token. Select the "Client scopes" tab of the `spark` client and select "Add client scope". Select the previously created scope, in our example this is `lakekeeper`. We recommend adding the scope as "Optional". By adding an optional scope the client can be re-used for other services, i.e. if Spark needs to access another catalog in the future. That's it! We can now use the second App Registration to sign into Lakekeeper using Spark or other query engines. A Spark configuration would look like: === "PyIceberg" ```python import pyiceberg.catalog import pyiceberg.catalog.rest import pyiceberg.typedef catalog = pyiceberg.catalog.rest.RestCatalog( name="my_catalog_name", uri="http://localhost:8181/catalog", warehouse="", credential=":", scope="lakekeeper", # Name of the created scope **{ "oauth2-server-uri": "http://localhost:30080/realms//protocol/openid-connect/token" }, ) print(catalog.list_namespaces()) ``` === "PySpark" ```python import pyspark conf = { "spark.jars.packages": "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-azure-bundle:1.10.1", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.lakekeeper": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.lakekeeper.type": "rest", "spark.sql.catalog.lakekeeper.uri": "http://localhost:8181/catalog", "spark.sql.catalog.lakekeeper.credential": ":", "spark.sql.catalog.lakekeeper.warehouse": "", "spark.sql.catalog.lakekeeper.scope": "lakekeeper", # Name of the created scope "spark.sql.catalog.lakekeeper.oauth2-server-uri": "http://localhost:30080/realms//protocol/openid-connect/token", } config = pyspark.SparkConf().setMaster("local") for k, v in conf.items(): config = config.set(k, v) spark = pyspark.sql.SparkSession.builder.config(conf=config).getOrCreate() try: spark.sql("USE `lakekeeper`") except Exception as e: print(e.stackTrace) raise e spark.sql("CREATE NAMESPACE IF NOT EXISTS `test`") spark.sql("CREATE OR REPLACE TABLE `test`.`test_tbl` AS SELECT 1 a") ``` If Authorization is enabled, the client will throw an error as no permissions have been granted yet. During this initial connect to the `/config` endpoint of Lakekeeper, the user is automatically provisioned so that it should show up when searching for users in the "Grant" dialog and user search endpoints. ### Entra-ID (Azure) We are creating three App-Registrations: The first for Lakekeeper itself, the second for the Lakekeeper UI the third for a machine client (e.g. Spark) to access Lakekeeper. Repeat step 3 for each machine client that is needed. While App-Registrations can also be shared, the recommended setup we propose here offers more flexibility and better security. ##### App 1: Lakekeeper UI Application 1. Create a new "App Registration" - **Name**: choose any, for this example we choose `Lakekeeper-UI` - **Redirect URI**: Add the URL where the Lakekeeper UI is reachable for the user suffixed by `/callback`. E.g.: `http://localhost:8181/ui/callback`. If asked, select type "Single Page Application (SPA)". 1. In the "Overview" page of the "App Registration" note down the `Application (client) ID`. Also note the `Directory (tenant) ID`. 1. Finally we recommend to set a policy for tokens to expire in 12 hours instead of the default ~1 hour. Please follow the [Microsoft Tutorial](https://learn.microsoft.com/en-us/entra/identity-platform/configure-token-lifetimes#create-a-policy-and-assign-it-to-an-app) to assign a corresponding policy to the Application. (If you find a good way to do this via the UI, please let us know so that we can update this documentation page!) Alternatively, the following snippets will setup the resources mentioned above: === "Terraform" ```terraform resource "azuread_application_registration" "lakekeeper_ui" { display_name = "Lakekeeper UI" } resource "azuread_application_redirect_uris" "lakekeeper_ui" { application_id = azuread_application_registration.lakekeeper_ui.id type = "SPA" redirect_uris = [ ] } resource "azuread_service_principal" "lakekeeper_ui" { client_id = azuread_application_registration.lakekeeper_ui.client_id feature_tags { enterprise = true } } ``` ##### App 2: Lakekeeper Application 1. Create a new "App Registration" - **Name**: choose any, for this example we choose `Lakekeeper` - **Redirect URI**: Leave empty. 1. When the App Registration is created, select "Manage" -> "Expose an API" and on the top select "Add" beside `Application ID URI`. ![](../../assets/idp-azure-application-id-uri.png) Note down the `Application ID URI` (should be `api://`). 1. Still in the "Expose an API" menus, select "Add a Scope". Fill the fields as follows: - **Scope name**: lakekeeper - **Who can consent?** Admins and users - **Admin consent display name**: Lakekeeper API - **Admin consent description**: Access Lakekeeper API - **State**: Enabled 1. After the `lakekeeper` scope is created, click "Add a client application" under the "Authorized client applications" headline. Select the previously created scope and paste as `Client ID` the previously noted ID from App 1. 1. In the "Overview" page of the "App Registration" note down the `Application (client) ID`. Alternatively, the following snippets will setup the resources mentioned above: === "Terraform" ```terraform resource "random_uuid" "lakekeeper_scope" {} resource "azuread_application" "lakekeeper" { display_name = "Lakekeeper" owners = [data.azuread_client_config.current.object_id] api { mapped_claims_enabled = true requested_access_token_version = 2 known_client_applications = [ azuread_application_registration.lakekeeper_ui.client_id ] oauth2_permission_scope { id = random_uuid.lakekeeper_scope.id value = "lakekeeper" enabled = true type = "User" admin_consent_description = "Lakekeeper API" admin_consent_display_name = "Access Lakekeeper API" user_consent_description = "Lakekeeper API" user_consent_display_name = "Access Lakekeeper API" } } lifecycle { ignore_changes = [ identifier_uris, ] } } resource "azuread_application_identifier_uri" "lakekeeper" { application_id = azuread_application.lakekeeper.id identifier_uri = "api://${azuread_application.lakekeeper.client_id}" } resource "azuread_service_principal" "lakekeeper_client" { client_id = azuread_application.lakekeeper.client_id feature_tags { enterprise = true } } resource "azuread_application_pre_authorized" "lakekeeper" { application_id = azuread_application.lakekeeper.id authorized_client_id = azuread_application_registration.lakekeeper_ui.client_id permission_ids = [ random_uuid.lakekeeper_scope.id ] } ``` We are now ready to deploy Lakekeeper and login via the UI. Set the following environment variables / configurations: === "bash" ```sh // Note the v2.0 at the End of the provider URI! LAKEKEEPER__OPENID_PROVIDER_URI=https://login.microsoftonline.com//v2.0 LAKEKEEPER__OPENID_AUDIENCE="api://" LAKEKEEPER__UI__OPENID_CLIENT_ID="" LAKEKEEPER__UI__OPENID_SCOPE="openid profile api:///lakekeeper" LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS="https://sts.windows.net//" ``` !!! info "Why is `OPENID_ADDITIONAL_ISSUERS` needed?" Entra ID issues **v1** or **v2** access tokens depending on the `accessTokenAcceptedVersion` property in the **resource API's** app manifest (App 2). The token version determines the `iss` (issuer) claim in the JWT: | Token Version | `accessTokenAcceptedVersion` | Issuer (`iss`) claim | |---|---|---| | v1 | `null` (default) or `1` | `https://sts.windows.net//` | | v2 | `2` | `https://login.microsoftonline.com//v2.0` | Even when using the v2.0 provider URI, Entra may still issue **v1 tokens** with the `sts.windows.net` issuer if `accessTokenAcceptedVersion` is not explicitly set to `2` in App 2's manifest. This is a common source of authentication failures. **Recommended:** Set `accessTokenAcceptedVersion` to `2` in App 2's manifest (or use `requested_access_token_version = 2` in Terraform as shown above). The `OPENID_ADDITIONAL_ISSUERS` setting is provided as a fallback to accept v1 tokens without changing your app registration. === "Terraform" ```terraform output "LAKEKEEPER__OPENID_PROVIDER_URI" { value = "https://login.microsoftonline.com/${azuread_service_principal.lakekeeper.application_tenant_id}/v2.0" } output "LAKEKEEPER__OPENID_AUDIENCE" { value = azuread_application.lakekeeper.client_id } output "LAKEKEEPER__UI__OPENID_CLIENT_ID" { value = azuread_application_registration.lakekeeper_ui.client_id } output "LAKEKEEPER__UI__OPENID_SCOPE" { value = "openid profile api://${azuread_application.lakekeeper.client_id}/lakekeeper" } output "LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS" { value = "https://sts.windows.net/${azuread_service_principal.lakekeeper.application_tenant_id}" } ``` Before continuing with App 2, we recommend to create a Warehouse using any of the supported storages. Please check the [Storage Documentation](./storage.md) for more information. Without a Warehouse, we won't be able to test App 3. ##### App 3: Machine User Repeat this process for each query engine / machine user that is required: 1. Create a new "App Registration" - **Name**: choose any, for this example we choose `Spark` - **Redirect URI**: Leave empty - we are going to use the Client Credential Flow 2. When the App Registration is created, select "Manage" -> "Certificates & secrets" and create a "New client secret". Note down the secrets "Value". 3. There might be an additional step needed before you can utilize the machine user. First, get the token for it using the credentials you created on previous steps: ``` curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \ https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token \ -d 'client_id={client_id}' \ -d 'grant_type=client_credentials' \ -d 'scope=email openid {APP2_client_id}%2F.default' \ -d 'client_secret={client_secret}' ``` Note that `scope` parameter might not accept `api://` prefix for the APP2 scope for some Entra tenants. In that case, simply use `app2_client_id/.default` as shown above. Copy the `access_token` from the response and decode it using jwt.io or any other JWT decode tool. In order for automatic registration to work, token must contain the following claims: - `app_displayname`: name of the APP3 assigned in step 1 - `appid`: application identifier (client identifier) of the App 3 - `idtyp`: "app" (indicates this is an Entra service principal) For some Entra installations you might not get any of those claims in the JWT. `idtyp` can be added via optional claims in the App Registration of the previously created "App 2". Add them to `access_token` of **App 2** and set `name` to `idtyp` and `essential` to `true`. Alternatively, the following snippets will setup the resources mentioned above: === "Terraform" ```terraform resource "azuread_application_registration" "my_lakekeeper_machine_user" { display_name = "My Lakekeeper Machine User" } resource "azuread_service_principal" "my_lakekeeper_machine_user" { client_id = azuread_application_registration.my_lakekeeper_machine_user.client_id } resource "azuread_application_password" "my_lakekeeper_machine_user" { application_id = azuread_application_registration.my_lakekeeper_machine_user.id } ``` That's it! We can now use the third App Registration to sign into Lakekeeper using Spark or other query engines. A Spark configuration would look like: === "PyIceberg" ```python import pyiceberg.catalog import pyiceberg.catalog.rest import pyiceberg.typedef catalog = pyiceberg.catalog.rest.RestCatalog( name="my_catalog_name", uri="http://localhost:8181/catalog", warehouse="", credential=":", scope="email openid api:///.default", **{ "oauth2-server-uri": "https://login.microsoftonline.com//oauth2/v2.0/token" }, ) print(catalog.list_namespaces()) ``` === "PySpark" ```python import pyspark conf = { "spark.jars.packages": "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-azure-bundle:1.10.1", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.azure-docs": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.azure-docs.type": "rest", "spark.sql.catalog.azure-docs.uri": "http://localhost:8181/catalog", "spark.sql.catalog.azure-docs.credential": ":", "spark.sql.catalog.azure-docs.warehouse": "", "spark.sql.catalog.azure-docs.scope": "email openid api:///.default", "spark.sql.catalog.azure-docs.oauth2-server-uri": "https://login.microsoftonline.com//oauth2/v2.0/token", } config = pyspark.SparkConf().setMaster("local") for k, v in conf.items(): config = config.set(k, v) spark = pyspark.sql.SparkSession.builder.config(conf=config).getOrCreate() try: spark.sql("USE `azure-docs`") except Exception as e: print(e.stackTrace) raise e spark.sql("CREATE NAMESPACE IF NOT EXISTS `test`") spark.sql("CREATE OR REPLACE TABLE `test`.`test_tbl` AS SELECT 1 a") ``` If Authorization is enabled, the client will throw an error as no permissions have been granted yet. During this initial connect to the `/config` endpoint of Lakekeeper, the user is automatically provisioned so that it should show up when searching for users in the "Grant" dialog and user search endpoints. While we try to extract the name of the application from its token, this might not be possible in all setups. As a fallback we use the `Client ID` as the name of the user. Once permissions have been granted, the user is able to perform actions. ### Google Identity Platform !!! warning At the time of writing (June 2025), Google Identity Platform lacks support for the standard OAuth2 Client Credentials Flow, which was established by the IETF in 2012 (!) specifically for machine-to-machine authentication. While the guide below explains how to secure Lakekeeper using Google Identity Platform, this solution only works for human users due to this limitation. For machine authentication, you would need to obtain access tokens through alternative methods outside of the Iceberg client ecosystem and provide them directly to your clients. However, such approaches fall outside the scope of this documentation. To see if google cloud supports client credentials in the meantime, check [Google's `.well-known/openid-configuration`](https://accounts.google.com/.well-known/openid-configuration), and search for `client_credentials` in the `grant_types_supported` section. When using Lakekeeper with multiple IdPs (i.e. Google & Kubernetes), the second IdP can still be used to authenticate Machines. Fist, read the warning box above (!). Additionally as of June 2025, the Google Identity Platform also does not support standard OAuth2 login flows for "public clients" such as Lakekeeper's Web-UI as part of the desired "Web Application" client type. Instead, [Google still promotes](https://developers.google.com/identity/protocols/oauth2/javascript-implicit-flow) the *OAuth Implicit Flow* instead of the much more secure *Authorization Code Flow with PKCE* for public clients. Using the implicit flow is [discouraged](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-04) by the IETF. As we don't want to lower our security or switch to legacy flows, we are using a workaround to register the Lakekeeper UI as a Native Application (Universal Windows Platform in this example), which allows the use of the proper flows, even though it is intended for a different purpose. If you're using Google Cloud Platform, please advocate for proper OAuth standard support by: 1. Reporting this concern to your Google sales representative 2. Upvoting these issues: [912693](https://www.googlecloudcommunity.com/gc/Developer-Tools/Authorization-Code-Flow-without-client-secret/m-p/912693#M3618), [33416](https://www.googlecloudcommunity.com/gc/Apigee/Supporting-OAuth-quot-Public-quot-clients-in-authorization-code/m-p/33416) 3. Sharing these discussions: [StackOverflow](https://stackoverflow.com/questions/60724690/using-google-oidc-with-code-flow-and-pkce) and [GitHub issue](https://github.com/authts/oidc-client-ts/issues/152) Due to these OAuth2 limitations in Google Identity Platform, we cannot recommend it for production deployments. Nevertheless, if you wish to proceed, here's how: ##### Google Auth Platform Project: Lakekeeper Application Create a new GCP Project - each Project serves a single application as part of the "Google Auth Platform". When the new project is created, create the new internal Lakekeeper Application: 1. Search for "Google Auth Platform", then select "Branding" on the left. 1. Select "Get started" or modify the pre-filled form: - **App Name**: Select a good Name, for example `Lakekeeper` - **User support email**: This is shown to users later - select a team e-mail address. - **Audience**: Internal (Otherwise people outside of your organization can login too) - **Contact Information / Email address**: Email Addresses of Lakekeeper Admins or Team Email Address 1. After the Branding is created, select "Data access" in the left menu, and add the following non-sensitive scopes: `.../auth/userinfo.email`, `.../auth/userinfo.profile`, `openid` ##### Client 1: Lakekeeper UI 1. After the app is created, click in the left menu on "Clients" in the "Google Auth Platform" service 1. Click on "+Create credentials" 1. Select "Universal Windows Platform (UWP)" due to the lack of support for public clients in the more appropriate "Web Application" type described above. Enter any randomly generated number in the "Store ID" field and give the Application a good name, such as `Lakekeeper UI`. Then click "Create". Note the `Client ID`. We are now ready to deploy Lakekeeper and login via the UI. Set the following environment variables / configurations: === "bash" ```sh LAKEKEEPER__OPENID_PROVIDER_URI=https://accounts.google.com LAKEKEEPER__OPENID_AUDIENCE="" LAKEKEEPER__UI__OPENID_CLIENT_ID="" LAKEKEEPER__UI__OPENID_SCOPE="openid profile" LAKEKEEPER__UI__OPENID_TOKEN_TYPE="id_token" ``` We are now able to login and bootstrap Lakekeeper. ## Multiple OIDC Providers For scenarios where you need to authenticate tokens from multiple identity providers simultaneously—such as Okta for human users and EKS OIDC for Kubernetes service accounts—Lakekeeper supports configuring multiple OIDC providers. When multiple providers are configured, each provider fetches its own JWKS keys independently. Incoming tokens are checked against each provider in order until one successfully validates the token. ### Configuration Configure each provider under `LAKEKEEPER__OPENID_PROVIDERS____`. These providers are added in addition to the single-provider `LAKEKEEPER__OPENID_PROVIDER_URI`, which remains the primary provider (`idp_id = "oidc"`). ```bash LAKEKEEPER__OPENID_PROVIDERS__OKTA__URI=https://company.okta.com LAKEKEEPER__OPENID_PROVIDERS__OKTA__AUDIENCE=https://company.okta.com LAKEKEEPER__OPENID_PROVIDERS__OKTA__SUBJECT_CLAIMS=sub LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERA__URI=https://oidc.eks.us-east-1.amazonaws.com/id/ABC123DEF456 LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERA__AUDIENCE=sts.amazonaws.com LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERA__SUBJECT_CLAIMS=sub LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERB__URI=https://oidc.eks.us-east-1.amazonaws.com/id/XYZ789GHI012 LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERB__AUDIENCE=sts.amazonaws.com LAKEKEEPER__OPENID_PROVIDERS__EKSCLUSTERB__SUBJECT_CLAIMS=sub ``` ### User Identity Format User IDs include the provider's `IDP_ID` as a prefix: `{idp_id}~{subject}`. For example: - `okta~user@example.com` for a user from Okta - `eksclustera~system:serviceaccount:namespace:my-app` for a Kubernetes service account This allows you to distinguish users from different identity providers when granting permissions. ### UI Login The Lakekeeper UI can redirect to only one OIDC provider for SSO — the primary provider configured via `LAKEKEEPER__OPENID_PROVIDER_URI`. Providers added under `LAKEKEEPER__OPENID_PROVIDERS` are used for API token validation only. If `LAKEKEEPER__OPENID_PROVIDER_URI` is not set, the UI login button is disabled by design; clients must obtain tokens out-of-band and call the API directly. ### Resilient Initialization By default, Lakekeeper refuses to start if a configured provider's OIDC/JWKS configuration cannot be loaded. Set `LAKEKEEPER__OPENID_PROVIDERS____REQUIRE_CONNECTED_ON_STARTUP=false` for providers that should be skipped while Lakekeeper continues starting with the remaining authenticators. The primary provider configured via `LAKEKEEPER__OPENID_PROVIDER_URI` always requires a successful connection on startup. ### When to Use Multiple Providers Common use cases include: - **Okta/Entra + EKS OIDC**: Human users authenticate via corporate IdP, while Kubernetes workloads use EKS OIDC tokens - **Multi-cluster Kubernetes**: Different EKS/GKE clusters each have their own OIDC provider - **Migration scenarios**: Gradually migrating from one IdP to another while both remain active See the [Configuration Reference](./configuration.md#multiple-oidc-providers) for the full list of available options per provider. **Identity continuity note:** Existing user IDs are formatted as `oidc~` from the primary provider configured via `LAKEKEEPER__OPENID_PROVIDER_URI`. Do not move that provider into `LAKEKEEPER__OPENID_PROVIDERS` under a different `IDP_ID` (e.g., `okta`), or existing role/user assignments will no longer match. ## Kubernetes If `LAKEKEEPER__ENABLE_KUBERNETES_AUTHENTICATION` is set to true, Lakekeeper validates incoming tokens against the default kubernetes context of the system. Lakekeeper uses the [`TokenReview`](https://kubernetes.io/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) to determine the validity of a token. By default the `TokenReview` resource is protected. When deploying Lakekeeper on Kubernetes, make sure to grant the `system:auth-delegator` Cluster Role to the service account used by Lakekeeper: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: allow-token-review roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: namespace: ``` The [Lakekeeper Helm Chart](https://github.com/lakekeeper/lakekeeper-charts/tree/main/charts/lakekeeper) creates the required binding by default. Applications running in Kubernetes pods can now authenticate using the service account token, which is typically mounted at `/var/run/secrets/kubernetes.io/serviceaccount/token`. Simply read this token and include it in the `Authorization` header. **Example with CURL:** ```bash curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ http://my-lakekeeper:8181/catalog/v1/config ``` **Example with Spark:** ```bash spark-submit \ --conf spark.sql.catalog.lakekeeper.token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ --conf spark.sql.catalog.lakekeeper.uri="http://my-lakekeeper:8181/catalog" \ my-spark-job.py ``` User identities appear in Lakekeeper as `k8s~~`. --- # Authorization (Cedar) Source: https://docs.lakekeeper.io/docs/latest/authorization-cedar/ # Authorization with Cedar {#authorization-with-cedar} !!! important "Using the Correct Cedar Schema Version" Always use the Cedar schema version that exactly matches your Lakekeeper deployment when developing policies. Schema mismatches can cause policy validation failures or unexpected authorization behavior. Download the schema from the Lakekeeper UI (Lakekeeper Plus 0.11.2+) or retrieve it via the `/management/v1/permissions/cedar/schema` endpoint. :material-download: Download Cedar Schema [Cedar](https://docs.cedarpolicy.com/) is an enterprise-grade, policy-based authorization system built into Lakekeeper that requires no external services. Cedar uses a declarative policy language to define access controls, making it ideal for organizations that prefer infrastructure-as-code approaches to authorization management. Check the [Authorization Configuration](./configuration.md#authorization) for configuration options. ## How it Works Lakekeeper uses the built-in Cedar Authorizer to evaluate whether a request is allowed. Each Cedar authorization request consists of three components: 1. **Principal**: The entity performing the request. Example: `Lakekeeper::User::"oidc~peter"` ("oidc~" prefix indicates users from the OIDC identity provider) 1. **Action**: The operation being performed. Example: `Lakekeeper::Action::"CommitTable"` 1. **Resource**: The target of the action. Example: `transactions` table in namespace `finance` (`Lakekeeper::Table::/`) To evaluate authorization requests, Cedar requires the following information: 1. **Policies**: Define which principals can perform which actions on which resources. Policies are provided via files (`LAKEKEEPER__CEDAR__POLICY_SOURCES__LOCAL_FILES`) or Kubernetes ConfigMaps (`LAKEKEEPER__CEDAR__POLICY_SOURCES__K8S_CM`). See [Policy Examples](#policy-examples) below. 1. **Entities**: Application data Cedar uses to make authorization decisions, such as tables (including name, ID, warehouse, namespace, properties, etc.). Lakekeeper automatically provides all required entities (Tables, Generic Tables, Namespaces, Warehouses, etc.) for each decision. User roles are also included if present in the user's token and `LAKEKEEPER__OPENID_ROLES_CLAIM` is configured. For scenarios where role information isn't available in tokens, you can provide external entities—see [External Entity Management](#external-entity-management). 1. **Context**: Transient request-specific data related to an action. For example, the `table_properties_updates` field is available when checking `Lakekeeper::Action::"CommitTable"`. Context is handled internally by Lakekeeper and requires no configuration. 1. **Schema**: Defines entity types recognized by the application. Lakekeeper uses a built-in schema (downloadable above) that can be customized via `LAKEKEEPER__CEDAR__SCHEMA_*` environment variables. We recommend schema customization only for advanced use cases. Most deployments only need to configure `LAKEKEEPER__CEDAR__POLICY_SOURCES__*` and optionally `LAKEKEEPER__OPENID_ROLES_CLAIM` if role information is available in user tokens. Generic (non-Iceberg) tables are a first-class resource in Cedar too: they have their own `Lakekeeper::GenericTable` entity and a parallel set of action groups — `GenericTableActions`, `GenericTableDescribeActions`, `GenericTableSelectActions` and `GenericTableModifyActions` — that mirror the regular `Table` actions. Use them in policies exactly as you would the `Table` equivalents. ## RBAC and ABAC Support Cedar supports both Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC). RBAC grants permissions based on `Lakekeeper::Role` entities, while ABAC uses resource attributes — such as Table, View, and Namespace properties — for authorization decisions. See the ABAC examples in [Policy Examples](#policy-examples) below for more information. ## Token-Based Role Matching with `project_roles` Every `Lakekeeper::User` entity carries a `project_roles` attribute — a flat set of records that represents the role memberships relevant to the project being accessed: ``` principal.project_roles → Set<{provider_id: String, source_id: String}> ``` Lakekeeper populates this set automatically from the user's token (when `LAKEKEEPER__OPENID_ROLES_CLAIM` is configured) for the project context of the current request. In external entity mode (`EXTERNALLY_MANAGED_USER_AND_ROLES=true`) you populate it yourself in the entity JSON file. The `Lakekeeper::User` entity also carries `provider_id` and `source_id` attributes identifying the user's own authentication provider and their ID within it: | Attribute | Example value | Description | |--------------------------------|------------------------------------------------|-----| | `provider_id` | `"oidc"` | Authentication provider of the user | | `source_id` | `"2f268e8b-8cc1-4edd-a9df-87d69f7e9deb"` | User's ID within the provider | | `project_roles` | `[{provider_id: "oidc", source_id: "admins"}]` | Provider-resolved role memberships as `{provider_id, source_id}` records. Includes roles from token claims and role providers (e.g. LDAP) relevant to the current project. | | `global_role_ids` | `["admins", "developers"]` | `source_id` of every provider-resolved role as a plain `Set`. Only populated when `LAKEKEEPER__CEDAR__GLOBAL_ROLE_IDS_ENABLED=true`. See below. | The `Lakekeeper::User` entity also exposes an optional `email` attribute extracted from the authentication token. Email uniqueness is not enforced — two distinct users may share an email. ### When to use `project_roles` vs `global_role_ids` vs `principal in Role::...` | Scenario | Recommended approach | |------------------------------------------------------------------|-----------| | Roles come from OIDC/token claims or a role provider (e.g. LDAP) | `principal.project_roles.contains({provider_id: "oidc", source_id: "my-group"})` | | Role `source_id` values are globally unique across all providers | `principal.global_role_ids.contains("my-group")` *(requires `GLOBAL_ROLE_IDS_ENABLED`)* | | Roles are managed in Lakekeeper (via the management API) | `principal in Lakekeeper::Role::"/oidc~my-role"` | | Roles come from an external entities file | Either approach works; `project_roles` is simpler | `project_roles` simplifies policies especially in single-project setups: to use `principal in Lakekeeper::Role::...` you need to know the project ID, which is an identifier that is inconvenient to embed in policy files. `project_roles` lets you match by provider and role name alone, with no project ID required. `global_role_ids` further simplifies policies when all configured role providers use globally unique `source_id` values (e.g. a single LDAP server or OIDC provider where group names are unique). Enable it with `LAKEKEEPER__CEDAR__GLOBAL_ROLE_IDS_ENABLED=true`; when disabled the attribute is always an empty set. ### Policy example ```cedar // Grant namespace/table/view access to users whose token contains the // "warehouse-1-admins" group from the OIDC provider. permit ( principal is Lakekeeper::User, action in [Lakekeeper::Action::"NamespaceActions", Lakekeeper::Action::"TableActions", Lakekeeper::Action::"ViewActions"], resource ) when { resource.warehouse.name == "wh-1" && principal.project_roles.contains( {provider_id: "oidc", source_id: "warehouse-1-admins"} ) }; ``` !!! note `project_roles` and `global_role_ids` are only populated when the request has a project context (i.e. for warehouse, namespace, table, and view operations). Both are empty sets for server-level actions that span multiple projects, so policies using either attribute will always deny server-level actions. Use the full Role ID or grant direct access to users for server-level policies. !!! tip "Monitoring role providers" Role provider availability is tracked via Prometheus metrics (`lakekeeper_role_provider_up`, `lakekeeper_role_provider_get_roles_duration_seconds`), emitted per `provider_id` for providers with an external backend such as LDAP. The built-in OIDC token provider does no external lookup, so it reports neither — with `persist_token_roles` it surfaces only through `lakekeeper_role_provider_sync_errors_total` on a failed catalog write. Lakekeeper deliberately excludes role provider health from the pod liveness probe — an unreachable provider causes graceful fallback to cached roles from Postgres rather than a pod restart. See [Monitoring — Role Provider Metrics](./monitoring.md#role-provider-metrics) for details and alerting guidance. !!! tip "Debugging role assignments" To see which roles are resolved for each user, temporarily set `LAKEKEEPER__ROLE_PROVIDER_CHAIN__LOG_ROLE_ASSIGNMENTS=true`. This emits an audit event listing every resolved role name after each request. The event is noisy and contains PII — disable it after debugging. See [Logging — Operational Audit Events](./logging.md) for the event schema and example output. ### Policy example — `global_role_ids` Use this simpler form when all your role providers are server-wide and use unique group names (e.g. a single LDAP directory). Requires `LAKEKEEPER__CEDAR__GLOBAL_ROLE_IDS_ENABLED=true`. ```cedar // Grant access to users who are members of the "data-engineers" group, // regardless of which provider that group came from. permit ( principal is Lakekeeper::User, action in [Lakekeeper::Action::"NamespaceActions", Lakekeeper::Action::"TableActions", Lakekeeper::Action::"ViewActions"], resource ) when { resource.warehouse.name == "my-warehouse" && principal.global_role_ids.contains("data-engineers") }; ``` ### Property-based `global_role_ids` matching When `GLOBAL_ROLE_IDS_ENABLED` is set, both `User` and `ResourcePropertyValue` expose `global_role_ids` as plain `Set`. This enables provider-agnostic property-based access control — no need to align provider prefixes between the user's roles and the property tag references: ```cedar // Grant read access when the user shares any role with the table's access_read tag. // Works regardless of whether roles come from OIDC, LDAP, or any other provider. permit ( principal is Lakekeeper::User, action in [Lakekeeper::Action::"TableSelectActions"], resource is Lakekeeper::Table ) when { resource.properties.hasTag("access_read") && resource.properties.getTag("access_read").global_role_ids.containsAny(principal.global_role_ids) }; ``` ## Property-Based Access Control Lakekeeper can parse roles and users directly from Table, Namespace, and View properties. This enables a powerful ABAC pattern where access control lists are stored as resource metadata, and Cedar policies grant access based on those lists — without maintaining a separate role-assignment file. ### How Properties Are Exposed to Cedar Every Table, Namespace, and View entity carries a `properties` attribute of type `ResourceProperties`. This is a Cedar entity with typed tags — one per property key — each holding a `ResourcePropertyValue` record: ``` type ResourcePropertyValue = { raw: String, // original value as stored roles: Set, // parsed Lakekeeper::Role entity references users: Set, // parsed Lakekeeper::User entity references global_role_ids: Set, // source_id of each parsed role (requires GLOBAL_ROLE_IDS_ENABLED) } ``` Properties are ordinary Iceberg table/namespace properties — you set them with the same tools you already use. For example, using Spark SQL: ```sql -- Set access-control properties when creating a table CREATE TABLE my_catalog.finance.transactions ( id BIGINT, amount DOUBLE, ts TIMESTAMP ) USING iceberg TBLPROPERTIES ( 'access-owners' = '["role-full:oidc~data-admins", "user:oidc~alice@example.com"]', 'access-readers' = '["role:analysts"]' ); -- Or add/update them on an existing table ALTER TABLE my_catalog.finance.transactions SET TBLPROPERTIES ( 'access-readers' = '["role:analysts", "role-full:oidc~reporting-team"]' ); -- Namespace properties work the same way ALTER NAMESPACE my_catalog.finance SET PROPERTIES ( 'access-readers' = '["role-full:oidc~finance-readers"]' ); ``` Keys that start with a configured parse prefix (default: `access-`, `access_`) are automatically parsed into `roles` and `users` sets. All other keys (e.g. `write.metadata.metrics.default-mode`) pass through as plain strings in `.raw` with empty `roles` and `users`. In a Cedar policy, properties are accessed using Cedar's tag syntax: ```cedar // Check if a property key exists resource.properties.hasTag("access-owners") // Read the raw string value resource.properties.getTag("access-owners").raw // Check whether the requesting principal is in the allowed roles principal in resource.properties.getTag("access-owners").roles // Check whether the requesting principal is explicitly listed as an allowed user principal in resource.properties.getTag("access-owners").users // Check either roles or users principal in resource.properties.getTag("access-owners").roles || principal in resource.properties.getTag("access-owners").users ``` The `principal in ` check leverages Cedar's entity hierarchy: a user is considered `in` a role if that role appears anywhere in the user's ancestry chain (as established by OIDC token claims or external entity definitions). ### Access-Control Property Keys Properties whose key starts with one of the configured **parse prefixes** are treated as **access-control properties**. The default prefixes are `access-` and `access_`; they can be changed or disabled entirely with `LAKEKEEPER__CEDAR__PROPERTY_PARSE_PREFIXES` (see [Configuration](#configuration) below). Access-control property values must be a JSON array of typed entity references: | Format | Description | |-------------------------------------------------|----------------------------| | `role:` | Short form — uses the default role provider. | | `role-full:~` | Full form — provider name is explicit. Works with any configured role or identity provider. | | `role-full:/~` | Full form with an explicit project scope. Useful in multi-project setups when referencing a role from a different project. | | `user:` | References a specific user by their identity-provider ID (e.g. `user:oidc~alice@example.com`). | The default provider for the `role:` short form is determined as follows: if a role provider (e.g. LDAP) is configured, its provider ID is used; otherwise, if exactly one identity provider (e.g. OIDC) is registered, it becomes the default. When there are multiple providers and no single default can be determined, you must use the `role-full:` form. The entire property value is a **JSON-encoded string** containing an array of these references. For example: ``` '["role:analysts", "role-full:oidc~data-admins", "user:oidc~alice@example.com"]' ``` A property with a single entry is still a JSON array, and an empty array (`'[]'`) is valid — it effectively grants access to nobody via that property. ### Configuration | Environment variable | Default | Description | |-----------------------------------------------------------|--------------------------|-----| | LAKEKEEPER__CEDAR__
PROPERTY_PARSE_PREFIXES | `["access_", "access-"]` | List of property key prefixes that trigger entity-reference parsing. Set to `[]` to disable parsing entirely. | ### Error Handling | Path | Behavior | |--------------------------------------------------------------|---------------| | **Read** (AuthZ checks for read/describe operations) | Parse errors in access-prefixed properties are logged as warnings. The property is still visible in Cedar with `raw` set to the original value and empty `roles`/`users` sets. Authorization is not blocked. | | **Write** (AuthZ checks for create/update/commit operations) | Parse errors in access-prefixed properties cause the request to be **rejected with HTTP 400**. This prevents malformed access-control data from ever being stored. | !!! tip Because malformed access-control values are rejected on write, you can rely on the `roles`/`users` sets being accurate and complete during read-path authorization. ## User Identity Derivations User derivations let you extract parts of a user's identity (`source_id` or `provider_id`) using regex named capture groups, and expose them as Cedar tags on a `UserDerivedAttributes` sub-entity. This enables policies that match users to resources based on identity patterns — for example, granting a user full access to namespaces that match their username. ### How It Works Each derivation rule specifies: - **`source`**: which identity field to match against — `source_id` (the user's subject in the IdP) or `provider_id` (e.g. `oidc`, `kubernetes`) - **`pattern`**: a regex with [named capture groups](https://docs.rs/regex/latest/regex/#grouping-and-flags) (`(?...)`) - **`transform`** *(optional)*: a transformation applied to every captured value before it becomes a tag — `none` (default), `lowercase`, or `uppercase` Every named group that matches a non-empty substring becomes a string tag on the `UserDerivedAttributes` entity. Empty captures are silently skipped. Because Cedar has no built-in case-insensitive string comparison or `toLowerCase()` function, use `transform = "lowercase"` to normalize captured values so that policies can compare them against known-case literals. If different capture groups need different transforms, define separate derivation entries with distinct regexes. ### Configuration Derivations are configured as a map under `LAKEKEEPER__CEDAR__USER_DERIVATIONS`. Each key is a human-readable name (used in error messages), and the value specifies `source` and `pattern`. **Environment variables:** ```sh # Extract "username" and "domain" from source_id (e.g. "Alice@Example.COM"), # lowercased so policies can compare against known-case literals. LAKEKEEPER__CEDAR__USER_DERIVATIONS__EMAIL_PARTS__SOURCE=source_id LAKEKEEPER__CEDAR__USER_DERIVATIONS__EMAIL_PARTS__PATTERN=^(?[^@]+)@(?.+)$ LAKEKEEPER__CEDAR__USER_DERIVATIONS__EMAIL_PARTS__TRANSFORM=lowercase # Extract Kubernetes service account parts from source_id (no transform needed) LAKEKEEPER__CEDAR__USER_DERIVATIONS__K8S_SA__SOURCE=source_id LAKEKEEPER__CEDAR__USER_DERIVATIONS__K8S_SA__PATTERN=^system:serviceaccount:(?[^:]+):(?.+)$ ``` **TOML (file-based config):** ```toml [cedar.user_derivations.email_parts] source = "source_id" pattern = "^(?[^@]+)@(?.+)$" transform = "lowercase" # "none" (default), "lowercase", "uppercase" [cedar.user_derivations.k8s_sa] source = "source_id" pattern = "^system:serviceaccount:(?[^:]+):(?.+)$" ``` Regex patterns are compiled once at startup. Invalid patterns cause a startup error with a clear message including the derivation name. ### Accessing Derived Attributes in Policies Derived attributes are stored on a `UserDerivedAttributes` entity linked from the `User` via the optional `derived_attributes` field. Access tags using Cedar's `hasTag()` and `getTag()` functions: ```cedar // Guard with `has` since derived_attributes is optional principal has derived_attributes && principal.derived_attributes.hasTag("username") && principal.derived_attributes.getTag("username") ``` ### Policy Examples **Grant users full access to their personal namespace in the `dev` warehouse:** If users authenticate with an OIDC provider where `source_id` is an email (e.g. `Alice@Example.COM`), and you configure a derivation with `transform = "lowercase"` to extract `username`, this policy lets each user perform any action on the namespace resource itself (e.g. list tables, create tables) — but only within the `dev` warehouse. The `lowercase` transform ensures the comparison works regardless of the casing in the IdP's subject claim. It does not automatically grant access to tables, views, or child namespaces within it; those require separate policies: ```cedar permit( principal is Lakekeeper::User, action, resource is Lakekeeper::Namespace ) when { resource.warehouse.name == "dev" && principal has derived_attributes && principal.derived_attributes.hasTag("username") && resource.name == principal.derived_attributes.getTag("username") }; ``` This allows user `alice@example.com` to perform any action on namespace `alice` in warehouse `dev`. ## Entity Hierarchy and Context For each authorization request, Lakekeeper provides Cedar with the complete entity hierarchy from the requested resource to the server root. This hierarchical context ensures policies have full visibility into the resource's location and relationships. **Example**: When a user queries table `ns1.ns2.transactions` in warehouse `wh-1` within project `my-project`, Cedar sees the following entities: - `Lakekeeper::Server::` (root) - `Lakekeeper::Project::""` - `Lakekeeper::Warehouse::""` (parent: Project) - `Lakekeeper::Namespace::""` (parent: Warehouse) - `Lakekeeper::Namespace::""` (parent: ns1) - `Lakekeeper::Table::""` (parent: ns2) This hierarchy allows policies to reference any level in the path — you can grant access based on warehouse names, namespace hierarchies, or specific table properties. ## Entity ID Formats The following table documents the ID format used for each Cedar entity type. These IDs appear as the `id` field inside `uid` in entity JSON, and as the string literal in policy rules (e.g. `Lakekeeper::User::"oidc~alice"`). | Entity type | ID format | Example | |--------------------------------------------------|---------------------------------------------|-----| | `Lakekeeper::Server` | UUIDv7 (auto-assigned, one per deployment) | `019c192e-cc20-7a13-a1ac-2e3390f81908` | | `Lakekeeper::Project` | String (alphanumeric, hyphens, underscores) | `my-project` or `019c192f-0613-7422-90f1-7dd6b09f033c` | | `Lakekeeper::Warehouse` | UUIDv7 (assigned at warehouse creation) | `d08dca76-ff69-11f0-9aa6-ab201d553ec5` | | `Lakekeeper::Namespace` | UUIDv7 (assigned at namespace creation) | `019c192f-18c2-7f93-848f-542d8f32bc3c` | | `Lakekeeper::Table` | `/` | `d08dca76-.../019c192f-...` | | `Lakekeeper::View` | `/` | `d08dca76-.../019c192f-...` | | `Lakekeeper::User` | `~` | `oidc~alice@example.com` | | `Lakekeeper::Role` | `/~` | `my-project/oidc~data-admins` | | `Lakekeeper::UserDerivedAttributes` | Same ID as the owning `User` (1:1) | `oidc~alice@example.com` | **Notes:** - User IDs are constructed by Lakekeeper from the token's issuer/provider and the subject claim. For OIDC the format is `oidc~`. - Role IDs combine the project ID, the provider ID, and the role's source ID within that provider. - All UUIDs shown in entity JSON are the literal string without braces. ## External Entity Management **Default Behavior**: Lakekeeper automatically includes `Lakekeeper::User` entities with information extracted from user tokens. When `LAKEKEEPER__OPENID_ROLES_CLAIM` is configured, Lakekeeper also provides `Lakekeeper::Role` entities, enabling role-based policies. **External Management**: In scenarios where role information isn't available in tokens, you can manage users and roles externally: 1. Set `LAKEKEEPER__CEDAR__EXTERNALLY_MANAGED_USER_AND_ROLES` to `true` 2. Provide entity definitions via `LAKEKEEPER__CEDAR__ENTITY_JSON_SOURCES*` configurations 3. Ensure your external entities conform to Lakekeeper's Cedar schema See [Entity Definition Example](#entity-definition-example) below for the JSON format. **Schema Reference**: The Lakekeeper Cedar schema defines all available entity types, attributes, and actions. All entities and policies are validated against this schema on startup and refresh. Download the schema above or view it on [GitHub](https://github.com/lakekeeper/lakekeeper/tree/main/docs/docs/api). ## Policy Examples The following examples demonstrate common Cedar policy patterns. Unless otherwise noted, examples assume a single-project setup (the project is not restricted). Note that warehouse names are only guaranteed to be unique within a project. ??? example "Allow everything for everyone" ```cedar permit ( principal, action, resource ); ``` ??? example "Allow everything for a specific user" ```cedar permit ( principal == Lakekeeper::User::"oidc~", // Add user name in comment for documentation action, resource ); ``` ??? example "Allow everything for all users in a role/group" **Option 1 — using the full Role entity ID** The Role ID has the form `/~`. You can look it up in the Lakekeeper UI or via the management API. ```cedar permit ( principal in Lakekeeper::Role::"my-project/oidc~data-engineers", action, resource ); ``` **Option 2 — using `project_roles`** `project_roles` is always an empty set for server-level actions (which carry no project context), so this policy will never permit them. Use Option 1 with the full Role ID when server-level permissions are required, or grant direct access to users. ```cedar permit ( principal is Lakekeeper::User, action, resource ) when { principal.project_roles.contains( {provider_id: "oidc", source_id: "data-engineers"} ) }; ``` ??? example "Grant access based on a token-sourced group (project_roles)" Use this pattern when roles come from OIDC token claims (configured via `LAKEKEEPER__OPENID_ROLES_CLAIM`). This avoids constructing the full role entity ID (which requires the project ID) and works identically in both token mode and external-entity mode. Note that `project_roles` is always an empty set for server-level actions — use the full Role ID for those. ```cedar permit ( principal is Lakekeeper::User, action in [Lakekeeper::Action::"NamespaceActions", Lakekeeper::Action::"TableActions", Lakekeeper::Action::"ViewActions"], resource ) when { resource.warehouse.name == "my-warehouse" && principal.project_roles.contains( {provider_id: "oidc", source_id: "data-engineers"} ) }; permit ( principal is Lakekeeper::User, action in [Lakekeeper::Action::"WarehouseModifyActions"], resource ) when { resource.name == "my-warehouse" && principal.project_roles.contains( {provider_id: "oidc", source_id: "data-engineers"} ) }; ``` The `provider_id` must match the Authenticator ID configured in Lakekeeper (typically `"oidc"`). The `source_id` is the role/group name as it appears in the token claim (without any prefix). ??? example "Allow everything for multiple specific users" ```cedar permit ( principal is Lakekeeper::User, action, resource ) when { [ Lakekeeper::User::"oidc~", // User 1 name for documentation Lakekeeper::User::"oidc~", // User 2 name for documentation Lakekeeper::User::"oidc~" // User 3 name for documentation ].contains(principal) }; ``` ??? example "Basic server and project permissions for all authenticated users" ```cedar permit ( principal, action in [ Lakekeeper::Action::"ProjectDescribeActions", // Applies to all projects unless resource is restricted ], resource ); ``` ??? example "Read and write access to a namespace and all its contents (recursive)" ```cedar permit ( principal == Lakekeeper::User::"oidc~", action in [Lakekeeper::Action::"NamespaceModifyActions", Lakekeeper::Action::"TableModifyActions", Lakekeeper::Action::"ViewModifyActions"], resource ) when { ( resource is Lakekeeper::Warehouse && resource.name == "dev" ) || ( resource is Lakekeeper::Namespace && resource.warehouse.name == "dev" && resource.name == "finance.revenue" ) || ( resource is Lakekeeper::Table && resource.warehouse.name == "dev" && resource.namespace.name like "finance.revenue*" ) || // Include sub-namespaces via wildcard ( resource is Lakekeeper::View && resource.warehouse.name == "dev" && resource.namespace.name like "finance.revenue*" ) }; ``` ??? example "Read access to a warehouse and all its contents for a group" **Option 1 — full Role ID:** ```cedar permit ( principal in Lakekeeper::Role::"my-project/oidc~warehouse-readers", action in [ Lakekeeper::Action::"WarehouseDescribeActions", Lakekeeper::Action::"NamespaceDescribeActions", Lakekeeper::Action::"TableSelectActions", Lakekeeper::Action::"ViewSelectActions" ], resource ) when { (resource has warehouse && resource.warehouse.name == "dev") || (resource is Lakekeeper::Warehouse && resource.name == "dev") }; ``` **Option 2 — `project_roles`, no project ID needed:** ```cedar permit ( principal is Lakekeeper::User, action in [ Lakekeeper::Action::"WarehouseDescribeActions", Lakekeeper::Action::"NamespaceDescribeActions", Lakekeeper::Action::"TableSelectActions", Lakekeeper::Action::"ViewSelectActions" ], resource ) when { principal.project_roles.contains({provider_id: "oidc", source_id: "warehouse-readers"}) && ((resource has warehouse && resource.warehouse.name == "dev") || (resource is Lakekeeper::Warehouse && resource.name == "dev")) }; ``` ??? example "Read access to a warehouse and all its contents in multi-project setups" ```cedar permit ( principal in Lakekeeper::Role::"my-project/oidc~warehouse-readers", action in [ Lakekeeper::Action::"WarehouseDescribeActions", Lakekeeper::Action::"NamespaceDescribeActions", Lakekeeper::Action::"TableSelectActions", Lakekeeper::Action::"ViewSelectActions" ], resource in Lakekeeper::Project::"my-project" ) when { (resource has warehouse && resource.warehouse.name == "dev") || (resource is Lakekeeper::Warehouse && resource.name == "dev") }; ``` ??? example "ABAC: Role-based table access using static role membership" This example grants read/write access to tables tagged with an `access-role` property matching the requesting user's role — using traditional RBAC role membership. The `access-role-*` keys use the `access-` prefix so Lakekeeper parses them as entity references; the `.raw` field always stores the original string. ```cedar @id("abac-role-based-access-marketing-select") @description("ABAC: Allow Read access to tables tagged with access-role-select:marketing to the marketing-select role") permit ( principal in Lakekeeper::Role::"my-project/lakekeeper~marketing-select", action in Lakekeeper::Action::"TableSelectActions", resource is Lakekeeper::Table ) when { resource.properties.hasTag("access-role-select") && resource.properties.getTag("access-role-select").raw == "marketing" }; @id("abac-role-based-access-marketing-modify") @description("ABAC: Allow Modify access to tables tagged with access-role-modify:marketing, but prevent removing or changing the tag itself") permit ( principal in Lakekeeper::Role::"my-project/lakekeeper~marketing-modify", action in Lakekeeper::Action::"TableModifyActions", resource is Lakekeeper::Table ) when { resource.properties.hasTag("access-role-modify") && resource.properties.getTag("access-role-modify").raw == "marketing" } unless { // Prevent users from removing or changing the access-control tag itself. action == Lakekeeper::Action::"CommitTable" && (context.table_properties_removal.contains("access-role-modify") || context.table_properties_updates.hasTag("access-role-modify")) }; @id("abac-role-based-access-marketing-admin") @description("ABAC: Allow full Modify access (including changing access tags) to marketing-admin role") permit ( principal in Lakekeeper::Role::"my-project/lakekeeper~marketing-admin", action in Lakekeeper::Action::"TableModifyActions", resource is Lakekeeper::Table ) when { resource.properties.hasTag("access-role-modify") && resource.properties.getTag("access-role-modify").raw == "marketing" }; ``` ??? example "ABAC: Access control lists stored directly in table properties" This is a more advanced ABAC pattern where each table carries its own access control list in an `access-owners` and `access-readers` property. The values are JSON arrays of entity references (roles and/or users), parsed automatically by Lakekeeper. **Tag the table** (e.g. via the Iceberg REST API or your ETL pipeline): ``` access-owners = ["role-full:oidc~data-admins", "user:oidc~alice@example.com"] access-readers = ["role:analysts", "role-full:oidc~reporting-team"] ``` **Cedar policies** (no role names are hardcoded — access is determined entirely by table metadata): ```cedar @id("abac-property-acl-select") @description("Allow read access to any table where the principal is listed in the access-readers property") permit ( principal, action in Lakekeeper::Action::"TableSelectActions", resource is Lakekeeper::Table ) when { resource.properties.hasTag("access-readers") && (principal in resource.properties.getTag("access-readers").roles || principal in resource.properties.getTag("access-readers").users) }; @id("abac-property-acl-modify") @description("Allow write access to any table where the principal is listed in the access-owners property") permit ( principal, action in Lakekeeper::Action::"TableModifyActions", resource is Lakekeeper::Table ) when { resource.properties.hasTag("access-owners") && (principal in resource.properties.getTag("access-owners").roles || principal in resource.properties.getTag("access-owners").users) } unless { // Owners can modify the table but cannot change the access-control properties themselves. // Grant the marketing-admin role a separate policy if escalation is needed. action == Lakekeeper::Action::"CommitTable" && (context.table_properties_removal.contains("access-owners") || context.table_properties_removal.contains("access-readers") || context.table_properties_updates.hasTag("access-owners") || context.table_properties_updates.hasTag("access-readers")) }; ``` !!! tip "Role resolution" `principal in resource.properties.getTag("access-readers").roles` uses Cedar's built-in entity hierarchy. A user is considered `in` a role if that role appears as an ancestor in the user entity's parent chain — exactly the same mechanism used for static role-based policies. This means the access control lists stored in table properties work seamlessly with both token-extracted roles (`LAKEKEEPER__OPENID_ROLES_CLAIM`) and externally managed role assignments. ??? example "ABAC: Namespace-level access control inherited by all tables" Apply access-control lists at the namespace level so that all tables in the namespace inherit the same restrictions. **Tag the namespace**: ``` access-readers = ["role-full:oidc~finance-readers"] access-writers = ["role-full:oidc~finance-engineers"] ``` **Cedar policies**: ```cedar @id("abac-namespace-acl-select") @description("Allow read access to tables when the namespace has access-readers listing the principal") permit ( principal, action in Lakekeeper::Action::"TableSelectActions", resource is Lakekeeper::Table ) when { resource.namespace.properties.hasTag("access-readers") && (principal in resource.namespace.properties.getTag("access-readers").roles || principal in resource.namespace.properties.getTag("access-readers").users) }; ``` ??? example "Recommended permissions for the OPA bridge user" ```cedar @id("opa-permissions") @description("Grant global permission read access to OPA user") permit ( principal == Lakekeeper::User::"oidc~", // OPA service account action in [ Lakekeeper::Action::"IntrospectServerAuthorization", Lakekeeper::Action::"IntrospectProjectAuthorization", Lakekeeper::Action::"IntrospectRoleAuthorization", Lakekeeper::Action::"WarehouseDescribeActions", Lakekeeper::Action::"IntrospectWarehouseAuthorization", Lakekeeper::Action::"NamespaceDescribeActions", Lakekeeper::Action::"IntrospectNamespaceAuthorization", Lakekeeper::Action::"TableDescribeActions", Lakekeeper::Action::"IntrospectTableAuthorization", Lakekeeper::Action::"ViewDescribeActions", Lakekeeper::Action::"IntrospectViewAuthorization", ], resource ); ``` ## Entity Definition Example Lakekeeper provides the following entities internally to Cedar: Server, Project, Warehouse, Namespace, Table, View. Additionally, if `LAKEKEEPER__OPENID_ROLES_CLAIM` is set, also User and Roles are provided to Cedar. A request on a table called "my-table" in Namespace "my-namespace" provides the following entities to Cedar: ??? example "Entities provided to Cedar internally" ```json [ { "uid": { "type": "Lakekeeper::Table", "id": "d08dca76-ff69-11f0-9aa6-ab201d553ec5/019c192f-18d0-7390-9d90-93facfb8e3d3" }, "attrs": { "namespace": { "__entity": { "type": "Lakekeeper::Namespace", "id": "019c192f-18c2-7f93-848f-542d8f32bc3c" } }, "protected": false, "warehouse": { "__entity": { "type": "Lakekeeper::Warehouse", "id": "d08dca76-ff69-11f0-9aa6-ab201d553ec5" } }, "name": "transactions", "project": { "__entity": { "type": "Lakekeeper::Project", "id": "019c192f-0613-7422-90f1-7dd6b09f033c" } } }, "tags": { // Table properties are stored as Cedar entity tags. // Access-prefixed keys (access- / access_) have roles and users parsed. "access-owners": { "raw": "[\"role-full:oidc~data-admins\", \"user:oidc~alice\"]", "roles": [ { "__entity": { "type": "Lakekeeper::Role", "id": "019c192f-0613-7422-90f1-7dd6b09f033c/oidc~data-admins" } } ], "users": [ { "__entity": { "type": "Lakekeeper::User", "id": "oidc~alice" } } ] }, "description": { "raw": "Financial transactions table", "roles": [], "users": [] } }, "parents": [ { "type": "Lakekeeper::Namespace", "id": "019c192f-18c2-7f93-848f-542d8f32bc3c" } ] }, { "uid": { "type": "Lakekeeper::Server", "id": "019c192e-cc20-7a13-a1ac-2e3390f81908" }, "attrs": {}, "parents": [] }, { "uid": { "type": "Lakekeeper::Project", "id": "019c192f-0613-7422-90f1-7dd6b09f033c" }, "attrs": {}, "parents": [ { "type": "Lakekeeper::Server", "id": "019c192e-cc20-7a13-a1ac-2e3390f81908" } ] }, { "uid": { "type": "Lakekeeper::Warehouse", "id": "d08dca76-ff69-11f0-9aa6-ab201d553ec5" }, "attrs": { "is_active": true, "protected": false, "project": { "__entity": { "type": "Lakekeeper::Project", "id": "019c192f-0613-7422-90f1-7dd6b09f033c" } }, "name": "wh-1" }, "parents": [ { "type": "Lakekeeper::Project", "id": "019c192f-0613-7422-90f1-7dd6b09f033c" } ] }, { "uid": { "type": "Lakekeeper::Namespace", "id": "019c192f-18c2-7f93-848f-542d8f32bc3c" }, "attrs": { "protected": false, "warehouse": { "__entity": { "type": "Lakekeeper::Warehouse", "id": "d08dca76-ff69-11f0-9aa6-ab201d553ec5" } }, "project": { "__entity": { "type": "Lakekeeper::Project", "id": "019c192f-0613-7422-90f1-7dd6b09f033c" } }, "name": "my-namespace" }, "tags": { "location": { "raw": "s3://tests/075272e23ed548d8bfd722a7a383cd50/019c192f-18c2-7f93-848f-542d8f32bc3c", "roles": [], "users": [] } }, "parents": [ { "type": "Lakekeeper::Warehouse", "id": "d08dca76-ff69-11f0-9aa6-ab201d553ec5" } ] }, { "uid": { "type": "Lakekeeper::User", "id": "oidc~2f268e8b-8cc1-4edd-a9df-87d69f7e9deb" }, "attrs": { // Lakekeeper-managed roles the user belongs to (from the management API). "roles": [], // Token-sourced roles flattened for the current project context. // Populated from LAKEKEEPER__OPENID_ROLES_CLAIM when present. "project_roles": [ {"provider_id": "oidc", "source_id": "analysts"} ], // source_id of each provider-resolved role; only populated when // LAKEKEEPER__CEDAR__GLOBAL_ROLE_IDS_ENABLED=true, otherwise []. "global_role_ids": [], "provider_id": "oidc", "source_id": "2f268e8b-8cc1-4edd-a9df-87d69f7e9deb" }, "parents": [] } ] ``` Lakekeeper can log all entities provided to Cedar for debugging purposes. See the [Cedar Configuration](./configuration.md#cedar) section for details on enabling entity logging. When `LAKEKEEPER__CEDAR__EXTERNALLY_MANAGED_USER_AND_ROLES` is set to `true`, Lakekeeper excludes User and Role entities from Cedar requests and expects you to provide them externally via `LAKEKEEPER__CEDAR__ENTITY_JSON_SOURCES*` configurations. The following example shows an `entity.json` file defining user-to-role assignments: ```json [ { "uid": { "type": "Lakekeeper::User", "id": "oidc~90471f73-e338-4032-9a6b-1e021cc3cb1e" }, "attrs": { // Roles the user is a member of. // Use the `parents` array (not this set) to establish the hierarchy; // keep both in sync. "roles": [ { "__entity": { "type": "Lakekeeper::Role", "id": "data-engineering" } } ], // Flat set of role identities relevant to the current project. // Enables principal.project_roles.contains({provider_id, source_id}) checks. // Provide these only in single project setups. "project_roles": [ { "provider_id": "oidc", "source_id": "warehouse-1-admins" } ], // source_id of each provider-resolved role as plain strings. // Required by the schema; use [] when GLOBAL_ROLE_IDS_ENABLED is off. "global_role_ids": [], // Authentication provider and subject ID of this user. "provider_id": "oidc", "source_id": "90471f73-e338-4032-9a6b-1e021cc3cb1e" }, "parents": [ { "type": "Lakekeeper::Role", "id": "data-engineering" } ] }, { "uid": { "type": "Lakekeeper::Role", "id": "data-engineering" }, "attrs": { "project": { "__entity": { "type": "Lakekeeper::Project", "id": "" } }, "provider_id": "entities-file", "source_id": "data-engineering" }, "parents": [ { "type": "Lakekeeper::Role", "id": "warehouse-1-admins" } ] }, { "uid": { "type": "Lakekeeper::Role", "id": "warehouse-1-admins" }, "attrs": { "project": { "__entity": { "type": "Lakekeeper::Project", "id": "" } }, "provider_id": "entities-file", "source_id": "warehouse-1-admins" }, "parents": [] } ] ``` !!! tip "Required User attributes" Every `Lakekeeper::User` entity in an external file **must** include `roles`, `project_roles`, `provider_id`, `source_id`, and `global_role_ids`. Omitting any of these will cause a schema validation error on startup. Use `[]` for `global_role_ids` when it is not used or `LAKEKEEPER__CEDAR__GLOBAL_ROLE_IDS_ENABLED` is disabled. Set `project_roles` to `[]` in multi-project setups. ## Policy and Entity Management **Startup Behavior:** - All policy and entity files are loaded and validated against the Cedar schema - If any file is unreadable or invalid, Lakekeeper fails to start with an error This ensures that authorization policies are always valid before serving requests **Refresh Behavior:** Configure automatic policy refresh using `LAKEKEEPER__CEDAR__REFRESH_INTERVAL_SECS` (default: 5 seconds): 1. **Change Detection**: Lightweight checks monitor ConfigMap versions and file timestamps 2. **Reload on Change**: Modified entity or policy files trigger a full reload of all files to guarantee consistency 3. **Atomic Updates**: The in-memory store is only updated if all files reload successfully 4. **Error Handling**: If any reload fails, the previous configuration is retained, an error is logged, and health checks report unhealthy status This approach ensures that authorization policies remain consistent and that partial updates never compromise security. ## Cedar Actions The following tables document all available Cedar actions. Use action groups for broad permissions or individual actions for fine-grained control. The **Audit log `action_name`** column lists the standardized snake_case identifier that appears in the `action.action_name` field of [audit log events](./logging.md#audit-logs) when that action is checked. For actions shared with OpenFGA (those derived from the authorizer-agnostic `Catalog*Action` enums) the same value is emitted regardless of which authorizer is configured. A dash (`—`) means the action is only reached through a silent backend pre-check (no audit event is emitted under a stable standardized name). Because the audit `action_name` deliberately omits the resource type (`delete`, `rename`, `get_metadata`, `introspect_authorization`, etc. appear across multiple Cedar action names), use the sibling `entity.entity_type` field on the audit event to pick the right Cedar action. For example, `action_name = "read_data"` with `entity.entity_type = "table"` corresponds to `Lakekeeper::Action::"ReadTableData"`: ```json { "action": { "action_name": "read_data" }, "entity": { "entity_type": "table", "warehouse-id": "faac5cb2-5902-11f1-b9a7-1360e98a724d", "namespace": "finance", "table": "products" } } ``` ### Server Actions | Action | Audit log `action_name` | Description | |---------------------------------------------------|--------------------------------------------|--------------------------| | `ListServerCedarEntitySources` | `list_cedar_entity_sources` | List Cedar entity sources configured at server level | | `ListCedarPoliciesFromServerSources` | `list_cedar_policies_from_server_sources` | View Cedar policies from server-level sources | | `ListServerCedarPolicySources` | `list_cedar_policy_sources` | List Cedar policy sources configured at server level | | `CreateProject` | `create_project` | Create new projects | | `UpdateUsers` | `update_users` | Modify user information | | `DeleteUsers` | `delete_users` | Remove users from the system | | `ListUsers` | `list_users` | View all users in the system | | `ProvisionUsers` | `provision_users` | Provision new users | | `IntrospectServerAuthorization` | `introspect_authorization` | Check access permissions on the server for **other** users (applies when `identity` parameter doesn't match current user) | | `EvaluateCedarPolicies` | `evaluate_cedar_policies` | Evaluate user-provided Cedar policies (development tool) | ### Project Actions | Action | Audit log `action_name` | Description | |-----------------------------------------------|----------------------------|------------------------------| | `GetProjectMetadata` | `get_metadata` | View project details and configuration | | `ListWarehouses` | `list_warehouses` | List all warehouses in the project | | `IncludeProjectInList` | `include_in_list` | Include project in list operations (visibility) | | `ListRoles` | `list_roles` | List all roles in the project | | `SearchRoles` | `search_roles` | Search for roles in the project | | `GetProjectEndpointStatistics` | `get_endpoint_statistics` | View API usage statistics for the project | | `GetProjectTaskQueueConfig` | `get_task_queue_config` | View task queue configuration for the project | | `GetProjectTasks` | `get_project_tasks` | List background tasks in the project | | `IntrospectProjectAuthorization` | `introspect_authorization` | Check access permissions on the project for other users | | `CreateWarehouse` | `create_warehouse` | Create new warehouses in the project | | `DeleteProject` | `delete` | Delete the project | | `RenameProject` | `rename` | Change project name | | `CreateRole` | `create_role` | Create new roles in the project | | `ModifyProjectTaskQueueConfig` | `modify_task_queue_config` | Update task queue configuration | | `ControlProjectTasks` | `control_project_tasks` | Manage background tasks (cancel, retry, etc.) | The following Action Groups are available: `ProjectDescribeActions` (read-only), `ProjectModifyActions` (includes Describe), `ProjectActions` (all) ### Role Actions | Action | Audit log `action_name` | Description | |--------------------------------------------|-------------------------|---------------------------------| | `AssumeRole` | `assume_role` | Assume this role (use role's permissions) | | `DeleteRole` | `delete` | Delete the role | | `UpdateRole` | `update` | Modify role properties | | `ReadRole` | `read` | View role details | | `ReadRoleMetadata` | `read_metadata` | View role metadata | | `IntrospectRoleAuthorization` | — | Check access permissions on the role for other users | The following Action Groups are available: `RoleActions` (all role operations) ### Warehouse Actions | Action | Audit log `action_name` | Description | |-------------------------------------------------|-----------------------------|----------------------------| | `UseWarehouse` | `use` | Use the warehouse (required for any warehouse operations) | | `ListNamespacesInWarehouse` | `list_namespaces` | List namespaces in the warehouse | | `GetWarehouseMetadata` | `get_metadata` | View warehouse configuration and details | | `GetConfig` | `get_config` | Get warehouse configuration for clients | | `IncludeWarehouseInList` | `include_in_list` | Include warehouse in list operations (visibility) | | `ListDeletedTabulars` | `list_deleted_tabulars` | List soft-deleted tables and views | | `GetTaskQueueConfig` | `get_task_queue_config` | View task queue configuration | | `GetAllTasks` | `get_all_tasks` | List all background tasks in the warehouse | | `ListEverythingInWarehouse` | `list_everything` | List all objects (namespaces, tables, views) in warehouse | | `GetWarehouseEndpointStatistics` | `get_endpoint_statistics` | View API usage statistics for the warehouse | | `IntrospectWarehouseAuthorization` | `introspect_authorization` (was `IntrospectAuthorization` until 0.12.2) | Check access permissions on the warehouse for other users | | `DeleteWarehouse` | `delete` | Delete the warehouse | | `UpdateStorage` | `update_storage` | Modify storage configuration | | `UpdateStorageCredential` | `update_storage_credential` | Update storage credentials | | `DeactivateWarehouse` | `deactivate` | Deactivate the warehouse (suspend operations) | | `ActivateWarehouse` | `activate` | Activate a deactivated warehouse | | `RenameWarehouse` | `rename` | Change warehouse name | | `ModifySoftDeletion` | `modify_soft_deletion` | Configure soft-deletion settings | | `ModifyTaskQueueConfig` | `modify_task_queue_config` | Update task queue configuration | | `ControlAllTasks` | `control_all_tasks` | Manage all background tasks | | `SetWarehouseProtection` | `set_protection` | Enable/disable deletion protection | | `CreateNamespaceInWarehouse` | `create_namespace` | Create namespaces directly in the warehouse | The following Action Groups are available: `WarehouseDescribeActions` (read-only), `WarehouseModifyActions` (includes Describe), `WarehouseActions` (all) ### Namespace Actions | Action | Audit log `action_name` | Description | |-------------------------------------------------|-------------------------|----------------------------| | `ListEverythingInNamespace` | `list_everything` | List all objects (tables, views, child namespaces) in namespace | | `GetNamespaceMetadata` | `get_metadata` | View namespace properties and configuration | | `IncludeNamespaceInList` | `include_in_list` | Include namespace in list operations (visibility) | | `ListTables` | `list_tables` | List tables in the namespace | | `ListViews` | `list_views` | List views in the namespace | | `ListNamespacesInNamespace` | `list_namespaces` | List child namespaces | | `IntrospectNamespaceAuthorization` | `introspect_authorization` (was `IntrospectAuthorization` until 0.12.2) | Check access permissions on the namespace for other users | | `DeleteNamespace` | `delete` | Delete the namespace | | `SetNamespaceProtection` | `set_protection` | Enable/disable deletion protection | | `CreateTable` | `create_table` | Create tables in the namespace | | `CreateView` | `create_view` | Create views in the namespace | | `CreateNamespaceInNamespace` | `create_namespace` | Create child namespaces | | `UpdateNamespaceProperties` | `update_properties` | Modify namespace properties | The following Action Groups are available: `NamespaceDescribeActions` (read-only), `NamespaceModifyActions` (includes Describe), `NamespaceActions` (all) ### Table Actions | Action | Audit log `action_name` | Description | |---------------------------------------------|-------------------------|--------------------------------| | `GetTableMetadata` | `get_metadata` | View table schema, metadata, and configuration | | `IncludeTableInList` | `include_in_list` | Include table in list operations (visibility) | | `GetTableTasks` | `get_tasks` | List background tasks for the table | | `ReadTableData` | `read_data` | Read data from the table (SELECT queries) | | `IntrospectTableAuthorization` | `introspect_authorization` (was `IntrospectAuthorization` until 0.12.2) | Check access permissions on the table for other users | | `DropTable` | `drop` | Delete the table | | `WriteTableData` | `write_data` | Write data to the table (INSERT, UPDATE, DELETE) | | `RenameTable` | `rename` | Change table name or move to different namespace | | `UndropTable` | `undrop` | Restore a soft-deleted table | | `ControlTableTasks` | `control_tasks` | Manage table background tasks | | `SetTableProtection` | `set_protection` | Enable/disable deletion protection | | `CommitTable` | `commit` | Commit table changes (schema updates, snapshots) | *Action Groups*: `TableDescribeActions` (metadata only), `TableSelectActions` (includes Describe + read data), `TableModifyActions` (includes Describe + Select + modifications), `TableActions` (all) ### View Actions | Action | Audit log `action_name` | Description | |--------------------------------------------|-------------------------|---------------------------------| | `GetViewMetadata` | `get_metadata` | View view definition and metadata | | `IncludeViewInList` | `include_in_list` | Include view in list operations (visibility) | | `GetViewTasks` | `get_tasks` | List background tasks for the view | | `SelectView` | `select` | Execute the view to produce rows (data-plane; required to traverse the view in a `referenced-by` chain) | | `IntrospectViewAuthorization` | `introspect_authorization` (was `IntrospectAuthorization` until 0.12.2) | Check access permissions on the view for other users | | `DropView` | `drop` | Delete the view | | `RenameView` | `rename` | Change view name or move to different namespace | | `UndropView` | `undrop` | Restore a soft-deleted view | | `ControlViewTasks` | `control_tasks` | Manage view background tasks | | `SetViewProtection` | `set_protection` | Enable/disable deletion protection | | `CommitView` | `commit` | Commit view changes (update definition, properties) | *Action Groups*: `ViewDescribeActions` (metadata only), `ViewSelectActions` (includes Describe + execute), `ViewModifyActions` (includes Describe + Select + modifications), `ViewActions` (all) ### Context-Aware Actions Some actions include additional context information in authorization requests. This enables ABAC policies to make decisions based on properties being created, updated, or removed—for example, preventing users from modifying specific property keys. All property contexts use the `ResourceProperties` entity type (same structure as `resource.properties`), giving you access to `.raw`, `.roles`, and `.users` on each property entry — including parsed role/user references in access-prefixed keys. | Action | Context fields | |-------------------------------------------|----------------------------------| | `CreateProject` | `project_name?: String`, `project_id?: String` | | `CreateWarehouse` | `warehouse_name?: String` | | `CreateRole` | `role_name?: String` | | `CreateNamespaceInWarehouse` | `namespace_name?: String`, `initial_namespace_properties: ResourceProperties` | | `CreateNamespaceInNamespace` | `namespace_name?: String`, `initial_namespace_properties: ResourceProperties` | | `CreateTable` | `table_name?: String`, `table_id?: String`, `initial_table_properties: ResourceProperties` | | `CreateView` | `view_name?: String`, `initial_view_properties: ResourceProperties` | | `UpdateNamespaceProperties` | `namespace_properties_updates: ResourceProperties`, `namespace_properties_removal: Set` | | `CommitTable` | `table_properties_updates: ResourceProperties`, `table_properties_removal: Set` | | `CommitView` | `view_properties_updates: ResourceProperties`, `view_properties_removal: Set` | **Example**: Prevent a table from being created with an `access-owners` property that doesn't include at least one owner from the `oidc~data-governance` role: ```cedar forbid ( principal, action == Lakekeeper::Action::"CreateTable", resource is Lakekeeper::Namespace ) when { context.initial_table_properties.hasTag("access-owners") && !(Lakekeeper::Role::"/oidc~data-governance" in context.initial_table_properties.getTag("access-owners").roles) }; ``` --- # Authorization (OpenFGA) Source: https://docs.lakekeeper.io/docs/latest/authorization-openfga/ # Authorization with OpenFGA Lakekeeper can use [OpenFGA](https://openfga.dev) to store and evaluate permissions. OpenFGA provides bi-directional inheritance, which is key for managing hierarchical namespaces in modern lakehouses. For query engines like Trino, Lakekeeper's OPA bridge translates OpenFGA permissions into Open Policy Agent (OPA) format. See the [OPA Bridge Guide](./opa.md) for details. Check the [Authorization Configuration](./configuration.md#authorization) for setup details. !!! note "Minimum OpenFGA version" **OpenFGA v1.11 or later is required.** The bootstrap and `lakekeeper openfga reconcile` paths use OpenFGA's idempotent-write semantics (`on_duplicate: ignore` / `on_missing: ignore`), introduced in v1.11. Earlier versions will fail with `cannot write a tuple which already exists` during a re-bootstrap or reconcile run. We test against v1.14. ## Grants The default permission model is focused on collaborating on data. Permissions are additive. The underlying OpenFGA model is defined in [`schema.fga` on GitHub](https://github.com/lakekeeper/lakekeeper/blob/main/authz/openfga/). The following grants are available: | Entity | Grant | |-----------|------------------------------------------------------------------| | server | admin, operator | | project | project_admin, security_admin, data_admin, role_creator, describe, select, create, modify | | warehouse | ownership, pass_grants, manage_grants, describe, select, create, modify | | namespace | ownership, pass_grants, manage_grants, describe, select, create, modify | | table | ownership, pass_grants, manage_grants, describe, select, modify | | view | ownership, pass_grants, manage_grants, describe, select, modify | | generic table | ownership, pass_grants, manage_grants, describe, select, modify | | role | assignee, ownership | ##### Ownership Owners of objects have all rights on the specific object. When principals create new objects, they automatically become owners of these objects. This enables powerful self-service scenarios where users can act autonomously in a (sub-)namespace. By default, Owners of objects are also able to access grants on objects, which enables them to expand the access to their owned objects to new users. Enabling [Managed Access](#managed-access) for a Warehouse or Namespace removes the `grant` privilege from owners. ##### Server: Admin A `server`'s `admin` role is the most powerful role (apart from `operator`) on the server. In order to guarantee auditability, this role can list and administrate all Projects, but does not have access to data in projects. While the `admin` can assign himself the `project_admin` role for a project, this assignment is tracked by `OpenFGA` for audits. `admin`s can also manage all projects (but no entities within it), server settings and users. ##### Server: Operator The `operator` has unrestricted access to all objects in Lakekeeper. It is designed to be used by technical users (e.g., a Kubernetes Operator) managing the Lakekeeper deployment. ##### Project: Security Admin A `security_admin` in a project can manage all security-related aspects, including grants and ownership for the project and all objects within it. However, they cannot modify or access the content of any object, except for listing and browsing purposes. ##### Project: Data Admin A `data_admin` in a project can manage all data-related aspects, including creating, modifying, and deleting objects within the project. They can delegate the `data_admin` role they already hold (for example to team members), but they do not have general grant or ownership administration capabilities. ##### Project: Admin A `project_admin` in a project has the combined responsibilities of both `security_admin` and `data_admin`. They can manage all security-related aspects, including grants and ownership, as well as all data-related aspects, including creating, modifying, and deleting objects within the project. ##### Project: Role Creator A `role_creator` in a project can create new roles within it. This role is essential for delegating the creation of roles without granting broader administrative privileges. ##### Describe The `describe` grant allows a user to view metadata and details about an object without modifying it. This includes listing objects and viewing their properties. The `describe` grant is inherited down the object hierarchy, meaning if a user has the `describe` grant on a higher-level entity, they can also describe all child entities within it. The `describe` grant is implicitly included with the `select`, `create`, and `modify` grants. ##### Select The `select` grant allows a user to read data from an object, such as tables or views. This includes querying and retrieving data. The `select` grant is inherited down the object hierarchy, meaning if a user has the `select` grant on a higher-level entity, they can select all views and tables within it. The `select` grant implicitly includes the `describe` grant. ##### Create The `create` grant allows a user to create new objects within an entity, such as tables, views, or namespaces. The `create` grant is inherited down the object hierarchy, meaning if a user has the `create` grant on a higher-level entity, they can also create objects within all child entities. The `create` grant implicitly includes the `describe` grant. ##### Modify The `modify` grant allows a user to change the content or properties of an object, such as updating data in tables or altering views. The `modify` grant is inherited down the object hierarchy, meaning if a user has the `modify` grant on a higher-level entity, they can also modify all child entities within it. The `modify` grant implicitly includes the `select` and `describe` grants. ##### Pass Grants The `pass_grants` grant allows a user to pass their own privileges to other users. This means that if a user has certain permissions on an object, they can grant those same permissions to others. However, the `pass_grants` grant does not include the ability to pass the `pass_grants` privilege itself. ##### Manage Grants The `manage_grants` grant allows a user to manage all grants on an object, including creating, modifying, and revoking grants. This also includes `manage_grants` and `pass_grants`. ## Inheritance * **Top-Down-Inheritance**: Permissions in higher up entities are inherited to their children. For example if the `modify` privilege is granted on a `warehouse` for a principal, this principal is also able to `modify` any namespaces, including nesting ones, tables and views within it. * **Bottom-Up-Inheritance**: Permissions on lower entities, for example tables, inherit basic navigational privileges to all higher layer principals. For example, if a user is granted the `select` privilege on table `ns1.ns2.table_1`, that user is implicitly granted limited list privileges on `ns1` and `ns2`. Only items in the direct path are presented to users. If `ns1.ns3` would exist as well, a list on `ns1` would only show `ns1.ns2`. ## Managed Access Managed access is a feature designed to provide stricter control over access privileges within Lakekeeper. It is particularly useful for organizations that require a more restrictive access control model to ensure data security and compliance. In some cases, the default ownership model, which grants all privileges to the creator of an object, can be too permissive. This can lead to situations where non-admin users unintentionally share data with unauthorized users by granting privileges outside the scope defined by administrators. Managed access addresses this concern by removing the `grant` privilege from owners and centralizing the management of access privileges. With managed access, admin-like users can define access privileges on high-level container objects, such as warehouses or namespaces, and ensure that all child objects inherit these privileges. This approach prevents non-admin users from granting privileges that are not authorized by administrators, thereby reducing the risk of unintentional data sharing and enhancing overall security. Managed access combines elements of Role-Based Access Control (RBAC) and Discretionary Access Control (DAC). While RBAC allows privileges to be assigned to roles and users, DAC assigns ownership to the creator of an object. By integrating managed access, Lakekeeper provides a balanced access control model that supports both self-service analytics and data democratization while maintaining strict security controls. Managed access can be enabled or disabled for warehouses and namespaces using the UI or the `../managed-access` Endpoints. Managed access settings are inherited down the object hierarchy, meaning if managed access is enabled on a higher-level entity, it applies to all child entities within it. ## Best Practices We recommend separating access to data from the ability to grant privileges. To achieve this, the `security_admin` and `data_admin` roles divide the responsibilities of the initial `project_admin`, who has the authority to perform tasks in both areas. ## OpenFGA in Production When deploying OpenFGA in production environments, ensure you follow the [OpenFGA Production Checklist](https://openfga.dev/docs/best-practices/running-in-production). Lakekeeper includes [Query Consistency](https://openfga.dev/docs/interacting/consistency) specifications with each authorization request to OpenFGA. For most operations, `MINIMIZE_LATENCY` consistency provides optimal performance while maintaining sufficient data consistency guarantees. For medium to large-scale deployments, we strongly recommend enabling caching in OpenFGA and increasing the database connection pool limits. These optimizations significantly reduce database load and improve authorization latency. Configure the following environment variables in OpenFGA (written for version 1.14). You may increase the number of connections further if your database deployment can handle additional connections: ```sh OPENFGA_DATASTORE_MAX_OPEN_CONNS=200 OPENFGA_DATASTORE_MAX_IDLE_CONNS=100 OPENFGA_CACHE_CONTROLLER_ENABLED=true OPENFGA_CHECK_QUERY_CACHE_ENABLED=true OPENFGA_CHECK_ITERATOR_CACHE_ENABLED=true ``` ## Reconciling OpenFGA against the catalog The Postgres catalog is the source of truth for *which objects exist* (projects, warehouses, namespaces, tables, views, roles). OpenFGA stores the **structural hierarchy** between those objects plus all permissions (grants, ownership, role assignments). Under normal operation Lakekeeper keeps the two in sync on every API call. Drift can still happen — for example after a backup/restore where Postgres and OpenFGA were snapshotted at different times, after pointing Lakekeeper at a fresh OpenFGA store, or after a rare bug. The `lakekeeper openfga reconcile` subcommand rebuilds the structural hierarchy in OpenFGA from the Postgres catalog. ```sh # Add any hierarchy edges the catalog implies but OpenFGA is missing. # Purely additive; never deletes. Safe default. lakekeeper openfga reconcile --mode add-missing # Same plus delete structural tuples the catalog contradicts. lakekeeper openfga reconcile --mode add-and-delete-drift # Show the diff without writing anything. lakekeeper openfga reconcile --mode add-and-delete-drift --dry-run ``` ### What reconcile touches Reconcile only operates on the **structural** parts of the OpenFGA store: the parent/child edges between server, projects, warehouses, namespaces, tables, views, and roles. Ownership tuples, grants, role assignments, bootstrap admin tuples, and authorization-model bookkeeping are left alone. Tuples whose endpoints both refer to objects that *don't* exist in the catalog are also untouched — there is no anchor by which to interpret them. ### Operational notes - Run during a low-traffic window. Reconcile does **not** stop API writes; concurrent renames/creates/deletes can produce transient inconsistencies that self-heal on the next run. - A Postgres advisory lock prevents two reconciles from running at once. The second invocation fails fast with the lock key in the error message; you can confirm a held lock with `SELECT * FROM pg_locks WHERE locktype = 'advisory'`. - Use `--dry-run` first when you intend to delete drift. ## Switching to OpenFGA or replacing the store OpenFGA can be enabled, or its store replaced, on an already-bootstrapped Lakekeeper. Reconcile rebuilds the **structural hierarchy** from the catalog — but the initial server `admin` / `operator` tuple, ownership records, grants, and role assignments are **not** stored in the catalog and cannot be reconstructed from it. Lakekeeper's `/management/v1/bootstrap` endpoint runs only once per catalog by design; the `lakekeeper reopen-bootstrap` CLI re-opens it for cases like this without touching `server_id`, catalog data, or existing OpenFGA tuples. ### Procedure 1. Stand up an OpenFGA deployment. 2. Configure Lakekeeper for OpenFGA (`LAKEKEEPER__AUTHZ_BACKEND=openfga` plus the `LAKEKEEPER__OPENFGA__*` settings). 3. Run `lakekeeper migrate` once. This installs the OpenFGA authorization model in the configured store. 4. Run `lakekeeper openfga reconcile --mode add-missing` to seed the structural hierarchy from the catalog into OpenFGA. 5. Re-open bootstrap and seed the initial admin/operator via the API: ```sh lakekeeper reopen-bootstrap --yes ``` This flips `server.open_for_bootstrap` back to `true`. The `server_id` is preserved, so the hierarchy tuples written in step 4 remain valid. 6. Open the Lakekeeper UI and complete the bootstrap flow as the intended initial admin (or operator) — same path as a fresh deploy. 7. From that admin/operator account, recreate the role assignments and grants you need through the management API. If you exported tuples from the previous OpenFGA store, you can also selectively reimport them with the [fga CLI](https://github.com/openfga/cli) — reconcile leaves non-structural tuples alone. Switching *away* from OpenFGA (for example to Cedar) is not covered by reconcile and generally requires a new Lakekeeper instance. > Instance admins are useful as a parallel safety net while the OpenFGA store has no admin tuples: they can still manage projects, warehouses, namespaces, and tables. They do **not** confer data-plane access (`ReadData`, `WriteData`, view `Select`) and they **cannot** write to the OpenFGA permission-management endpoints — see [Instance Admins](./authorization.md#instance-admins). --- # Authorization Source: https://docs.lakekeeper.io/docs/latest/authorization/ # Authorization ## Overview Authentication verifies *who* you are, while authorization determines *what* you can do. Authorization can only be enabled if Authentication is enabled. Please check the [Authentication Docs](./authentication.md) for more information. Lakekeeper currently supports the following Authorizers: * **AllowAll**: A simple authorizer that allows all requests. This is mainly intended for development and testing purposes. * **OpenFGA**: A fine-grained authorization system based on the CNCF project [OpenFGA](https://openfga.dev). OpenFGA requires an additional OpenFGA service to be deployed (this is included in our self-contained examples and our helm charts). See the [Authorization with OpenFGA](./authorization-openfga.md) guide for details. * **Cedar**: An enterprise-grade policy-based authorization system based on [Cedar](https://cedarpolicy.com). The Cedar authorizer is built into Lakekeeper and requires no additional external services. See the [Authorization with Cedar](./authorization-cedar.md) guide for details. * **Custom**: Lakekeeper supports custom authorizers via the `Authorizer` trait. Check the [Authorization Configuration](./configuration.md#authorization) for setup details. ## Instance Admins *Available since Lakekeeper 0.12.1.* **Instance admins** are principals listed directly in Lakekeeper's static configuration. They bypass the configured Authorizer for administrative actions, so they can always manage the catalog — even if the Authorizer itself is broken or misconfigured. The typical instance admin is an automation account: a Kubernetes Operator reconciling Lakekeeper resources, for example, or an infrastructure admin responsible for operating the deployment. Without this mechanism, common failure modes would lock everyone out — for instance, deleting the last OpenFGA admin tuple, or deploying a Cedar policy that denies everything. ### Scope Instance admins bypass authorization for **control-plane** operations: - Bootstrap. - Project, role, warehouse, namespace management. - Table / view metadata operations, including `GetMetadata`, `Commit`, `Drop`, `Rename`, property changes. - User management. Instance admins do **not** bypass authorization for: - **Data-plane operations** — `CatalogTableAction::ReadData`, `CatalogTableAction::WriteData`, and `CatalogViewAction::Select` still route through the configured Authorizer. If the instance admin does not hold the relevant grants, reads and writes of table row data (and execution of views via the referenced-by chain) are denied. In the default OpenFGA model `Select` and `GetMetadata` resolve to the same underlying grant, so ordinary users see no behavioural change — the two exist as distinct actions so that the bypass carve-out can exclude `Select`. - **Role assumption** (`x-assume-role` header) — an instance admin must act with their own identity. Assuming a role opts into that role's narrower scope. - **Permission-management endpoints** exposed by the active Authorizer (for example `/management/v1/permissions/...` under OpenFGA; Cedar exposes its own set) — the instance-admin bypass does **not** apply to these. Writes go through the Authorizer's own grant-check path, so an instance admin cannot directly make Alice a `project_admin`. Ongoing permission administration stays with a principal that holds real grants in the configured Authorizer. This split keeps a leaked operator credential from being trivially used either to exfiltrate data or to escalate arbitrary principals to admin. ### Configuration Set `LAKEKEEPER__INSTANCE_ADMINS` to a **TOML inline array** of user IDs. For simple string arrays this is syntactically identical to a JSON array: ```yaml # e.g. in a Kubernetes deployment's env block env: - name: LAKEKEEPER__INSTANCE_ADMINS value: '["kubernetes~system:serviceaccount:lakekeeper:operator","oidc~alice"]' ``` Each entry is a Lakekeeper user ID of the form `~`. The `idp_id` matches the identifier of a configured Authenticator (for example, `kubernetes` or `oidc`). The `subject` is the resolved subject claim — for Kubernetes ServiceAccount tokens that is `system:serviceaccount::`; for OIDC it is whatever the configured subject claim produces. A bare string (e.g. `oidc~alice`) is **rejected** — even a single admin must be wrapped in brackets: `["oidc~alice"]`. The indexed-variable pattern that some other config systems accept (`LAKEKEEPER__INSTANCE_ADMINS__0=...`) is **not** supported. ### Operational notes - **Not a recovery mechanism.** If OpenFGA is unreachable or the authn layer is misconfigured such that the instance admin's identity cannot be resolved, the bypass does not engage. Instance admins are for day-to-day operator access, not break-glass recovery. - **Rotation.** The admin list is read once at process startup. Adding or removing an admin requires a redeploy. This is intentional: the mechanism is a deployment-config concern, not a runtime one. - **Audit.** Authorization events include a `privilege_source` field indicating how the decision was reached: `"internal"` (in-process call), `"instance_admin"` (config-granted bypass), or `"authorizer"` (configured Authorizer backend decision). See the [Logging guide](./logging.md#audit-logs-and-rust_log) for the event schema. - **Role-assumed requests.** Setting `x-assume-role` on a request from an instance admin drops the bypass for that request — the effective scope is whatever the assumed role holds. - **Permission administration.** Because instance admins cannot write to the OpenFGA permission-management endpoints, day-to-day management of role grants and assignments is done by a human (or service) principal that was bootstrapped through OpenFGA. The operator use case is provisioning (creating projects/warehouses, initial bootstrap), not ongoing user administration. --- # Bootstrap / Initialize Source: https://docs.lakekeeper.io/docs/latest/bootstrap/ # Bootstrap / Initialize After the initial deployment, Lakekeeper needs to be bootstrapped. This can be done via the UI or the `/management/v1/bootstrap` endpoint. A typical POST request to bootstrap Lakekeeper looks like this: ```bash curl --location 'https:///management/v1/bootstrap' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "accept-terms-of-use": true }' ``` `` is obtained by logging into the IdP before bootstrapping Lakekeeper. If authentication is disabled, no token is required. Lakekeeper can only be bootstrapped once. During bootstrapping, Lakekeeper performs the following actions: * Grants the server's `admin` role to the user performing the POST request. The user is identified by their token. If authentication is disabled, the `Authorization` header is not required, and no `admin` is set, as permissions are disabled in this case. * Stores the current [Server ID](./concepts.md#server) to prevent unwanted future changes of the Server ID that would break permissions. * Accepts terms of use as defined by our [License](../../about/license.md). * If `LAKEKEEPER__ENABLE_DEFAULT_PROJECT` is enabled (default), a default project with the NIL Project ID ("00000000-0000-0000-0000-000000000000") is created. If the initial user is a technical user (e.g., a Kubernetes Operator) managing the Lakekeeper deployment, the `admin` role might not be sufficient as it limits access to projects until the `admin` grants themselves permission. For technical users, the `operator` role grants full access to all APIs and can be obtained by adding `"is-operator": true` to the JSON body of the bootstrap request. --- # Concepts Source: https://docs.lakekeeper.io/docs/latest/concepts/ # Concepts ## Architecture Lakekeeper is an implementation of the Apache Iceberg REST Catalog API. Lakekeeper depends on the following, partially optional, external dependencies:
![Lakekeeper Overview](../../assets/interfaces-v2.svg){ width="100%" }
Connected systems. Green boxes are recommended for production.
* **Persistence Backend / Catalog** (required): We currently support only Postgres, but plan to expand our support to more Databases in the future. * **Warehouse Storage** (required): When a new Warehouse is created, storage credentials are required. * **Identity Provider** (optional): Lakekeeper can authenticate incoming requests using any OIDC capable Identity Provider (IdP). Lakekeeper can also natively authenticate kubernetes service accounts. * **Authorization System** (optional): For permission management, Lakekeeper supports different Authorizers. Please refer to the [Authorization Documentation](./authorization.md) for more information. * **Secret Store** (required): Lakekeeper requires a Secret Store to stores secrets such as Warehouse credentials. By default, Lakekeeper uses the default Postgres connection to store encrypted secrets. To increase security, Lakekeeper can also use external systems to store secrets. Currently all Hashicorp-Vault like stores are supported. * **Event Store** (optional): Lakekeeper can send Change Events to an Event Store. We support [NATS](http://nats.io) and [Apache Kafka](http://kafka.apache.org) * **Data Contract System** (optional): Lakekeeper can interface with external data contract systems to prohibit breaking changes to your tables. To get started quickly with the latest version of Lakekeeper check our [Getting Started Guide](../../getting-started.md). ## Identifier Case Sensitivity All entity names in Lakekeeper — including Projects, Warehouses, Namespaces, Tables, Views, and Roles — are **case-insensitive but case-preserving**: - **Case-insensitive matching**: Looking up `my_table`, `My_Table`, or `MY_TABLE` all resolve to the same entity. This applies to all operations: reads, writes, renames, drops, and listing. - **Case-preserving storage**: The name you provide at creation time is stored exactly as given. Lakekeeper does not normalize names to lowercase. - **No case-only duplicates**: You cannot create two entities whose names differ only in case within the same scope. For example, creating namespace `Analytics` and then `analytics` in the same warehouse will fail with a conflict error. - **Requested case in responses**: API responses return entity names using the case from the *request*, not the case stored in the database. For example, if a table was created as `my_table` and you query for `MY_TABLE`, the response will contain `MY_TABLE`. This behavior is implemented via PostgreSQL's ICU collation (`und-u-ks-level2`) on all identifier columns and is transparent to all query engines — no client-side configuration is needed. ### Why this design? Query engines disagree on identifier case — and they disagree in the worst possible way. Some fold to lowercase, one folds to **uppercase**, and some preserve case exactly. Without a case-insensitive catalog, a table created by one engine can become invisible to another: | Engine | Unquoted identifiers | Quoted identifiers | Sent to catalog | |--------|---------------------|--------------------|-----------------| | Spark | Lowercased | Backticks preserve case | Lowercase | | Trino / Starburst | Lowercased | `"..."` preserves case | Lowercase (unquoted), preserved (quoted) | | DuckDB | Lowercased | `"..."` preserves case | Lowercase (unquoted), preserved (quoted) | | StarRocks | Lowercased | Generally lowercased | Lowercase | | RisingWave | Lowercased | `"..."` preserves case | Lowercase (unquoted), preserved (quoted) | | Athena (Spark) | **Always lowercased** | **Quoted still lowercased** | **Always lowercase** | | Snowflake | **Uppercased** | `"..."` preserves case | **Uppercase** (unquoted), preserved (quoted) | | PyIceberg | N/A (programmatic) | N/A | **Exactly as provided** | Notice the conflict: Spark sends `monthly_revenue`, Snowflake sends `MONTHLY_REVENUE`, and PyIceberg sends whatever the user typed. With a case-sensitive catalog, **the same table has three different names depending on which engine created it** — and none of them can find each other's tables. Consider this real-world scenario with a case-sensitive catalog: 1. A data engineer creates a table via PyIceberg: `catalog.create_table("Reporting.Monthly Revenue", schema)` — stored as `Monthly Revenue`. 2. An analyst in Spark runs `SELECT * FROM reporting.monthly_revenue` — Spark sends `monthly_revenue`. **:x: Table not found.** 3. A BI team queries from Snowflake: `SELECT * FROM reporting.monthly_revenue` — Snowflake sends `MONTHLY_REVENUE`. **:x: Table not found.** 4. Even the original engineer, switching to Trino, tries `SELECT * FROM reporting."Monthly Revenue"` — Trino lowercases the unquoted parts. **:x: Table not found** (unless quoted exactly right). The table exists. It has data. Yet three out of four engines cannot see it. The only way to access it is to know the exact case used at creation time and quote it precisely — fragile and error-prone. **With Lakekeeper, all four queries find the table instantly.** No quoting tricks, no configuration, no coordination between teams about naming conventions. #### Why not just lowercase everything? An alternative approach is to normalize all identifiers to lowercase on the catalog side. This solves interoperability but destroys information. If a data team carefully names their gold layer tables `Monthly Revenue`, `Customer Events`, or `Order Line Items`, a normalizing catalog turns them all into `monthly revenue`, `customer events`, and `order line items`. The intent behind the original naming is lost. This matters most for **UI and management tools**. Dashboards, the Lakekeeper UI, data catalogs, and lineage tools all display table names to humans. `Monthly Revenue` is immediately readable; `monthly revenue` is slightly worse; and once you have hundreds of tables, the difference between well-cased names and a wall of lowercase text adds up. By preserving the original case, Lakekeeper lets teams maintain meaningful, readable names while every query engine can still access them with whatever case it uses internally. You get proper casing for humans and seamless access for machines — without compromise. | Scenario | Lakekeeper | Case-sensitive catalog | |----------|:----------:|:----------------------:| | Spark reads table created by PyIceberg | :white_check_mark: | :x: | | Snowflake reads table created by Spark | :white_check_mark: | :x: | | Spark reads table created by Snowflake | :white_check_mark: | :x: | | Trino reads table created by Spark | :white_check_mark: | :white_check_mark: | | PyIceberg reads table created by Spark | :white_check_mark: | :white_check_mark: (1) | | Trino with quoted `"MyTable"` finds Spark's `mytable` | :white_check_mark: | :x: | | Athena reads table created by any engine | :white_check_mark: | :x: (2) | | UI shows `Monthly Revenue` + all engines can query it | :white_check_mark: | :x: | (1) Only if PyIceberg uses the exact same case that the creating engine sent (typically lowercase). (2) Athena always lowercases identifiers, including quoted ones. Tables created with mixed case via PyIceberg or quoted identifiers in Trino are unreachable from Athena in a case-sensitive catalog. ## Entity Hierarchy In addition to entities defined in the Apache Iceberg specification or the REST specification (Namespaces, Tables, etc.), Lakekeeper introduces new entities for permission management and multi-tenant setups. The following entities are available in Lakekeeper:
![Lakekeeper Entity Hierarchy](../../assets/entity-hierarchy-v1.svg){ width="100%" }
Lakekeeper Entity Hierarchy

Project, Server, User and Roles are entities unknown to the Iceberg Rest Specification. Lakekeeper serves two APIs: 1. The Iceberg REST API is served at endpoints prefixed with `/catalog`. External query engines connect to this API to interact with the Lakekeeper. Lakekeeper also implements the S3 remote signing API which is hosted at `//v1/aws/s3/sign`. 1. The Lakekeeper Management API is served at endpoints prefixed with `/management`. It is used to configure Lakekeeper and manage entities that are not part of the Iceberg REST Catalog specification, such as permissions. ### Server The Server is the highest entity in Lakekeeper, representing a single instance or a cluster of Lakekeeper pods sharing a common state. Each server has a unique identifier (UUID). The Server ID is generated randomly on first startup and stored in the Database Backend. ### Project For single-company setups, we recommend using a single Project setup, which is the default. Unless `LAKEKEEPER__ENABLE_DEFAULT_PROJECT` is explicitly set to `false`, a default project is created during [bootstrapping](./bootstrap.md) with the nil UUID. ### Warehouse Each Project can contain multiple Warehouses. Query engines connect to Lakekeeper by specifying a Warehouse name in the connection configuration. Each Warehouse is associated with a unique location on object stores. Never share locations between Warehouses to ensure no data is leaked via vended credentials. Each Warehouse stores information on how to connect to its location via a `storage-profile` and an optional `storage-credential`. Warehouses can be configured to use [Soft-Deletes](./concepts.md#soft-deletion). When enabled, tables are not eagerly deleted but kept in a deleted state for a configurable amount of time. During this time, they can be restored. Please note that Warehouses and Namespaces cannot be deleted via the `/catalog` API if child objects are present. This includes soft-deleted Tables. A cascade-drop API is added in one of the next releases as part of the `/management` API. ### Namespaces Each Warehouses can contain multiple Namespaces. Namespaces can be nested and serve as containers for Namespaces, Tables and Views. Using the `/catalog` API, a Namespace cannot be dropped unless it is empty. A cascade-drop API is added in one of the next releases as part of the `/management` API. ### Tables & Views Each Namespace can contain multiple Tables and Views. When creating new Tables and Views, we recommend to not specify the `location` explicitly. If locations are specified explicitly, the location must be a valid sub location of the `storage-profile` of the Warehouse - this is validated by Lakekeeper upon creation. Lakekeeper also ensures that there are no Tables or Views that use a parent- or sub-folder as their `location` and that the location is empty on creation. These checks are required to ensure that no data is leaked via vended-credentials. ### Users Lakekeeper is no Identity Provider. The identities of users are exclusively managed via an external Identity Provider to ensure compliance with basic security standards. Lakekeeper does not store any Password / Certificates / API Keys or any other secret that grants access to data for users. Instead, we only store Name, Email and type of users with the sole purpose of providing a convenient search while assigning privileges. Users can be provisioned to Lakekeeper by either of the following endpoints: * Explicit user creation via the POST `/management/user` endpoint. This endpoint is called automatically by the UI upon login. Thus, users are "searchable" after their first login to the UI. * Implicit on-the-fly creation when calling GET `/catalog/v1/config`. This can be used to register technical users simply by connecting to the Lakekeeper with your favorite tool (i.e. Spark). The initial connection will probably fail because privileges are missing to use this endpoint, but the user is provisioned anyway so that privileges can be assigned before re-connecting. ### Roles Projects can contain multiple Roles, allowing Roles to be reused in all Warehouses within the Project. Roles can be nested arbitrarily, meaning that a role can contain other roles within it. Roles can be provisioned automatically using the `/management/v1/role` endpoint or manually created via the UI. We are looking into SCIM support to simplify role provisioning. Please consider upvoting the corresponding [GitHub Issue](https://github.com/lakekeeper/lakekeeper/issues/497) if this would be of interest to you. ## Dropping Tables Currently all tables stored in Lakekeeper are assumed to be managed by Lakekeeper. The concept of "external" tables will follow in a later release. When managed tables are dropped, Lakekeeper defaults to setting `purgeRequested` parameter of the `dropTable` endpoint to true unless explicitly set to false. Currently most query engines do not set this flag, which defaults to enabling purge. If purge is enabled for a drop, all files of the table are removed. ## Soft Deletion Lakekeeper allows warehouses to enable soft deletion as a data protection mechanism. When enabled: - Tables and views aren't immediately removed from the catalog when dropped - Instead, they're marked as deleted and scheduled for cleanup - The data remains recoverable until the configured expiration period elapses - Recovery is only possible for warehouses with soft deletion enabled - The expiration delay is fixed at the time of dropping - changing warehouse settings only affects newly dropped tables Soft deletion works correctly only when clients follow these behaviors: 1. `DROP TABLE xyz` (standard): Clients should not remove any files themselves, and should call the `dropTable` endpoint without the `purgeRequested` flag. Lakekeeper handles file removal for managed tables. This works well with all query engines. 2. `DROP TABLE xyz PURGE`: Clients should not delete files themselves, and should call the `dropTable` endpoint with the `purgeRequested` flag set to true. Lakekeeper will remove files for managed tables (and for unmanaged tables in a future release). Unfortunately not all query engines adhere to this behavior, as described below. Unfortunately, some Java-based query engines like Spark don't follow the expected behavior for `PURGE` operations. Instead, they immediately delete files, which undermines soft deletion functionality. The Apache Iceberg community has [agreed to fix this in Iceberg 2.0](https://github.com/apache/iceberg/pull/11317#issuecomment-2604912801). For Iceberg 1.x versions, we're working on a new `io.client-side.purge-enabled` flag for better control. !!! warning Never use **`DROP TABLE xyz PURGE`** with clients like Spark that immediately remove files when soft deletion is enabled! For S3-based storage, Lakekeeper provides a protective configuration option in storage profiles: `push-s3-delete-disabled`. When set to `true`, this: - Prevents clients from deleting files by pushing the `s3.delete-enabled: false` setting to clients - Preserves soft deletion functionality even when `PURGE` is specified - Affects all file deletion operations, including maintenance procedures like `expire_snapshots` When running table maintenance procedures that need to remove files with `push-s3-delete-disabled: true`, you must explicitly override with `s3.delete-enabled: true` in your client configuration: ```python import pyspark import pyspark.sql pyspark_version = pyspark.__version__ pyspark_version = ".".join(pyspark_version.split(".")[:2]) # Strip patch version iceberg_version = "1.10.1" # Disable the jars which are not needed spark_jars_packages = ( f"org.apache.iceberg:iceberg-spark-runtime-{pyspark_version}_2.12:{iceberg_version}," f"org.apache.iceberg:iceberg-aws-bundle:{iceberg_version}," ) catalog_name = "lakekeeper" configuration = { "spark.jars.packages": spark_jars_packages, "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.defaultCatalog": catalog_name, f"spark.sql.catalog.{catalog_name}": "org.apache.iceberg.spark.SparkCatalog", f"spark.sql.catalog.{catalog_name}.catalog-impl": "org.apache.iceberg.rest.RESTCatalog", f"spark.sql.catalog.{catalog_name}.uri": "", # ... Additional configuration options # THE FOLLOWING IS THE NEW OPTION: # Enabling s3 deletion explicitly - this overrides any Lakekeeper setting f"spark.sql.catalog.{catalog_name}.s3.delete-enabled": "true", } spark_conf = pyspark.SparkConf().setMaster("local[*]") for k, v in configuration.items(): spark_conf = spark_conf.set(k, v) spark = pyspark.sql.SparkSession.builder.config(conf=spark_conf).getOrCreate() spark.sql(f"USE {catalog_name}") ``` ## Protection and Deletion Mechanisms in Lakekeeper Lakekeeper provides several complementary mechanisms for protecting data assets and managing their deletion while balancing flexibility and data governance. ### Protection Protection prevents accidental deletion of important entities in Lakekeeper. When an entity is protected, attempts to delete it through standard API calls will be rejected. Protection can be applied to Warehouses, Namespaces, Tables, and Views via the Management API. ### Recursive Deletion on Namespaces By default, Lakekeeper enforces that namespaces must be empty before deletion. Recursive deletion provides a way to delete a namespace and all its contained entities in a single operation. When deleting a namespace, add the recursive=true query parameter to the request. Protected entities within the hierarchy will prevent recursive deletion unless force is also used. ### Force Deletion Force deletion is an administrative override that allows deletion of protected entities and bypasses certain safety checks: - Bypasses protection settings - Overrides soft-deletion mechanisms for immediate hard deletion Add the `force=true` query parameter to deletion requests: ``` DELETE /catalog/v1/{prefix}/namespaces/{namespace}?force=true ``` Force can be combined with recursive deletion (`recursive=true&force=true`) to delete an entire protected hierarchy. The `purgeRequested` flag for tables is still respected and determines if the physical data of the table should be removed. Purge defaults to true for tables managed by Lakekeeper. ## Upgrades & Migration Lakekeeper relies on a persistent backend (Postgres) and an optional authorization system (OpenFGA). As Lakekeeper evolves, these systems may need schema or configuration updates to support new features and improvements. The `lakekeeper migrate` command initializes and updates both Postgres schemas (creating necessary tables and structures) and authorization models to ensure compatibility with your current Lakekeeper version. **Migration is required before each Lakekeeper upgrade.** You must run the migration before starting the `lakekeeper serve` command to ensure all system components are properly updated and configured. Without running the migration first, the `lakekeeper serve` command will fail to start with the error: "Database is not up to date with binary, make sure to run the migrate command before starting the server." Migrations are designed to be resilient - you can safely skip intermediate versions and migrate directly to your target version. If the system is already up to date, the migration command will exit immediately without making any changes. **All migrations run within a transaction,** ensuring that either the entire migration completes successfully or the database remains unchanged. This prevents partial migrations that could leave your system in an inconsistent state. **Always create a backup of your Postgres database before running migrations.** While migrations are designed to be safe, having a backup ensures you can restore your system to a known good state if needed. When using the Lakekeeper Helm Chart, migrations are handled automatically through a dedicated job during deployment. --- # Configuration Source: https://docs.lakekeeper.io/docs/latest/configuration/ # Configuration Lakekeeper is configured via environment variables. Settings listed in this page are shared between all projects and warehouses. Previous to Lakekeeper Version `0.5.0` please prefix all environment variables with `ICEBERG_REST__` instead of `LAKEKEEPER__`. For most deployments, we recommend to set at least the following variables: `LAKEKEEPER__PG_DATABASE_URL_READ`, `LAKEKEEPER__PG_DATABASE_URL_WRITE`, `LAKEKEEPER__PG_ENCRYPTION_KEY`. ## Routing and Base-URL Some Lakekeeper endpoints return links pointing at Lakekeeper itself. By default, these links are generated using the `x-forwarded-host`, `x-forwarded-proto`, `x-forwarded-port` and `x-forwarded-prefix` headers, if these are not present, the `host` header is used. If this is not working for you, you may set the `LAKEKEEPER_BASE_URI` environment variable to the base-URL where Lakekeeper is externally reachable. This may be necessary if Lakekeeper runs behind a reverse proxy or load balancer, and you cannot set the headers accordingly. In general, we recommend relying on the headers. To respect the `host` header but not the `x-forwarded-` headers, set `LAKEKEEPER__USE_X_FORWARDED_HEADERS` to `false`. ### General | Variable | Example | Description | |----------------------------------------------------|----------------------------------------|-----| | `LAKEKEEPER__BASE_URI` | `https://example.com:8181` | Optional base-URL where the catalog is externally reachable. Default: `None`. See [Routing and Base-URL](#routing-and-base-url). | | `LAKEKEEPER__ENABLE_DEFAULT_PROJECT` | `true` | If `true`, the NIL Project ID ("00000000-0000-0000-0000-000000000000") is used as a default if the user does not specify a project when connecting. This option is enabled by default, which we recommend for all single-project (single-tenant) setups. Default: `true`. | | `LAKEKEEPER__RESERVED_NAMESPACES` | `system,examples,information_schema` | Reserved Namespaces that cannot be created via the REST interface | | `LAKEKEEPER__METRICS__PORT` | `9000` | Port where the Prometheus metrics endpoint is reachable. Default: `9000` | | `LAKEKEEPER__LISTEN_PORT` | `8181` | Port Lakekeeper listens on. Default: `8181` | | `LAKEKEEPER__BIND_IP` | `0.0.0.0`, `::1`, `::` | IP Address Lakekeeper binds to. Default: `0.0.0.0` (listen to all incoming IPv4 packages) | | `LAKEKEEPER__SECRET_BACKEND` | `postgres` | The secret backend to use. If `kv2` (Hashicorp KV Version 2) is chosen, you need to provide [additional parameters](#vault-kv-version-2) Default: `postgres`, one-of: [`postgres`, `kv2`] | | `LAKEKEEPER__SERVE_SWAGGER_UI` | `true` | If `true`, Lakekeeper serves a swagger UI for management & catalog openAPI specs under `/swagger-ui` | | `LAKEKEEPER__ALLOW_ORIGIN` | `*` | A comma separated list of allowed origins for CORS. | | `LAKEKEEPER__USE_X_FORWARDED_HEADERS` | `false` | If true, Lakekeeper respects the `x-forwarded-host`, `x-forwarded-proto`, `x-forwarded-port` and `x-forwarded-prefix` headers in incoming requests. This is mostly relevant for the `/config` endpoint. Default: `true` (Headers are respected.) | ### Pagination Lakekeeper has default values for `default` and `max` page sizes of paginated queries. These are safeguards against malicious requests and the problems related to large page sizes described below. The REST catalog [spec](https://github.com/apache/iceberg/blob/404c8057275c9cfe204f2c7cc61114c128fbf759/open-api/rest-catalog-open-api.yaml#L2030-L2032) requires servers to return *all* results if `pageToken` is not set in the request. To obtain that behavior, set `LAKEKEEPER__PAGINATION_SIZE_MAX` to 4294967295, which corresponds to `u32::MAX`. Larger page sizes would lead to practical problems. Things to keep in mind: - Retrieving huge numbers of rows is expensive, which might be exploited by malicious requests. - Requests may time out or responses may exceed size limits for huge numbers of results. | Variable | Example | Description | |---------------------------------------------------|--------------------|-----| | `LAKEKEEPER__PAGINATION_SIZE_DEFAULT` | `1024` | The default page size used for paginated queries. This value is used if the request's `pageToken` is set but empty. Default: `100` | | `LAKEKEEPER__PAGINATION_SIZE_MAX` | `2048` | The max page size used for paginated queries. This value is used if the request's `pageToken` is not set. Default: `1000` | ### Storage | Variable | Example | Description | |-------------------------------------------------------------|--------------------|-----| | LAKEKEEPER__
ENABLE_AWS_SYSTEM_CREDENTIALS | `true` | Lakekeeper supports using AWS system identities (i.e. through `AWS_*` environment variables or EC2 instance profiles) as storage credentials for warehouses. This feature is disabled by default to prevent accidental access to restricted storage locations. To enable AWS system identities, set LAKEKEEPER__
ENABLE_AWS_SYSTEM_CREDENTIALS to `true`. Default: `false` (AWS system credentials disabled) | | LAKEKEEPER__
S3_ENABLE_DIRECT_SYSTEM_CREDENTIALS | `true` | By default, when using AWS system credentials, users must specify an `assume-role-arn` for Lakekeeper to assume when accessing S3. Setting this option to `true` allows Lakekeeper to use system credentials directly without role assumption, meaning the system identity must have direct access to warehouse locations. Default: `false` (direct system credential access disabled) | | LAKEKEEPER__
S3_REQUIRE_EXTERNAL_ID_FOR_SYSTEM_CREDENTIALS | `true` | Controls whether an `external-id` is required when assuming a role with AWS system credentials. External IDs provide additional security when cross-account role assumption is used. Default: true (external ID required) | | LAKEKEEPER__
ENABLE_AZURE_SYSTEM_CREDENTIALS | `true` | Lakekeeper supports using Azure system identities (i.e. through `AZURE_*` environment variables or VM managed identities) as storage credentials for warehouses. This feature is disabled by default to prevent accidental access to restricted storage locations. To enable Azure system identities, set LAKEKEEPER__
ENABLE_AZURE_SYSTEM_CREDENTIALS to `true`. Default: `false` (Azure system credentials disabled) | | LAKEKEEPER__
ENABLE_GCP_SYSTEM_CREDENTIALS | `true` | Lakekeeper supports using GCP system identities (i.e. through `GOOGLE_APPLICATION_CREDENTIALS` environment variables or the Compute Engine Metadata Server) as storage credentials for warehouses. This feature is disabled by default to prevent accidental access to restricted storage locations. To enable GCP system identities, set LAKEKEEPER__
ENABLE_GCP_SYSTEM_CREDENTIALS to `true`. Default: `false` (GCP system credentials disabled) | ### Persistence Store Currently Lakekeeper supports only Postgres as a persistence store. You may either provide connection strings using `PG_DATABASE_URL_*` or use the `PG_*` environment variables. Connection strings take precedence. Postgres needs to be Version 15 or higher. Lakekeeper supports configuring separate database URLs for read and write operations, allowing you to utilize read replicas for better scalability. By directing read queries to dedicated replicas via `LAKEKEEPER__PG_DATABASE_URL_READ`, you can significantly reduce load on your database primary (specified by `LAKEKEEPER__PG_DATABASE_URL_WRITE`), improving overall system performance as your deployment scales. This separation is particularly beneficial for read-heavy workloads. When using read replicas, be aware that replication lag may occur between the primary and replica databases depending on your Database setup. This means that immediately after a write operation, the changes might not be instantly visible when querying a read-only Lakekeeper endpoint (which uses the read replica). Consider this potential lag when designing applications that require immediate read-after-write consistency. For deployments where read-after-write consistency is critical, you can simply omit the `LAKEKEEPER__PG_DATABASE_URL_READ` setting, which will cause all operations to use the primary database connection. | Variable | Example | Description | |--------------------------------------------------------|-------------------------------------------------------|-----| | `LAKEKEEPER__PG_DATABASE_URL_READ` | `postgres://postgres:password@localhost:5432/iceberg` | Postgres Database connection string used for reading. Defaults to `LAKEKEEPER__PG_DATABASE_URL_WRITE`. | | `LAKEKEEPER__PG_DATABASE_URL_WRITE` | `postgres://postgres:password@localhost:5432/iceberg` | Postgres Database connection string used for writing. If `LAKEKEEPER__PG_DATABASE_URL_READ` is not specified, this connection is also used for reading. | | `LAKEKEEPER__PG_ENCRYPTION_KEY` | `This is unsafe, please set a proper key` | If `LAKEKEEPER__SECRET_BACKEND=postgres`, this key is used to encrypt secrets. It is required to change this for production deployments. | | `LAKEKEEPER__PG_READ_POOL_CONNECTIONS` | `10` | Number of connections in the read pool | | `LAKEKEEPER__PG_WRITE_POOL_CONNECTIONS` | `5` | Number of connections in the write pool | | `LAKEKEEPER__PG_HOST_R` | `localhost` | Hostname for read operations. Defaults to `LAKEKEEPER__PG_HOST_W`. | | `LAKEKEEPER__PG_HOST_W` | `localhost` | Hostname for write operations | | `LAKEKEEPER__PG_PORT` | `5432` | Port number | | `LAKEKEEPER__PG_USER` | `postgres` | Username for authentication | | `LAKEKEEPER__PG_PASSWORD` | `password` | Password for authentication | | `LAKEKEEPER__PG_DATABASE` | `iceberg` | Database name | | `LAKEKEEPER__PG_SSL_MODE` | `require` | SSL mode (disable, allow, prefer, require) | | `LAKEKEEPER__PG_SSL_ROOT_CERT` | `/path/to/root/cert` | Path to SSL root certificate | | `LAKEKEEPER__PG_ENABLE_STATEMENT_LOGGING` | `true` | Enable SQL statement logging | | `LAKEKEEPER__PG_TEST_BEFORE_ACQUIRE` | `true` | Test connections before acquiring from the pool | | `LAKEKEEPER__PG_CONNECTION_MAX_LIFETIME` | `1800` | Maximum lifetime of connections in seconds | | `LAKEKEEPER__PG_ACQUIRE_TIMEOUT` | `10` | Timeout to acquire a new postgres connection in seconds. Default: `5` | #### Required Postgres extensions Lakekeeper migrations require the following extensions: `uuid-ossp`, `pgcrypto`, `pg_trgm`, `btree_gin`, `btree_gist`. They are part of the standard `postgresql-contrib` package and are pre-installed on most managed Postgres offerings (AWS RDS, Cloud SQL, Azure Database, etc.). If the role Lakekeeper connects as has `CREATE` privilege on the database, the migrations will create the extensions automatically. Otherwise — for example, when running Lakekeeper as a low-privilege role (see below) — an administrator must pre-create them once per database: ```sql CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; CREATE EXTENSION IF NOT EXISTS "pgcrypto"; CREATE EXTENSION IF NOT EXISTS "pg_trgm"; CREATE EXTENSION IF NOT EXISTS "btree_gin"; CREATE EXTENSION IF NOT EXISTS "btree_gist"; ``` #### Using a non-`public` Postgres schema By default Lakekeeper creates its tables in whichever schema Postgres resolves via `search_path` — typically `public`. To install it into a dedicated schema (e.g. for tenant isolation or policies that disallow DDL in `public`), set the default `search_path` on the role Lakekeeper connects as, server-side: ```sql CREATE SCHEMA IF NOT EXISTS lakekeeper AUTHORIZATION lakekeeper; ALTER ROLE lakekeeper SET search_path = lakekeeper, public; ``` Postgres applies this before any query runs on a new session, so both migrations and runtime queries land in `lakekeeper`. Keep `public` in `search_path` so functions installed by extensions there (e.g. `uuid_generate_v1mc` from `uuid-ossp`) still resolve. If you use separate roles for `LAKEKEEPER__PG_DATABASE_URL_READ` and `LAKEKEEPER__PG_DATABASE_URL_WRITE`, run `ALTER ROLE` for both. Setting `search_path` via the URL `options` parameter is fragile — encoding pitfalls and connection poolers (e.g. PgBouncer in transaction pooling mode) often drop it — so the role-level default is the recommended approach. ### Vault KV Version 2 Configuration parameters if a Vault KV version 2 (i.e. Hashicorp Vault) compatible storage is used as a backend. Currently, we only support the `userpass` authentication method. Configuration may be passed as single values like `LAKEKEEPER__KV2__URL=http://vault.local` or as a compound value: `LAKEKEEPER__KV2='{url="http://localhost:1234", user="test", password="test", secret_mount="secret"}'` | Variable | Example | Description | |----------------------------------------------|-----------------------|-------| | `LAKEKEEPER__KV2__URL` | `https://vault.local` | URL of the KV2 backend | | `LAKEKEEPER__KV2__USER` | `admin` | Username to authenticate against the KV2 backend | | `LAKEKEEPER__KV2__PASSWORD` | `password` | Password to authenticate against the KV2 backend | | `LAKEKEEPER__KV2__SECRET_MOUNT` | `kv/data/iceberg` | Path to the secret mount in the KV2 backend | ### Task Queues Lakekeeper uses task queues internally to remove soft-deleted tabulars and purge tabular files. The following global configuration options are available: | Variable | Example | Description | |-----------------------------------------------------------------------------------|------------|-----| | `LAKEKEEPER__TASK_POLL_INTERVAL` | 3600ms/30s | Interval between polling for new tasks. Default: 10s. Supported units: ms (milliseconds) and s (seconds), leaving the unit out is deprecated, it'll default to seconds but is due to be removed in a future release. | | LAKEKEEPER__
TASK_TABULAR_EXPIRATION_WORKERS | 2 | Number of workers spawned to expire soft-deleted tables and views. | | `LAKEKEEPER__TASK_TABULAR_PURGE_WORKERS` | 2 | Number of workers spawned to purge table files after dropping a table with the purge option. | | LAKEKEEPER__
TASK_EXPIRE_SNAPSHOTS_WORKERS | 2 | Number of workers spawned that work on expire Snapshots tasks. See [Expire Snapshots Docs](./table-maintenance.md#expire-snapshots) for more information. | ### NATS Lakekeeper can publish change events to NATS. The following configuration options are available: | Variable | Example | Description | |--------------------------------------------|-------------------------|-------| | `LAKEKEEPER__NATS_ADDRESS` | `nats://localhost:4222` | The URL of the NATS server to connect to | | `LAKEKEEPER__NATS_TOPIC` | `iceberg` | The subject to publish events to | | `LAKEKEEPER__NATS_USER` | `test-user` | User to authenticate against NATS, needs `LAKEKEEPER__NATS_PASSWORD` | | `LAKEKEEPER__NATS_PASSWORD` | `test-password` | Password to authenticate against nats, needs `LAKEKEEPER__NATS_USER` | | `LAKEKEEPER__NATS_CREDS_FILE` | `/path/to/file.creds` | Path to a file containing NATS credentials | | `LAKEKEEPER__NATS_TOKEN` | `xyz` | NATS token to use for authentication | ### Kafka Lakekeeper uses [rust-rdkafka](https://github.com/fede1024/rust-rdkafka) to enable publishing events to Kafka. The following features of rust-rdkafka are enabled: - tokio - ztstd - gssapi-vendored - curl-static - ssl-vendored - libz-static This means that all features of [librdkafka](https://github.com/confluentinc/librdkafka) are usable. All necessary dependencies are statically linked and cannot be disabled. If you want to use dynamic linking or disable a feature, you'll have to fork Lakekeeper and change the features accordingly. Please refer to the documentation of rust-rdkafka for details on how to enable dynamic linking or disable certain features. To publish events to Kafka, set the following environment variables: | Variable | Example | Description | |----------------------------------------------|---------------------------------------------------------------------------|-----| | `LAKEKEEPER__KAFKA_TOPIC` | `lakekeeper` | The topic to which events are published | | `LAKEKEEPER__KAFKA_CONFIG` | `{"bootstrap.servers"="host1:port,host2:port","security.protocol"="SSL"}` | [librdkafka Configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) as "Dictionary". Note that you cannot use "JSON-Style-Syntax". Also see notes below | | `LAKEKEEPER__KAFKA_CONFIG_FILE` | `/path/to/config_file` | [librdkafka Configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md) to be loaded from a file. Also see notes below | ##### Notes `LAKEKEEPER__KAFKA_CONFIG` and `LAKEKEEPER__KAFKA_CONFIG_FILE` are mutually exclusive and the values are not merged, if both variables are set. In case that both are set, `LAKEKEEPER__KAFKA_CONFIG` is used. A `LAKEKEEPER__KAFKA_CONFIG_FILE` could look like this: ``` { "bootstrap.servers"="host1:port,host2:port", "security.protocol"="SASL_SSL", "sasl.mechanisms"="PLAIN", } ``` Checking configuration parameters is deferred to `rdkafka` ### Logging Cloudevents Cloudevents can also be logged, if you do not have Nats or Kafka up and running. This feature can be enabled by setting `LAKEKEEPER__LOG_CLOUDEVENTS=true` ### Authentication To prohibit unwanted access to data, we recommend to enable Authentication. Authentication is enabled if: * `LAKEKEEPER__OPENID_PROVIDER_URI` is set OR * `LAKEKEEPER__OPENID_PROVIDERS` has at least one configured provider OR * `LAKEKEEPER__ENABLE_KUBERNETES_AUTHENTICATION` is set to true In Lakekeeper multiple Authentication mechanisms can be enabled together, for example OpenID + Kubernetes. Lakekeeper builds an internal Authenticator chain of up to three identity providers. Incoming tokens need to be JWT tokens - Opaque tokens are not yet supported. Incoming tokens are introspected, and each Authentication provider checks if the given token can be handled by this provider. If it can be handled, the token is authenticated against this provider, otherwise the next Authenticator in the chain is checked. The following Authenticators are available. Enabled Authenticators are checked in order: 1. **OpenID / OAuth2**
**Enabled if:** `LAKEKEEPER__OPENID_PROVIDER_URI` is set
**Validates Token with:** Locally with JWKS Keys fetched from the well-known configuration.
**Accepts JWT if** (both must be true):
- Issuer matches the issuer provided in the `.well-known/openid-configuration` of the `LAKEKEEPER__OPENID_PROVIDER_URI` OR issuer matches any of the `LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS`.
- If `LAKEKEEPER__OPENID_AUDIENCE` is specified, any of the configured audiences must be present in the token
1. **Kubernetes**
**Enabled if:** `LAKEKEEPER__ENABLE_KUBERNETES_AUTHENTICATION` is true
**Validates Token with:** Kubernetes `TokenReview` API **Accepts JWT if:**
- Token audience matches any of the audiences provided in `LAKEKEEPER__KUBERNETES_AUTHENTICATION_AUDIENCE`
- If `LAKEKEEPER__KUBERNETES_AUTHENTICATION_AUDIENCE` is not set, all tokens proceed to validation! We highly recommend to configure audiences, for most deployments `https://kubernetes.default.svc` works.
1. **Kubernetes Legacy Tokens**
**Enabled if:** `LAKEKEEPER__ENABLE_KUBERNETES_AUTHENTICATION` is true and `LAKEKEEPER__KUBERNETES_AUTHENTICATION_ACCEPT_LEGACY_SERVICEACCOUNT` is true
**Validates Token with:** Kubernetes `TokenReview` API
**Accepts JWT if:**
- Tokens issuer is `kubernetes/serviceaccount` or `https://kubernetes.default.svc.cluster.local` If `LAKEKEEPER__OPENID_PROVIDER_URI` is specified, Lakekeeper will verify access tokens against this provider. The provider must provide the `.well-known/openid-configuration` endpoint and the openid-configuration needs to have `jwks_uri` and `issuer` defined. Typical values for `LAKEKEEPER__OPENID_PROVIDER_URI` are: * Keycloak: `https://keycloak.local/realms/{your-realm}` * Entra-ID: `https://login.microsoftonline.com/{your-tenant-id-here}/v2.0/` Please check the [Authentication Guide](./authentication.md) for more details. | Variable | Example | Description | |---------------------------------------------------------------------------|----------------------------------------------|-----| | `LAKEKEEPER__OPENID_PROVIDER_URI` | `https://keycloak.local/realms/{your-realm}` | OpenID Provider URL. Lakekeeper expects to find `/.well-known/openid-configuration` and load JWKS tokens from there. Do not include the `/.well-known/openid-configuration` in the provided URL. | | `LAKEKEEPER__OPENID_AUDIENCE` | `the-client-id-of-my-app` | Strongly recommended. If set, the `aud` of the provided token must match the value provided. Multiple allowed audiences can be provided as a comma separated list. If unset, audience validation is **skipped** — tokens for any audience are accepted as long as signature and issuer validate. Set this in production. | | `LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS` | `https://sts.windows.net//` | A comma separated list of additional issuers to trust. The issuer defined in the `issuer` field of the `.well-known/openid-configuration` is always trusted. `LAKEKEEPER__OPENID_ADDITIONAL_ISSUERS` has no effect if `LAKEKEEPER__OPENID_PROVIDER_URI` is not set. | | `LAKEKEEPER__OPENID_SCOPE` | `lakekeeper` | Specify a scope that must be present in provided tokens received from the openid provider. | | `LAKEKEEPER__OPENID_SUBJECT_CLAIM` | `sub` or `oid,sub` | Specify the claim(s) in the user's JWT used to identify a User. Accepts a single claim name or a comma-separated list of claim names; the first claim present in the token is used. By default Lakekeeper tries `oid` first, then falls back to `sub`. We strongly recommend setting this configuration explicitly in production deployments. Entra-ID users want to use `oid`; users from all other IdPs most likely want to use `sub`. | | `LAKEKEEPER__OPENID_ROLES_CLAIM` | `resource_access.lakekeeper.roles` | Specify the claim to use in provided JWT tokens to extract roles. The field should contain an array of strings or a single string. Supports nested claims using dot notation, e.g., "resource_access.account.roles". Used by authorizers that consume token roles, including Cedar and custom implementations. The default OpenFGA implementation does not use token roles. Requires a project ID to be set via the `x-project-id` header or `LAKEKEEPER__DEFAULT_PROJECT_ID`. | | LAKEKEEPER__
ENABLE_KUBERNETES_AUTHENTICATION | true | If true, kubernetes service accounts can authenticate to Lakekeeper. This option is compatible with `LAKEKEEPER__OPENID_PROVIDER_URI` - multiple IdPs (OIDC and Kubernetes) can be enabled simultaneously. | | LAKEKEEPER__
KUBERNETES_AUTHENTICATION_AUDIENCE | `https://kubernetes.default.svc` | Audiences that are expected in Kubernetes tokens. Only has an effect if LAKEKEEPER__
ENABLE_KUBERNETES_AUTHENTICATION is true. | | LAKEKEEPER__
KUBERNETES_AUTHENTICATION_ACCEPT_LEGACY_SERVICEACCOUNT | `false` | Add an authenticator that handles tokens with no audiences and the issuer set to `kubernetes/serviceaccount`. Only has an effect if LAKEKEEPER__
ENABLE_KUBERNETES_AUTHENTICATION is true. | #### Multiple OIDC Providers For advanced scenarios requiring multiple OIDC providers (e.g., Okta for users + EKS OIDC for Kubernetes workloads), configure providers under `LAKEKEEPER__OPENID_PROVIDERS____`. These providers are added in addition to `LAKEKEEPER__OPENID_PROVIDER_URI` (the primary provider, `idp_id = "oidc"`). The `` key is the identity-provider ID used in user IDs like `~`. Each `` must match `[a-z0-9-]+` — lowercase letters, digits, and hyphens. Figment lowercases env var segments and uses `__` as a nesting separator, so `LAKEKEEPER__OPENID_PROVIDERS__MY-PROVIDER__URI=...` yields `idp_id = "my-provider"`; use a single `-` to separate words (a `__` in the IDP segment would create a nested key, not part of the name). `oidc` and `kubernetes` are reserved. **Chain order.** Tokens are tried against authenticators in this order: the primary provider from `LAKEKEEPER__OPENID_PROVIDER_URI` (if set), then providers from `LAKEKEEPER__OPENID_PROVIDERS` in **alphabetical order of `idp_id`**, then Kubernetes (if enabled). A token is routed to the **first** authenticator whose `iss` set contains the token's `iss` claim **and** whose `aud` set intersects the token's `aud` claim (an unset issuer or audience matches everything). Once routed, that authenticator performs full signature + issuer + audience validation; if validation fails the request is rejected — the chain does **not** fall through to the next link. As a consequence, if two providers' (issuer, audience) criteria overlap, the first chain link owns the overlap. Make `iss` × `aud` pairs disjoint across providers to avoid surprises. **Provider Fields:** | Variable suffix | Required | Example | Description | |-----------------|----------|---------|-------------| | `__URI` | Yes | `https://company.okta.com` | OIDC provider URI (must expose `.well-known/openid-configuration`). | | `__AUDIENCE` | No (strongly recommended) | `lakekeeper,warehouse` | Expected audience(s) for tokens. Comma-separated for multiple. If unset, audience validation is **skipped** — tokens for any audience are accepted as long as signature and issuer validate. Set this in production. | | `__ADDITIONAL_ISSUERS` | No | `https://sts.windows.net/tenant/` | Additional issuers to trust (comma-separated). | | `__SCOPE` | No | `lakekeeper` | Scope that must be present in tokens. | | `__SUBJECT_CLAIMS` | No | `sub` or `oid,sub` | Claims to use as user ID (comma-separated, in order of preference). Defaults to `oid,sub`. | | `__ROLES_CLAIM` | No | `resource_access.lakekeeper.roles` | Claim to use in provided JWT tokens to extract roles. | | `__REQUIRE_CONNECTED_ON_STARTUP` | No | `true` | When `true` (default), Lakekeeper refuses to start if this provider's OIDC/JWKS configuration cannot be loaded. Set to `false` to skip this provider while continuing startup. | **Example: Okta + EKS OIDC** ```bash LAKEKEEPER__OPENID_PROVIDERS__OKTA__URI=https://company.okta.com LAKEKEEPER__OPENID_PROVIDERS__OKTA__AUDIENCE=https://company.okta.com LAKEKEEPER__OPENID_PROVIDERS__OKTA__SUBJECT_CLAIMS=sub LAKEKEEPER__OPENID_PROVIDERS__OKTA__ROLES_CLAIM=resource_access.lakekeeper.roles LAKEKEEPER__OPENID_PROVIDERS__EKSPROD__URI=https://oidc.eks.us-east-1.amazonaws.com/id/ABC123DEF456 LAKEKEEPER__OPENID_PROVIDERS__EKSPROD__AUDIENCE=sts.amazonaws.com LAKEKEEPER__OPENID_PROVIDERS__EKSPROD__SUBJECT_CLAIMS=sub LAKEKEEPER__OPENID_PROVIDERS__EKSPROD__REQUIRE_CONNECTED_ON_STARTUP=false ``` !!! note Providers fail startup by default if their OIDC endpoint cannot be loaded. Set `REQUIRE_CONNECTED_ON_STARTUP=false` for providers that should be skipped while Lakekeeper continues starting. ### Authorization Authorization is only effective if [Authentication](#authentication) is enabled. We strongly recommend bootstrapping new deployments with authorization already enabled. Switching the configured `AUTHZ_BACKEND` after bootstrap is supported when moving to (or replacing the OpenFGA store behind) the OpenFGA backend — see [Switching to OpenFGA](./authorization-openfga.md#switching-to-openfga-or-replacing-the-store) for the procedure and its limitations (structural hierarchy is rebuilt from the catalog; ownership, grants, and role assignments are not). For other backends, create a new Lakekeeper instance and migrate your tables. | Variable | Example | Description | |------------------------------------------|----------------------------------------------------------------------|----------------------| | `LAKEKEEPER__AUTHZ_BACKEND` | `allowall` | The authorization backend to use. If `openfga` or `cedar` is chosen, additional parameters are required (see below). The `allowall` backend disables authorization - authenticated users can access all endpoints. Default: `allowall`, one-of: [`openfga`, `allowall`, `cedar`] | | `LAKEKEEPER__INSTANCE_ADMINS` | `["kubernetes~system:serviceaccount:lakekeeper:operator","oidc~alice"]` | TOML inline array of user IDs (`~`) that are granted instance-admin privileges via deployment config. Even a single admin must be wrapped in brackets. See [Instance Admins](./authorization.md#instance-admins) for scope and rationale. Default: `[]`. | ##### OpenFGA | Variable | Example | Description | |----------------------------------------------------------|----------------------------------------------------------------------------|-----| | `LAKEKEEPER__OPENFGA__ENDPOINT` | `http://localhost:35081` | OpenFGA Endpoint (gRPC). | | `LAKEKEEPER__OPENFGA__STORE_NAME` | `lakekeeper` | The OpenFGA Store to use. Default: `lakekeeper` | | `LAKEKEEPER__OPENFGA__API_KEY` | `my-api-key` | The API Key used for [Pre-shared key authentication](https://openfga.dev/docs/getting-started/setup-openfga/configure-openfga#pre-shared-key-authentication) to OpenFGA. If `LAKEKEEPER__OPENFGA__CLIENT_ID` is set, the API Key is ignored. If neither API Key nor Client ID is specified, no authentication is used. | | `LAKEKEEPER__OPENFGA__CLIENT_ID` | `12345` | The Client ID to use for Authenticating if OpenFGA is secured via [OIDC](https://openfga.dev/docs/getting-started/setup-openfga/configure-openfga#oidc). | | `LAKEKEEPER__OPENFGA__CLIENT_SECRET` | `abcd` | Client Secret for the Client ID. | | `LAKEKEEPER__OPENFGA__TOKEN_ENDPOINT` | `https://keycloak.example.com/realms/master/protocol/openid-connect/token` | Token Endpoint to use when exchanging client credentials for an access token for OpenFGA. Required if Client ID is set | | `LAKEKEEPER__OPENFGA__SCOPE` | `openfga` | Additional scopes to request in the Client Credential flow. | | LAKEKEEPER__OPENFGA__
AUTHORIZATION_MODEL_PREFIX | `collaboration` | Explicitly set the Authorization model prefix. Defaults to `collaboration` if not set. We recommend to use this setting only in combination with LAKEKEEPER__OPENFGA__
AUTHORIZATION_MODEL_PREFIX. | | LAKEKEEPER__OPENFGA__
AUTHORIZATION_MODEL_VERSION | `3.1` | Version of the model to use. If specified, the specified model version must already exist. This can be used to roll-back to previously applied model versions or to connect to externally managed models. Migration is disabled if the model version is set. Version should have the format .. | | LAKEKEEPER__OPENFGA__
MAX_BATCH_CHECK_SIZE | `50` | p The maximum number of checks than can be handled by a batch check request. This is a [configuration option](https://openfga.dev/docs/getting-started/setup-openfga/configuration#OPENFGA_MAX_CHECKS_PER_BATCH_CHECK) of the `OpenFGA` server with default value 50. | ##### Cedar Please check the [Authorization User Guide](./authorization-cedar.md#authorization-with-cedar) for more information on Cedar. | Variable | Example | Description | |--------------------------------------------------------|---------------------------------------------------------|-----| | LAKEKEEPER__CEDAR__POLICY_SOURCES__
LOCAL_FILES | `[/path/to/policies1.cedar,/path/to/policies2.cedar]` | List of local file paths containing Cedar policies in Cedar format (not JSON). | | LAKEKEEPER__CEDAR__ENTITY_JSON_SOURCES__
LOCAL_FILES | `[/path/to/entities1.json,/path/to/entities2.json]` | List of local JSON file paths containing additional Cedar entities (typically roles). | | LAKEKEEPER__CEDAR__POLICY_SOURCES__
K8S_CM | `[my-cm-1, my-cm-2]` | List of Kubernetes ConfigMap names in the same namespace as Lakekeeper. Every key ending with `.cedar` is treated as a policy source in Cedar format (not JSON). | | LAKEKEEPER__CEDAR__ENTITY_JSON_SOURCES__
K8S_CM | `[my-cm-1, my-cm-2]` | List of Kubernetes ConfigMap names in the same namespace as Lakekeeper. Every key ending with `.cedarentities.json` is treated as an entity source. | | `LAKEKEEPER__CEDAR__REFRESH_INTERVAL_SECS` | `5` | Refresh interval in seconds for reloading policies and entities from Kubernetes ConfigMaps and local files. Default: `5` seconds. See [Cedar Authorization](./authorization-cedar.md#authorization-with-cedar) for more information. | | `LAKEKEEPER__CEDAR__REFRESH_DISABLED` | `false` | When set to `true`, disables periodic reloading of policies and entities entirely. Useful in environments where Cedar configuration is known to be static and the polling overhead is undesirable. Default: `false`. | | LAKEKEEPER__CEDAR__
EXTERNALLY_MANAGED_USER_AND_ROLES | `false` | When set to `true`, Lakekeeper expects all roles and users to be managed externally via entities.json and does not extract `Lakekeeper::Role` or `Lakekeeper::User` entities from the user's token. When set to `false` (default), Lakekeeper automatically provides `Lakekeeper::Role` and `Lakekeeper::User` entities to Cedar based on information extracted from the user's token. When set to `false`, ensure `LAKEKEEPER__OPENID_ROLES_CLAIM` is configured to specify which claim in the token contains role information. | | `LAKEKEEPER__CEDAR__SCHEMA_FILE` | `/path/to/custom/schema.cedarschema` | Path to a custom Cedar schema file that replaces the embedded default schema entirely. Use this only when you need complete control over the schema definition. Your custom schema must maintain compatibility with all Lakekeeper-provided entities (Server, Project, Warehouse, Namespace, Table, View, and optionally User & Role). For most use cases, prefer `LAKEKEEPER__CEDAR__SCHEMA_FRAGMENT_FILE` to extend the built-in schema. | | `LAKEKEEPER__CEDAR__SCHEMA_FRAGMENT_FILE` | `/path/to/schema-fragment.cedarschema` | Path to a Cedar schema fragment file that extends the embedded default schema. This is the recommended approach for adding custom entity types or grouped actions while preserving compatibility with Lakekeeper's built-in schema. The fragment is merged with the default schema at startup. | | LAKEKEEPER__CEDAR__
PROPERTY_PARSE_PREFIXES | `["access_", "access-"]` | List of property key prefixes that trigger entity-reference parsing for ABAC. Table, Namespace, and View properties whose key starts with one of these prefixes are parsed as JSON arrays of `role:` / `role-full:` / `user:` references. Parsed values are exposed in Cedar as `roles: Set` and `users: Set` on each `ResourcePropertyValue`. Set to `[]` to disable parsing entirely. Default: `["access_", "access-"]`. See [Property-Based Access Control](./authorization-cedar.md#property-based-access-control). | | LAKEKEEPER__CEDAR__
GLOBAL_ROLE_IDS_ENABLED | `false` | When `true`, the `global_role_ids: Set` attribute on every `Lakekeeper::User` entity is populated with the `source_id` of every provider-resolved role (token claims, LDAP, etc.). This enables simpler policies such as `principal.global_role_ids.contains("admins")` without needing to specify a `provider_id`. Only meaningful when all configured role providers use globally unique `source_id` values (i.e. no two providers assign the same `source_id` to different roles). When `false` (default), `global_role_ids` is always an empty set. | | `LAKEKEEPER__CEDAR__USER_DERIVATIONS____SOURCE` | `source_id` | Source field for a user identity derivation rule. Supported values: `source_id` (the user's subject in the IdP) or `provider_id` (e.g. `oidc`, `kubernetes`). `` is a human-readable key (e.g. `EMAIL_PARTS`) used in error messages. See [User Identity Derivations](./authorization-cedar.md#user-identity-derivations). | | `LAKEKEEPER__CEDAR__USER_DERIVATIONS____PATTERN` | `^(?[^@]+)`
`@(?.+)$` | Regex pattern with named capture groups for a user identity derivation rule. Each named group that matches a non-empty substring becomes a string tag on the `UserDerivedAttributes` entity, accessible in policies via `principal.derived_attributes.hasTag("…")` / `principal.derived_attributes.getTag("…")`. Invalid patterns cause a startup error. See [User Identity Derivations](./authorization-cedar.md#user-identity-derivations). | | `LAKEKEEPER__CEDAR__USER_DERIVATIONS____TRANSFORM` | `lowercase` | Optional transformation applied to all captured values before they become Cedar tags. Supported values: `none` (default — keep as-is), `lowercase`, `uppercase`. Because Cedar string comparison is case-sensitive, use `lowercase` to normalize captured values so policies can compare against a known-case literal (e.g. `getTag("domain") == "example.com"`). If different capture groups need different transforms, use separate derivation entries with distinct regexes. See [User Identity Derivations](./authorization-cedar.md#user-identity-derivations). | **Debug configurations for Cedar** | Variable | Example | Description | |-------------------------------------------------------|---------|------------| | `LAKEKEEPER__CEDAR__DEBUG__LOG_ENTITIES` | `false` | If `true`, logs all internal entities (excluding externally managed entities) for each authorization request at debug level. This is useful for debugging authorization issues but can be verbose and impacts performance. Logging only occurs when both this flag is `true` AND debug logging is enabled (`RUST_LOG=debug`). Default: `false`. | ### UI When using the built-in UI which is hosted as part of the Lakekeeper binary, most values are pre-set with the corresponding values of Lakekeeper itself. Customization is typically required if Authentication is enabled. Please check the [Authentication guide](./authentication.md) for more information. | Variable | Example | Description | |----------------------------------------------------|----------------------------------------------|-----| | `LAKEKEEPER__UI__OPENID_PROVIDER_URI` | `https://keycloak.local/realms/{your-realm}` | OpenID provider URI used for login in the UI. Defaults to `LAKEKEEPER__OPENID_PROVIDER_URI`. Set this only if the IdP is reachable under a different URI from the users browser and lakekeeper. | | `LAKEKEEPER__UI__OPENID_CLIENT_ID` | `lakekeeper-ui` | Client ID to use for the Authorization Code Flow of the UI. Required if Authentication is enabled. Defaults to `lakekeeper` | | `LAKEKEEPER__UI__OPENID_REDIRECT_PATH` | `/callback` | Path where the UI receives the callback including the tokens from the users browser. Defaults to: `/callback` | | `LAKEKEEPER__UI__OPENID_SCOPE` | `openid email` | Scopes to request from the IdP. Defaults to `openid profile email`. | | `LAKEKEEPER__UI__OPENID_RESOURCE` | `lakekeeper-api` | Resources to request from the IdP. If not specified, the `resource` field is omitted (default). | | LAKEKEEPER__UI__
OPENID_POST_LOGOUT_REDIRECT_PATH | `/logout` | Path the UI calls when users are logged out from the IdP. Defaults to `/logout` | | `LAKEKEEPER__UI__LAKEKEEPER_URL` | `https://example.com/lakekeeper` | URI where the users browser can reach Lakekeeper. Defaults to the value of `LAKEKEEPER__BASE_URI`. | | `LAKEKEEPER__UI__OPENID_TOKEN_TYPE` | `access_token` | The token type to use for authenticating to Lakekeeper. The default value `access_token` works for most IdPs. Some IdPs, such as the Google Identity Platform, recommend the use of the OIDC ID Token instead. To use the ID token instead of the access token for Authentication, specify a value of `id_token`. Possible values are `access_token` and `id_token`. | | `LAKEKEEPER__UI__ENABLE_SURVEYS` | `true` | The UI occasionally shows in-app user surveys to gather feedback on Lakekeeper. All responses are collected anonymously. Set to `false` to opt out; the UI then never initializes the survey SDK and makes no third-party requests. Defaults to `true`. | | `LAKEKEEPER__UI__BRANDING` | `eyJ0aGVtZXMiOns...` | Base64-encoded JSON that white-labels the UI with custom theme colors and partner logos. See [UI Branding](./ui-branding.md). Lakekeeper Plus only. | ### Caching Lakekeeper uses in-memory caches to speed up certain operations. Most cache entries' time-to-live is jittered downward by a small random fraction (up to 10%), so an entry lives 90–100% of the configured TTL. This desynchronizes expiry across replicas that warmed the same key at the same time, preventing a fleet-wide refresh stampede on the TTL boundary. The configured `..._TIME_TO_LIVE_SECS` remains the upper bound — jitter only ever shortens an entry's life, never extends it. **Short-Term Credentials (STC) Cache** When Lakekeeper vends short-term credentials for cloud storage access (S3 STS, Azure SAS tokens, or GCP access tokens), these credentials can be cached to reduce load on cloud identity services and improve response times. | Variable | Example | Description | |-------------------------------------------------|---------|------------------| | `LAKEKEEPER__CACHE__STC__ENABLED` | `true` | Enable or disable the short-term credentials cache. Default: `true` | | `LAKEKEEPER__CACHE__STC__CAPACITY` | `10000` | Maximum number of credential entries to cache, **per cloud provider** — S3, Azure, and GCP each maintain a separate cache. A single-cloud deployment caches at most this many entries; a server vending for multiple clouds can hold up to this many per provider in use. Default: `10000` | *Expiry Mechanism*: Cached credentials automatically expire based on the validity period of the underlying cloud credentials. Lakekeeper caches credentials for half their lifetime (e.g., if GCP STS returns credentials valid for 1 hour, they're cached for 30 minutes) with a maximum cache duration of 1 hour. This ensures credentials remain fresh while reducing unnecessary identity service calls. *Metrics*: The STC cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="stc"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="stc"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="stc"}`: Total number of cache misses **Warehouse Cache** Caches warehouse metadata to reduce database queries for warehouse lookups. | Configuration Key | Type | Default | Description | |---------------------------------------------------------------|---------|---------|-----| | `LAKEKEEPER__CACHE__WAREHOUSE__ENABLED` | boolean | `true` | Enable/disable warehouse caching. Default: `true` | | `LAKEKEEPER__CACHE__WAREHOUSE__CAPACITY` | integer | `1000` | Maximum number of warehouses to cache. Default: `1000` | | LAKEKEEPER__CACHE__WAREHOUSE__
TIME_TO_LIVE_SECS | integer | `60` | Time-to-live for cache entries in seconds. Default: `60` | If the cache is enabled, changes to Storage Profile may take up to the configured TTL (default: 60 seconds) to be reflected in all Lakekeeper workers. If a single worker is used, the Cache is always up to date. Warehouse metadata is guaranteed to be fresh for load table & view operations also for multi-worker deployments. *Metrics*: The Warehouse cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="warehouse"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="warehouse"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="warehouse"}`: Total number of cache misses **Namespace Cache** Caches namespace metadata and hierarchies to reduce database queries for namespace lookups. Namespace lookups are also required for table & view operations. | Configuration Key | Type | Default | Description | |---------------------------------------------------------------|---------|---------|-----| | `LAKEKEEPER__CACHE__NAMESPACE__ENABLED` | boolean | `true` | Enable/disable namespace caching. Default: `true` | | `LAKEKEEPER__CACHE__NAMESPACE__CAPACITY` | integer | `1000` | Maximum number of namespaces to cache. Default: `1000` | | LAKEKEEPER__CACHE__NAMESPACE__
TIME_TO_LIVE_SECS | integer | `60` | Time-to-live for cache entries in seconds. Default: `60` | If the cache is enabled, changes to namespace properties may take up to the configured TTL (default: 60 seconds) to be reflected in all Lakekeeper workers. If a single worker is used, the Cache is always up to date. The namespace cache stores both individual namespaces and their parent hierarchies for efficient lookups. *Metrics*: The Namespace cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="namespace"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="namespace"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="namespace"}`: Total number of cache misses **Secrets Cache** Caches storage secrets to reduce load on the secret store. Since Lakekeeper never updates secrets, long TTLs can significantly increase resilience against secret store outages, especially when the secret store is external to the main database backend. | Configuration Key | Type | Default | Description | |-------------------------------------------------------------|---------|---------|-----| | `LAKEKEEPER__CACHE__SECRETS__ENABLED` | boolean | `true` | Enable/disable secrets caching. Default: `true` | | `LAKEKEEPER__CACHE__SECRETS__CAPACITY` | integer | `500` | Maximum number of secrets to cache. Default: `500` | | LAKEKEEPER__CACHE__SECRETS__
TIME_TO_LIVE_SECS | integer | `600` | Time-to-live for cache entries in seconds. Default: `600` (10 minutes) | *Metrics*: The Secrets cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="secrets"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="secrets"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="secrets"}`: Total number of cache misses **Role Cache** Caches role metadata to reduce database queries for role lookups. The role cache uses a two-tier caching mechanism: a primary cache indexed by role ID and a secondary index by project ID and role identifier, enabling efficient lookups from both identifiers. Note that this cache only stores role definitions and does not include any information about role assignments to users or principals. | Configuration Key | Type | Default | Description | |----------------------------------------------------------|---------|---------|-----| | `LAKEKEEPER__CACHE__ROLE__ENABLED` | boolean | `true` | Enable/disable role caching. Default: `true` | | `LAKEKEEPER__CACHE__ROLE__CAPACITY` | integer | `10000` | Maximum number of roles to cache. Default: `10000` | | LAKEKEEPER__CACHE__ROLE__
TIME_TO_LIVE_SECS | integer | `120` | Time-to-live for cache entries in seconds. Default: `120` (2 minutes) | If the cache is enabled, changes to role metadata may take up to the configured TTL (default: 120 seconds) to be reflected in all Lakekeeper workers. If a single worker is used, the cache is always up to date. The cache is automatically invalidated when roles are updated or deleted. *Metrics*: The Role cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="role"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="role"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="role"}`: Total number of cache misses **User Assignments Cache** Caches the set of roles assigned to each user (`UserId → role assignments`). This is the hot-path cache checked on every authorization request and is also the in-memory layer used by the LDAP role provider's two-layer caching scheme. The TTL must not exceed `LAKEKEEPER__CACHE__ROLE__TIME_TO_LIVE_SECS` to bound the window in which a deleted role can still appear in assignment results. | Configuration Key | Type | Default | Description | |----------------------------------------------------------------------|---------|---------|-----| | LAKEKEEPER__CACHE__USER_ASSIGNMENTS__
ENABLED | boolean | `true` | Enable/disable user-assignments caching. Default: `true` | | LAKEKEEPER__CACHE__USER_ASSIGNMENTS__
CAPACITY | integer | `50000` | Maximum number of users whose assignments are held in memory. Default: `50000` | | LAKEKEEPER__CACHE__USER_ASSIGNMENTS__
TIME_TO_LIVE_SECS | integer | `120` | Time-to-live for cache entries in seconds. Must not exceed LAKEKEEPER__CACHE__ROLE__
TIME_TO_LIVE_SECS. Default: `120` (2 minutes) | *Metrics*: The User Assignments cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="user_assignments"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="user_assignments"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="user_assignments"}`: Total number of cache misses **Role Members Cache** Caches the members of each role (`RoleId → role members`). This is a cold-path cache populated only by admin/provider queries that list a role's members. Each entry holds a role's full member list, so the default capacity is deliberately low. | Configuration Key | Type | Default | Description | |------------------------------------------------------------------|---------|---------|-----| | `LAKEKEEPER__CACHE__ROLE_MEMBERS__ENABLED` | boolean | `true` | Enable/disable role-members caching. Default: `true` | | LAKEKEEPER__CACHE__ROLE_MEMBERS__
CAPACITY | integer | `1000` | Maximum number of roles whose member lists are held in memory. Default: `1000` | | LAKEKEEPER__CACHE__ROLE_MEMBERS__
TIME_TO_LIVE_SECS | integer | `120` | Time-to-live for cache entries in seconds. Default: `120` (2 minutes) | *Metrics*: The Role Members cache exposes Prometheus metrics for monitoring: - `lakekeeper_cache_size{cache_type="role_members"}`: Current number of entries in the cache - `lakekeeper_cache_hits_total{cache_type="role_members"}`: Total number of cache hits - `lakekeeper_cache_misses_total{cache_type="role_members"}`: Total number of cache misses ### Endpoint Statistics Lakekeeper collects statistics about the usage of its endpoints. Every Lakekeeper instance accumulates endpoint calls for a certain duration in memory before writing them into the database. The following configuration options are available: | Variable | Example | Description | |--------------------------------------------------------|---------|-----------| | `LAKEKEEPER__ENDPOINT_STAT_FLUSH_INTERVAL` | 30s | Interval in seconds to write endpoint statistics into the database. Default: 30s, valid units are (s\|ms) | ### SSL Dependencies Lakekeeper validates outbound TLS connections against two root stores combined: - **Mozilla root CAs** bundled into the binary via `webpki-roots` / `webpki-root-certs`. Public endpoints (AWS, GCS, Azure, OIDC providers) work out of the box, including on minimal images with no system CA bundle (scratch, `distroless`, `ubi-micro`). - **System / custom CAs** loaded via `rustls-native-certs`, which respects `SSL_CERT_FILE` and `SSL_CERT_DIR`. Point these at a PEM bundle or directory to trust self-signed certificates (e.g. for MinIO, internal IdPs). When unset, the standard host paths are consulted (`/etc/ssl/certs/...`, `/etc/pki/tls/certs/...`, etc.). The certificate presented by an endpoint cannot itself be a CA — it must be an end-entity certificate, otherwise TLS handshakes fail with `CaUsedAsEndEntity`. Two code paths do *not* use the bundled webpki roots and therefore require a system CA bundle at `/etc/ssl/certs/ca-certificates.crt` (or one of the other standard locations): - **Vault** integration (via `vaultrs` → `rustls-platform-verifier`). You can also pass a PEM bundle explicitly through the Vault client configuration. - **Kafka** integration (via `librdkafka` / OpenSSL). The official `distroless` and `ubi` images ship a CA bundle, so these work without extra configuration. If you roll your own minimal image and use Vault or Kafka, install `ca-certificates` or copy a bundle to `/etc/ssl/certs/ca-certificates.crt`. ### Request Limits Lakekeeper allows you to configure limits on incoming requests to protect against resource exhaustion and denial-of-service attacks. | Variable | Example | Description | |--------------------------------------------------|-----------|---------------| | `LAKEKEEPER__MAX_REQUEST_BODY_SIZE` | `2097152` | Maximum request body size in bytes. Default: `2097152` (2 MB) | | `LAKEKEEPER__MAX_REQUEST_TIME` | `30s` | Maximum time allowed for a request to complete. Accepts format `{number}{ms\|s}`. Default: `30s` | ### Roles Limits applied to the role model. `LAKEKEEPER__ROLE__MAX_NESTING_DEPTH` bounds how deeply roles may be nested in one another (role→role membership). The depth is the number of role→role edges in a chain; direct user assignments do not count. Adding a membership edge that would make any chain longer than this limit is rejected with `RoleMembershipDepthExceeded` (HTTP 409). This bound is enforced on the catalog (Postgres) write path. When using the OpenFGA authorization backend, nesting depth is governed by OpenFGA's own resolution limits rather than this setting — the same asymmetry as role-membership cycle prevention. | Variable | Example | Description | |-----------------------------------------------------|---------|-------------| | `LAKEKEEPER__ROLE__MAX_NESTING_DEPTH` | `10` | Maximum number of role→role edges in any nesting chain. Default: `10` | ### Maintenance Mode Captured at startup; not dynamic. While `read-only`: - Mutating requests (anything other than `GET`/`HEAD`/`OPTIONS`) on `/catalog/v1` and `/management/v1` return `503` with `Retry-After: 60` and `error.type = "MaintenanceModeError"`. `/health` is unaffected. - Built-in task queue workers are not started. - `GET /v1/config` skips user auto-registration. - `GET /health` exposes the current mode as `maintenance_mode` for operator fan-out checks. | Variable | Example | Description | |------------------------------------------------|-------------|-------------| | `LAKEKEEPER__MAINTENANCE_MODE` | `read-only` | `off` (default) or `read-only`. Captured at startup; not dynamic. | ### Idempotency Lakekeeper supports the [Iceberg REST Catalog Idempotency](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) specification. When enabled, clients can send an `Idempotency-Key` header on mutation requests to guarantee at-most-once execution. The server advertises support via the `idempotency-key-lifetime` field in the `GET /v1/config` response. | Variable | Example | Description | |---|---|---| | `LAKEKEEPER__IDEMPOTENCY__ENABLED` | `true` | Enable idempotency key support. When enabled, `idempotency-key-lifetime` is advertised in `getConfig`. Default: `true` | | `LAKEKEEPER__IDEMPOTENCY__LIFETIME` | `PT30M` | How long idempotency records are kept, in ISO-8601 duration format. This value is advertised to clients. Default: `PT30M` (30 minutes) | | `LAKEKEEPER__IDEMPOTENCY__GRACE_PERIOD` | `PT5M` | Grace period added on top of lifetime for clock skew and transit delays, in ISO-8601 duration format. Default: `PT5M` (5 minutes) | | `LAKEKEEPER__IDEMPOTENCY__CLEANUP_TIMEOUT` | `PT30S` | Maximum time a background cleanup task may run before being considered dead. If exceeded, the next attempt takes over. Default: `PT30S` (30 seconds) | ### Audit Logging Lakekeeper can generate detailed audit logs for all authorization events. Audit logs are written to the standard logging output and can be filtered by the `event_source = "audit"` field. For more information, see the [Logging Guide](./logging.md). | Variable | Example | Description | |----------------------------------------------------|---------|---------------| | `LAKEKEEPER__AUDIT__TRACING__ENABLED` | `true` | Enable audit logging for authorization events. When enabled, all authorization checks (both successful and failed) are logged at the `INFO` level with `event_source = "audit"`. Audit logs include the actor, action, resource, and outcome. Default: `false` | ### Trusted Engines Trusted engines enable Lakekeeper to make context-aware authorization decisions for views with delegated execution (DEFINER security model). When configured, Lakekeeper: 1. **Protects the owner property** — only requests from a matched engine can set or remove the view property that controls delegated execution (e.g. `trino.run-as-owner`). 2. **Evaluates `referenced-by` chains** — when a trusted engine sends the `referenced-by` query parameter on `loadTable` / `loadView`, Lakekeeper resolves the full view chain and checks permissions for the correct user at each step. For a detailed explanation of DEFINER vs INVOKER views, see the [View Security](./view-security.md) guide. Trusted engines are configured as a map under `LAKEKEEPER__TRUSTED_ENGINES`. Each entry has a logical name (the map key), a type, the owner property name, and one or more identities that define which tokens are trusted. #### Configuration | Variable | Example | Description | |----------|---------|-------------| | LAKEKEEPER__TRUSTED_ENGINES__<NAME>__TYPE | `trino` | Engine type. Currently only `trino` is supported. | | LAKEKEEPER__TRUSTED_ENGINES__<NAME>__OWNER_PROPERTY | `trino.run-as-owner` | The view property that identifies the owner for delegated execution. | Each engine requires one or more **identities** — keyed by Identity Provider ID (e.g. `oidc`, `kubernetes` as configured in [Authentication](./authentication.md)) — that define which tokens are trusted. A token matches an identity if the Identity Provider ID matches AND (any audience matches OR any subject matches). Multiple engines can match a single token. List values use bracket syntax: `[value1, value2]` — even for a single value: `[value]`. | Variable | Example | Description | |----------|---------|-------------| | `LAKEKEEPER__TRUSTED_ENGINES____`
`IDENTITIES____AUDIENCES` | `[trino_dev, trino_prod]` | List of JWT audiences. A token matches if any of its audiences appears in this list. | | `LAKEKEEPER__TRUSTED_ENGINES____`
`IDENTITIES____SUBJECTS` | `[trino-sa]` | List of JWT subjects. A token matches if its subject appears in this list. Useful for service accounts. | **Example:** Trust a Trino engine whose tokens come from the `oidc` provider with audience `trino`: ```bash LAKEKEEPER__TRUSTED_ENGINES__TRINO__TYPE=trino LAKEKEEPER__TRUSTED_ENGINES__TRINO__OWNER_PROPERTY=trino.run-as-owner LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__OIDC__AUDIENCES=[trino] ``` **Example with multiple IdPs:** Trust Trino from both an OIDC provider and a Kubernetes service account: ```bash LAKEKEEPER__TRUSTED_ENGINES__TRINO__TYPE=trino LAKEKEEPER__TRUSTED_ENGINES__TRINO__OWNER_PROPERTY=trino.run-as-owner LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__OIDC__AUDIENCES=[trino_dev, trino_prod] LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__KUBERNETES__SUBJECTS=[trino-sa] ``` ### Role Provider Authorizers such as `Cedar` support pluggable role providers that resolve a user's group memberships from an external directory (e.g. LDAP / Active Directory). Multiple providers can be configured in parallel, each with a unique identifier. `OpenFGA` does not use role providers — roles are stored directly in OpenFGA. Role providers that resolve groups over HTTPS — the Microsoft Graph (Entra ID) provider — honor the standard `HTTPS_PROXY`, `HTTP_PROXY`, and `NO_PROXY` environment variables for outbound requests. There is no per-provider proxy setting. ##### Chain settings | Variable | Default | Description | |----------------------------------------------------------------------|---------|-----| | LAKEKEEPER__ROLE_PROVIDER_CHAIN__
LOG_UNHANDLED_USERS | `true` | When `true`, an audit event is emitted whenever a user is not matched by any configured role provider. Useful for detecting misconfigured domain filters. Set to `false` to suppress these events for deployments where some users are intentionally not covered by any provider. | | LAKEKEEPER__ROLE_PROVIDER_CHAIN__
LOG_ROLE_ASSIGNMENTS | `false` | When `true`, an audit event listing every resolved role name is emitted after each successful role resolution. Very noisy — intended for debugging role-provider configuration only. See [Logging — Operational Audit Events](./logging.md) for the event schema. | | LAKEKEEPER__ROLE_PROVIDER_CHAIN__
PERSIST_TOKEN_ROLES | `false` | When `true`, roles extracted from OIDC tokens are persisted to the catalog database so they can be reused when the user is not the live caller — see [Token role provider](#token-role-provider) below. | ##### Token role provider When `LAKEKEEPER__OPENID_ROLES_CLAIM` is set, Lakekeeper extracts roles directly from the authenticated user's JWT. A built-in token role provider is added to the chain **automatically** — no additional configuration is required. The token role provider only applies to OIDC-authenticated users (those whose identity was established via the configured OpenID Connect provider). It is a no-op for users authenticated through other mechanisms (e.g. Kubernetes service accounts). The provider uses the reserved identifier `oidc`. If you declare a role provider with this identifier in your configuration, the automatic provider is suppressed and your custom provider takes its place. By default, token roles live only for the duration of the request — they are read from the JWT and never stored. This means they are unavailable for authorization decisions about a user who is **not** the current caller, most notably [DEFINER views](./view-security.md), where access is evaluated as the view's owner rather than the querying user. Set `LAKEKEEPER__ROLE_PROVIDER_CHAIN__PERSIST_TOKEN_ROLES=true` to mirror each user's token roles into the catalog database. Persisted roles are then served whenever that user's permissions are evaluated while they are not the requesting caller — including DEFINER views. On each request, the caller's token roles are compared against the stored set and written only when they differ; requests carrying unchanged roles do not write. A definer's roles are therefore as current as the most recent token that user presented. Roles are scoped to the project they were issued for: in multi-project deployments the owner must have presented a token for the project where the view lives. An empty roles claim is ignored rather than clearing the stored set. ##### LDAP role provider Each LDAP provider is configured under a unique `` of your choosing. All variables below use the prefix `LAKEKEEPER__ROLE_PROVIDER____`. **Required fields:** | Variable | Example | Description | |--------------------------------|----------------------------------------|-----| | `…__TYPE` | `ldap` | Provider type. Must be `ldap`. | | `…__URL` | `ldaps://ldap.example.com:636` | LDAP server URL. Use `ldap://` for plain-text or STARTTLS, `ldaps://` for TLS. | | `…__DOMAINS` | `["example.com","*.corp.example.com"]` | JSON array of domain patterns. Only users whose login name ends with one of these domains are resolved via this provider. Supports `*` (any number of characters) and `?` (exactly one character). | | `…__USER_BASE_DN` | `ou=people,dc=example,dc=com` | Base DN for the LDAP user search. | **Authentication:** | Variable | Default | Description | |---------------------------------|-------------|------------------------------| | `…__BIND_DN` | (anonymous) | Distinguished name of the service account used to bind. Omit for anonymous bind. | | `…__BIND_PASSWORD` | | Password for the service account. Required when `…__BIND_DN` is set; can also be supplied via `…__BIND_PASSWORD_FILE`. | **User search:** | Variable | Default | Description | |--------------------------------------|-----------------|---------------------| | `…__USER_SEARCH_FILTER` | `(uid=${USER})` | LDAP filter used to locate a user entry. The literal `${USER}` is replaced with the subject portion of the user's login name (the part before `@`). | | `…__USER_SEARCH_SCOPE` | `sub` | Search scope: `sub` (entire subtree), `one` (one level below base), or `base`. | **Group / role mapping:** The LDAP role provider supports three resolution modes, selected via `__GROUP_RESOLUTION_MODE`: | Mode | When to use | |------|-------------| | `attribute` *(default)* | Read group DNs directly from a `memberOf`-style attribute on the user entry. Correct for Active Directory / ADFS, OpenLDAP with the `memberof` overlay, and most setups where every user-of-interest has a populated `memberOf`. | | `search` | Run a paged subtree LDAP search to find groups whose member attribute references the user. Useful for directories without `memberOf`, for filtering to a specific group name prefix (e.g. `ACME-*`) at the directory level instead of post-filtering, and for AD transitive resolution via `LDAP_MATCHING_RULE_IN_CHAIN`. | | `branching` *(since 0.12.2)* | Per-user-DN branching: a regex tested against the user's DN selects between a Search-style filter (with named captures available as `${name}` placeholders) and an explicit `else` branch. | The selector and shared fields: | Variable | Default | Description | |---------------------------------------|-------------|--------------------------------------------------------------------------------------------------------| | `…__GROUP_RESOLUTION_MODE` | `attribute` | One of `attribute`, `search`, or `branching`. The remaining fields depend on the mode you choose. | | `…__GROUP_BASE_DN` | | Base DN for the group search. Required by `search` and `branching`; ignored by `attribute`. | | `…__GROUP_CASE` | `keep` | Case transformation applied to the resolved group name before it is stored as a role. One of `keep`, `upper`, or `lower`. | **Attribute mode (`group_resolution_mode = "attribute"`):** | Variable | Default | Description | |--------------------------------------------|------------|--------------------| | `…__USER_MEMBER_OF_ATTRIBUTE` | `memberOf` | Multi-valued attribute on the user entry that lists the groups the user belongs to. | | `…__GROUP_NAME_SOURCE` | `dn_cn` | How to derive the role name from a group entry. `dn_cn` extracts the `CN=` component from the group's distinguished name (recommended for AD/ADFS). | **Search mode (`group_resolution_mode = "search"`)** — *available since 0.12.2 with `${USER}` / `${DOMAIN}` placeholder support and the composed default filter*: | Variable | Default | Description | |-----------------------------------------|-------------------------------|--------------------------------------------------------------------------------------------------------| | `…__GROUP_SEARCH_FILTER` | `({GROUP_MEMBER_ATTRIBUTE}=${USER_DN})` | LDAP filter for the group search. May reference `${USER_DN}`, `${USER}`, and `${DOMAIN}` placeholders. When omitted, composed from `…__GROUP_MEMBER_ATTRIBUTE` — works on AD (`objectClass=group`), OpenLDAP (`groupOfNames` with `member`, or `groupOfUniqueNames` with `uniqueMember`), and 389-DS. For AD transitive resolution, set to `(member:1.2.840.113556.1.4.1941:=${USER_DN})`. | | `…__GROUP_MEMBER_ATTRIBUTE` | `member` | Attribute on the group entry that references members; drives the default filter when `…__GROUP_SEARCH_FILTER` is unset. Use `uniqueMember` for `groupOfUniqueNames` directories. | | `…__GROUP_NAME_ATTRIBUTE` | `cn` | Attribute on each returned group entry used as the role name. Use `sAMAccountName` for AD if you want the short name instead of the CN. | **Branching mode (`group_resolution_mode = "branching"`)** — *available since 0.12.2* — per-user-DN dispatch between a Search-style `then` branch and an explicit `else` branch: | Variable | Description | |----------------------------------------------------------|-------------| | `…__BRANCH_IF_USER_DN_MATCHES` | Regex tested against the user's DN (case-insensitive by default; prepend `(?-i)` for strict casing). Named captures `(?…)` become `${name}` placeholders in the `then` filter. Reserved names `USER`, `DOMAIN`, `USER_DN` are rejected. | | `…__BRANCH_THEN__GROUP_SEARCH_FILTER` | LDAP filter for the `then` branch. May reference `${USER_DN}`, `${USER}`, `${DOMAIN}`, and any named capture from `…__BRANCH_IF_USER_DN_MATCHES`. Every placeholder must resolve at startup (unknown placeholders are rejected). | | `…__BRANCH_THEN__GROUP_MEMBER_ATTRIBUTE` | Same role as in Search mode — drives the default filter when `…__BRANCH_THEN__GROUP_SEARCH_FILTER` is unset. Defaults to `member`. | | `…__BRANCH_THEN__GROUP_NAME_ATTRIBUTE` | Attribute used as the role name for groups returned by the `then` branch. Defaults to `cn`. | | `…__BRANCH_ELSE__MODE` | **Required.** One of `attribute` (run Attribute-mode resolution against the user entry) or `none` (return empty roles, emit `outcome = "dn_no_match"` to audit). No implicit default — pick explicitly so audit logs distinguish "no roles" from "no match". | | `…__BRANCH_ELSE__USER_MEMBER_OF_ATTRIBUTE` | When `…__BRANCH_ELSE__MODE=attribute`: the `memberOf`-style attribute to read. Defaults to `memberOf`. | | `…__BRANCH_ELSE__GROUP_NAME_SOURCE` | When `…__BRANCH_ELSE__MODE=attribute`: how to derive the role name. Defaults to `dn_cn`. | !!! note "Capture-driven filters and cross-scope memberships" A capture-driven filter (e.g. `sAMAccountName=${tenant}*ABC-*`) constrains resolution to the captured value; memberships outside that scope are not returned. For cross-scope memberships, add another role provider in the chain with a broader filter. !!! note "AD primary group is never returned" Active Directory's primary group (typically `Domain Users`, identified by `primaryGroupID` rather than a `member` link) is not returned by any mode — `memberOf` omits it and `LDAP_MATCHING_RULE_IN_CHAIN` does not walk it. To expose it as a Lakekeeper role, add explicit `member` entries in the directory. !!! note "Regex is case-insensitive; `${USER_DN}` preserves directory casing" `…__BRANCH_IF_USER_DN_MATCHES` is compiled case-insensitive by default (prepend `(?-i)` to make it strict). The DN substituted into `${USER_DN}` is escaped per RFC 4515 and inserted with the casing the directory returned. DN-typed attribute predicates (`member=`, `memberOf=`, …) match server-side regardless of case. !!! note "`group_base_dn` is always required in branching mode" Set `…__GROUP_BASE_DN` even if you expect every user to take the `else.mode = attribute` branch — the validator can't know which branch a given user will hit. Use the same value you would use in Search mode. !!! warning "Role cache persists across restarts; changing modes does not invalidate it" Role assignments are cached in the catalog database, not just in process memory. Restarting Lakekeeper with a new `group_resolution_mode` does **not** invalidate the DB cache — users keep their previously-resolved roles for up to `…__SYNC_INTERVAL_SECS`. During a config rollout, either lower `…__SYNC_INTERVAL_SECS` temporarily or flush the role-assignments table for the affected provider. **Example — Search mode with AD transitive resolution:** ```bash LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_RESOLUTION_MODE=search LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_BASE_DN=dc=corp,dc=example,dc=com LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_SEARCH_FILTER='(&(sAMAccountName=ACME-*)(member:1.2.840.113556.1.4.1941:=${USER_DN}))' LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_NAME_ATTRIBUTE=sAMAccountName ``` **Example — Branching mode (tenant users → tenant-scoped search; service accounts → `memberOf`):** ```bash LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_RESOLUTION_MODE=branching LAKEKEEPER__ROLE_PROVIDER__CORP_AD__GROUP_BASE_DN=dc=corp,dc=example,dc=com # Extract the tenant code from the user's DN (the OU directly below OU=Tenants) LAKEKEEPER__ROLE_PROVIDER__CORP_AD__BRANCH_IF_USER_DN_MATCHES='OU=(?[^,]+),OU=Tenants,' # THEN branch — tenant users get tenant-scoped, transitively-resolved roles LAKEKEEPER__ROLE_PROVIDER__CORP_AD__BRANCH_THEN__GROUP_SEARCH_FILTER='(&(sAMAccountName=${tenant}-ACME-*)(member:1.2.840.113556.1.4.1941:=${USER_DN}))' LAKEKEEPER__ROLE_PROVIDER__CORP_AD__BRANCH_THEN__GROUP_NAME_ATTRIBUTE=sAMAccountName # ELSE branch — service accounts under OU=Service fall back to memberOf LAKEKEEPER__ROLE_PROVIDER__CORP_AD__BRANCH_ELSE__MODE=attribute LAKEKEEPER__ROLE_PROVIDER__CORP_AD__BRANCH_ELSE__USER_MEMBER_OF_ATTRIBUTE=memberOf ``` **Connection and TLS:** | Variable | Default | Description | |----------------------------------------|---------|---------------------------| | `…__STARTTLS` | `false` | Upgrade a plain TCP connection with STARTTLS before binding. Only applies to `ldap://` URLs. | | `…__ALLOW_INSECURE` | `false` | Skip TLS certificate verification. **Do not use in production.** | | `…__CONNECT_TIMEOUT_SECS` | `30` | Seconds to wait when establishing the initial connection. | | `…__READ_TIMEOUT_SECS` | `60` | Seconds to wait for an LDAP response. | **Caching & performance:** Each LDAP provider uses a two-layer cache to avoid a network round-trip to the LDAP server on every request: 1. **In-memory layer** — role assignments are held in a per-node moka cache (see [User Assignments Cache](#caching) above). Reads that hit this layer incur no I/O at all. 2. **Database layer** — on an in-memory miss, role assignments are read from (and re-populate) the database. The database record includes a `synced_at` timestamp that is compared against `SYNC_INTERVAL_SECS` to decide whether the data is still fresh. If the database record is older than `SYNC_INTERVAL_SECS`, Lakekeeper contacts LDAP, writes the fresh assignments back to both the database and the in-memory cache, and returns the result. If LDAP is temporarily unreachable, the stale database record is served instead and an audit warning is emitted — the request is never failed solely due to an LDAP outage. | Variable | Default | Description | |--------------------------------------|---------|-----------------------------| | `…__SYNC_INTERVAL_SECS` | `300` | Maximum age (in seconds) of a cached role-assignment record before Lakekeeper re-fetches from LDAP. Increase to reduce LDAP traffic; decrease when group membership changes must propagate more quickly. Also controls the TTL of the corresponding database record. | **Startup and resilience:** | Variable | Default | Description | |------------------------------------------------|---------|-------------------| | `…__REQUIRE_CONNECTED_ON_STARTUP` | `false` | When `true`, Lakekeeper refuses to start if this provider cannot connect. Useful for catching misconfiguration early. When `false`, the provider starts in a disconnected state and reconnects automatically on first use. | | `…__RECONNECT_COOLDOWN_SECS` | `30` | Minimum seconds between reconnection attempts after a failure. | **IDP filtering (optional):** | Variable | Default | Description | |---------------------------|--------------|-----------------------------------| | `…__IDP_IDS` | *(all IDPs)* | JSON array of identity provider IDs. When set, only users from these IDPs are resolved via this provider. Omit to allow all IDPs. | **Example — minimal LDAP provider (env vars):** ```bash LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__TYPE=ldap LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__URL=ldaps://ldap.corp.example.com:636 LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__DOMAINS=["corp.example.com"] LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__USER_BASE_DN=ou=people,dc=corp,dc=example,dc=com LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__BIND_DN=cn=svc-lakekeeper,ou=service-accounts,dc=corp,dc=example,dc=com LAKEKEEPER__ROLE_PROVIDER__MY_LDAP__BIND_PASSWORD_FILE=/run/secrets/ldap-password ``` ##### Microsoft Graph (Entra ID) role provider Resolves a user's **transitive** Microsoft Entra ID group memberships via the Microsoft Graph API and maps each group to a role — keyed by the group's object id, with the group `displayName` as the role name. Each provider is configured under a unique `` of your choosing; all variables use the prefix `LAKEKEEPER__ROLE_PROVIDER____`. The app registration this provider authenticates as needs the Microsoft Graph **application** permissions `GroupMember.Read.All` and `User.Read.All`, admin-consented. **Required fields:** | Variable | Example | Description | |----------|---------|-------------| | `…__TYPE` | `entra-graph` | Provider type. Must be `entra-graph` (aliases: `entra_graph`, `azure-ad`). | | `…__CREDENTIAL__METHOD` | `secret` | Azure credential type — one of `secret`, `certificate`, `managed_identity`, `workload_identity`. Selects the remaining `…__CREDENTIAL__*` fields below. | **Credential** — the `…__CREDENTIAL__*` fields depend on `…__CREDENTIAL__METHOD`: | Method | Fields | |--------|--------| | `secret` | `…__CREDENTIAL__TENANT_ID`, `…__CREDENTIAL__CLIENT_ID`, `…__CREDENTIAL__CLIENT_SECRET` (all required) | | `certificate` | `…__CREDENTIAL__TENANT_ID`, `…__CREDENTIAL__CLIENT_ID`, `…__CREDENTIAL__CERTIFICATE_PATH` (PKCS#12 / PFX, read at startup); optional `…__CREDENTIAL__CERTIFICATE_PASSWORD` | | `managed_identity` | Omit `…__CREDENTIAL__USER_ASSIGNED_ID` for the system-assigned identity. For a user-assigned identity set `…__CREDENTIAL__USER_ASSIGNED_ID__KIND` (`client_id`, `object_id`, or `resource_id`) and `…__CREDENTIAL__USER_ASSIGNED_ID__VALUE`. | | `workload_identity` | Optional `…__CREDENTIAL__TENANT_ID`, `…__CREDENTIAL__CLIENT_ID`, `…__CREDENTIAL__TOKEN_FILE_PATH`; each falls back to the standard `AZURE_*` environment variables when omitted. | **Cloud / endpoints:** | Variable | Default | Description | |----------|---------|-------------| | `…__CLOUD` | `public` | Sovereign cloud: `public`, `us_government`, `china`, or `custom`. Drives the default Graph endpoint and the token authority. | | `…__GRAPH_BASE` | *(per cloud)* | Microsoft Graph endpoint base. Overrides the cloud default; **required** for `custom`. | | `…__AUTHORITY_HOST` | *(per cloud)* | Entra ID token authority. Overrides the cloud default (e.g. a national-cloud proxy); **required** for `custom`. | Built-in cloud endpoints: | Cloud | Graph base | Authority | |-------|------------|-----------| | `public` | `https://graph.microsoft.com` | `https://login.microsoftonline.com` | | `us_government` | `https://graph.microsoft.us` | `https://login.microsoftonline.us` | | `china` | `https://microsoftgraph.chinacloudapi.cn` | `https://login.chinacloudapi.cn` | | `custom` | *(none — set `…__GRAPH_BASE`)* | *(none — set `…__AUTHORITY_HOST`)* | **HTTP timeouts:** | Variable | Default | Description | |----------|---------|-------------| | `…__CONNECT_TIMEOUT_SECS` | `10` | Seconds to wait when establishing a connection to Graph. | | `…__REQUEST_TIMEOUT_SECS` | `30` | Seconds to wait for a Graph response. | Transient failures (`429` honoring `Retry-After`, transient `5xx`, and connection/timeout errors) are retried a few times with exponential backoff before the request fails and the cache falls back to the last good result. Outbound requests honor the standard `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` environment variables. **Caching:** | Variable | Default | Description | |----------|---------|-------------| | `…__SYNC_INTERVAL_SECS` | `300` | Maximum age of a cached role-assignment record before Lakekeeper re-fetches from Graph. Uses the same two-layer (in-memory + database) cache as the LDAP provider, including stale-fallback on a Graph outage. | **Startup:** | Variable | Default | Description | |----------|---------|-------------| | `…__REQUIRE_CONNECTED_ON_STARTUP` | `false` | When `true`, Lakekeeper refuses to start if it cannot acquire a Graph token on startup. | **IDP filtering (optional):** | Variable | Default | Description | |----------|---------|-------------| | `…__IDP_IDS` | *(all IDPs)* | JSON array of **OIDC provider** IDs — the IdP a user logged in through, i.e. the default provider's reserved id `oidc`, or a [multi-OIDC](./authentication.md#multiple-oidc-providers) provider's configured id. When set, only users who authenticated via those providers are resolved here. Omit to handle all — Entra subjects are object ids and carry no domain to filter on. | **Example — client-secret credential (env vars):** ```bash LAKEKEEPER__ROLE_PROVIDER__ENTRA__TYPE=entra-graph LAKEKEEPER__ROLE_PROVIDER__ENTRA__CREDENTIAL__METHOD=secret LAKEKEEPER__ROLE_PROVIDER__ENTRA__CREDENTIAL__TENANT_ID= LAKEKEEPER__ROLE_PROVIDER__ENTRA__CREDENTIAL__CLIENT_ID= LAKEKEEPER__ROLE_PROVIDER__ENTRA__CREDENTIAL__CLIENT_SECRET= # Only resolve users who logged in via the OIDC provider with id `oidc` # (the default provider's reserved id; a multi-OIDC provider uses its own id). LAKEKEEPER__ROLE_PROVIDER__ENTRA__IDP_IDS=["oidc"] ``` ##### File-based configuration All providers can alternatively be configured through a single TOML file. This is convenient when secrets management or config management tools produce a single artefact (e.g. Vault agent, Kubernetes projected volumes, Ansible templates). Point `LAKEKEEPER__ROLE_PROVIDER_FILE` at a standard TOML file. Each provider is a section `[role_provider.]` where `` is the provider ID you choose. Multiple providers can be defined in the same file. **Example — two LDAP providers in one file:** `/etc/lakekeeper/role-providers.toml`: ```toml [role_provider.corporate] type = "ldap" url = "ldaps://ldap.corp.example.com:636" domains = ["corp.example.com"] user_base_dn = "ou=people,dc=corp,dc=example,dc=com" bind_dn = "cn=svc-lakekeeper,ou=service-accounts,dc=corp,dc=example,dc=com" bind_password = "s3cr3t" [role_provider.subsidiary] type = "ldap" url = "ldaps://ldap.subsidiary.example.com:636" domains = ["subsidiary.example.com"] user_base_dn = "ou=users,dc=subsidiary,dc=example,dc=com" bind_dn = "cn=svc-lakekeeper,ou=service-accounts,dc=subsidiary,dc=example,dc=com" bind_password = "s3cr3t" ``` Then set the single environment variable: ```bash LAKEKEEPER__ROLE_PROVIDER_FILE=/etc/lakekeeper/role-providers.toml ``` > **Combining file and environment variables:** The file and env-var approaches can be combined. The file is loaded first and env vars are merged on top — env vars override individual fields for the same provider while unset fields are preserved from the file. This makes it easy to store non-sensitive configuration in the file and inject secrets via env vars: > > ```toml > # /etc/lakekeeper/role-providers.toml (checked in, no secrets) > [role_provider.corporate] > type = "ldap" > url = "ldaps://ldap.corp.example.com:636" > domains = ["corp.example.com"] > user_base_dn = "ou=people,dc=corp,dc=example,dc=com" > bind_dn = "cn=svc-lakekeeper,ou=service-accounts,dc=corp,dc=example,dc=com" > ``` > > ```bash > # Injected at runtime (e.g. from a secrets manager) > LAKEKEEPER__ROLE_PROVIDER_FILE=/etc/lakekeeper/role-providers.toml > LAKEKEEPER__ROLE_PROVIDER__CORPORATE__BIND_PASSWORD=s3cr3t > ``` ### Tokio Runtime Metrics Lakekeeper emits [Tokio Runtime Metrics](https://github.com/tokio-rs/tokio-metrics?tab=readme-ov-file#runtime-metrics) with a default report interval of 30 seconds. If necessary, this interval can be fine-tuned. | Variable | Example | Description | |------------------------------------------------------------|---------|----------------------------------------------------------------------------------------------| | LAKEKEEPER__METRICS__TOKIO__
REPORT_INTERVAL | `30s` | Length of interval for which Tokio Runtime Metrics are collected and emitted. Default: `30s` | ### Debug Lakekeeper provides debugging options to help troubleshoot issues during development. These options should **not** be enabled in production environments as they can expose sensitive data and impact performance. | Variable | Example | Description | |------------------------------------------------------------|---------|-------| | `LAKEKEEPER__DEBUG__LOG_REQUEST_BODIES` | `true` | If set to `true`, Lakekeeper will log all incoming and outgoing request bodies at debug level. This is useful for debugging API interactions but should **never** be enabled in production as it can expose sensitive data (credentials, tokens, etc.) and significantly impact performance. Default: `false` | | `LAKEKEEPER__DEBUG__MIGRATE_BEFORE_SERVE` | `true` | If set to `true`, Lakekeeper waits for the DB (30s) and runs migrations when `serve` is called. Default: `false` | | `LAKEKEEPER__DEBUG__AUTO_SERVE` | `true` | If set to `true`, Lakekeeper will automatically start the server when no subcommand is provided (i.e., when running the binary without arguments). This is useful for development environments to quickly start the server without explicitly specifying the `serve` command. Default: `false` | | `LAKEKEEPER__DEBUG__EXTENDED_LOGS` | `false` | Controls whether file names and line numbers are included in JSON log output. When set to `false`, these fields are omitted for cleaner logs. When set to `true`, each log entry includes `filename` and `line_number` fields for easier debugging. Default: `false` | | LAKEKEEPER__DEBUG__
LOG_AUTHORIZATION_HEADER | `false` | If set to `true`, the `Authorization` header is included in request trace spans for the `/catalog/v1/config` and `/management/v1/info` endpoints. This exposes sensitive credentials (tokens, passwords) and should **never** be enabled in production. Default: `false` | **Warning**: Debug options can expose sensitive information in logs and should only be used in secure development environments. ### Test Configurations | Variable | Example | Description | |---------------------------------------------------|---------|----------------| | `LAKEKEEPER__SKIP_STORAGE_VALIDATION` | true | If set to true, Lakekeeper does not validate the provided storage configuration & credentials when creating or updating Warehouses. This is not suitable for production. Default: false | ### License Configuration Lakekeeper Plus requires a License to operate. The license can be provided via either of the following environment variables. If both are set, `LAKEKEEPER__LICENSE__KEY` takes precedence. | Variable | Example | Description | |----------------------------------------------|------------------------|------| | `LAKEKEEPER__LICENSE__KEY` | `` | License key as a string. Takes precedence over `LAKEKEEPER__LICENSE__KEY_PATH` if both are set. | | `LAKEKEEPER__LICENSE__KEY_PATH` | `/path/to/license.lic` | Path to a file containing the license key. | --- # Customize Source: https://docs.lakekeeper.io/docs/latest/customize/ # Customize Customizability is one of the core features that sets Lakekeeper apart from other Iceberg REST Catalog implementations. Almost every part of the server is a Rust trait or an injectable hook, so you can replace a backend or extend behaviour — for example to grant access to tables via your company's data-governance solution, persist secrets in a vault you already operate, react to every table change, expose custom endpoints, or publish events to your messaging system. There are two ways to extend Lakekeeper, and the canonical list of everything you can supply is the [`ServeConfiguration`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/serve.rs) struct passed to [`serve()`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/serve.rs). ## Compile-time backends (generic parameters) These are the type parameters of `serve` and [`CatalogServer`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/server/mod.rs). Choosing an implementation is a build-time decision, so an integration is type-checked when you compile the server — there is no runtime plugin loading. | Component | Trait | Responsibility | Ships with | |-----------|-------|----------------|------------| | `Catalog` | [`CatalogStore`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/catalog_store.rs) | Persistence for Warehouses, Namespaces, Tables, Views, and other entities. | Postgres (`lakekeeper-storage-postgres`) | | `SecretStore` | [`SecretStore`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/secrets.rs) | Secure storage for storage credentials and other secrets. | Postgres, HashiCorp Vault-compatible KV v2 (`lakekeeper-secrets-kv2`) | | `Authorizer` | [`Authorizer`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/authz/mod.rs) | The permission system. May expose its own management APIs. Default: `AllowAllAuthorizer`. | `AllowAll`, OpenFGA | | `Authenticator` | [`limes::Authenticator`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/serve.rs) | Token authentication chain. Default: `AuthenticatorEnum`. | OIDC/OAuth2, Kubernetes | ## Runtime extension points (`ServeConfiguration`) These are fields of `ServeConfiguration`, set with its builder when the server starts. Several accept a list, so multiple implementations can be active at once. | Field | Trait / type | What it lets you do | |-------|--------------|---------------------| | `event_dispatcher` | [`EventListener`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/events/dispatch.rs) | React to strongly-typed domain events — table created/dropped/renamed/registered, transaction committed, view changes, and more. Override only the events you care about. | | `cloud_event_sinks` | [`CloudEventBackend`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/events/publisher.rs) | Receive those changes as serialized [CloudEvents](https://cloudevents.io) and forward them somewhere. Ships with Kafka, NATS, and tracing sinks. | | `contract_verification` | [`ContractVerification`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/contract_verification.rs) | Veto table/view changes — for example when a data contract would be violated. Implementations are collected into `ContractVerifiers`. | | `admission_gates` | [`AdmissionGate`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/admission.rs) | Accept or reject an already-authenticated request before it reaches any handler — e.g. consult an external entitlement service, suspend a tenant or principal, or honor a token deny-list. Gates run after the actor and instance-admin status are resolved; they are evaluated in order and the first rejection wins. | | `stats` | [`EndpointStatisticsSink`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/service/endpoint_statistics.rs) | Consume per-project, per-endpoint API call statistics — for chargeback, quotas, or your own monitoring pipeline. | | `modify_router_fn` | `fn(axum::Router) -> axum::Router` | Modify the HTTP router before serving: add custom routes, middleware, or layers. (The built-in UI is injected exactly this way.) | | `register_additional_task_queues_fn` | [`RegisterTaskQueueFn`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/serve.rs) | Register custom background task queues alongside the built-in soft-delete expiration and purge queues. | | `register_additional_background_services_fn` | [`RegisterBackgroundServiceFn`](https://github.com/lakekeeper/lakekeeper/blob/main/crates/lakekeeper/src/serve.rs) | Spawn arbitrary additional background services / futures that share the server's graceful-shutdown lifecycle. | | `enable_built_in_task_queues` | `bool` | Disable the built-in task queues if you want to run them out-of-process. | `ServeConfiguration` also accepts `license_status`, `build_info`, and `system_roles`, which downstream binaries (such as Lakekeeper Plus) use to inject build metadata and seed catalog-managed system roles. ### Two layers of event handling Reacting to changes has two levels, depending on what you need: * Implement an **`EventListener`** when you want to run typed Rust logic in-process on specific events (the default trait methods are no-ops, so you override only what you care about). Register it via the `EventDispatcher`. * Provide a **`CloudEventBackend`** when you just want changes shipped out as CloudEvents. The built-in CloudEvents publisher is itself an `EventListener` that fans every event out to all configured `cloud_event_sinks`. ## What the hooks are for A few common reasons to reach for each one: * **Custom task queues** run persisted, scheduled background work on Lakekeeper's own task infrastructure — the tasks are stored in the database and picked up by workers across your Lakekeeper instances, so you don't have to stand up a separate scheduler or worry about a single worker dying mid-job. Beyond the built-in soft-delete expiration and purge queues, typical uses include triggering table maintenance/compaction, custom retention and cleanup policies, recomputing statistics, refreshing downstream materialized views, running data-quality or lineage scans, and pushing periodic digests to external systems. * **Event listeners / cloud-event sinks** keep external systems in step with the catalog without polling — feeding data-discovery and lineage tools, audit pipelines, cache invalidation, or webhooks. * **Contract verification** enforces organisational rules at write time, for example rejecting a schema change that would break a published data contract. * **Admission gates** make a coarse allow/deny decision about an already-authenticated principal *before* any handler runs — distinct from the per-resource `Authorizer`. Use them to consult an external control-plane entitlement service, suspend a tenant or principal, or reject revoked tokens. A gate returns either a terminal `403` or, when it fails closed because an upstream it depends on is unreachable, a `503` with a `Retry-After`. Because they run on every authenticated request, gates should cache aggressively and enforce their own timeouts. * **Endpoint-statistics sinks** route usage data into billing/chargeback, per-tenant quotas, or your own dashboards. * **Router modifications** expose extra endpoints (custom health/readiness checks, internal admin APIs) or middleware alongside the catalog — this is how the built-in UI is mounted. ## Adding a custom implementation Extending Lakekeeper always follows the same shape: 1. **Implement the trait.** Each extension point above links to its trait definition — that source is the authoritative, always-current signature. 2. **Wire it in.** For a runtime hook, pass your implementation to the matching `ServeConfiguration` builder field (for example `.cloud_event_sinks(...)`, `.stats(...)`, or `.modify_router_fn(...)`). For a compile-time backend, pass your type as the generic parameter to `serve()` (`CatalogStore`, `SecretStore`, `Authorizer`, `Authenticator`). The best examples are the implementations Lakekeeper ships — they are compiled against the current traits and never drift out of date: * **Event sinks** — the Kafka (`lakekeeper-events-kafka`) and NATS (`lakekeeper-events-nats`) crates, plus the in-tree `TracingPublisher`. * **Catalog & secrets** — `lakekeeper-storage-postgres` and `lakekeeper-secrets-kv2`. * **Authorizer** — the OpenFGA implementation in [`crates/authz-openfga`](https://github.com/lakekeeper/lakekeeper/tree/main/crates/authz-openfga). See the [Developer Guide](./developer-guide.md) for building Lakekeeper from sources. --- # Developer Guide Source: https://docs.lakekeeper.io/docs/latest/developer-guide/ # Developer Guide All commits to main go through a PR. CI checks have to pass before merging the PR. Keep in mind that CI checks include lints. Before merge, commits are squashed, but GitHub is taking care of this, so don't worry. PR titles should follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). We encourage small and orthogonal PRs. If you want to work on a bigger feature, please open an issue and discuss it with us first. If you want to work on something but don't know what, take a look at our issues tagged with `help wanted`. If you're still unsure, please reach out to us via the [Lakekeeper Discord](https://discord.gg/jkAGG8p93B). If you have questions while working on something, please use the GitHub issue or our Discord. We are happy to guide you! ## Foundation & CLA We hate red tape. Currently, all committers need to sign the CLA in GitHub. To ensure the future of Lakekeeper, we want to donate the project to a foundation. We are not sure yet if this is going to be Apache, Linux, a Lakekeeper foundation or something else. Currently, we prefer to spend our time on adding cool new features to Lakekeeper, but we will revisit this topic during 2026. ## Initial Setup To work on small and self-contained features, it is usually enough to have a Postgres database running while setting a few envs. The code block below should get you started up to running most unit tests as well as clippy. ```bash # start postgres docker run -d --name postgres-16 -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:17 # set envs echo 'export DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres' > .env echo 'export ICEBERG_REST__PG_ENCRYPTION_KEY="abc"' >> .env echo 'export ICEBERG_REST__PG_DATABASE_URL_READ="postgresql://postgres:postgres@localhost/postgres"' >> .env echo 'export ICEBERG_REST__PG_DATABASE_URL_WRITE="postgresql://postgres:postgres@localhost/postgres"' >> .env source .env # Migrate db (make sure you have sqlx installed `cargo install sqlx-cli`). # sqlx-cli auto-loads `.env` from the workspace root, so DATABASE_URL is picked up. sqlx database create sqlx migrate run --source crates/lakekeeper-storage-postgres/migrations # Run tests (make sure you have cargo nextest installed, `cargo install cargo-nextest`) cargo nextest run --all-features # run clippy just check-clippy # formatting the code (make sure you have cargo-sort installed, `cargo install cargo-sort`) # You may have to install nightly rust toolchain just fix-format ``` Keep in mind that some tests are excluded by the `default-filter` in `.config/nextest.toml`. You can find a list of them in the [Testing section](#test-cloud-storage-profiles) below or by searching for modules whose name contains `_integration_tests` within files ending with `.rs`. There are a few cargo commands we run on CI. You may install [just](https://crates.io/crates/just) to run them conveniently. If you made any changes to SQL queries, please follow [Working with SQLx](#working-with-sqlx) before submitting your PR. ### Required tools for OpenAPI regeneration The `just update-management-openapi` and `just update-generic-table-openapi` recipes — plus several `add-*-to-rest-openapi` recipes — require **Go yq** ([mikefarah/yq](https://github.com/mikefarah/yq)). The Python `yq` (kislyuk) shipped via `pip install yq` is **not compatible**: it uses different flags (`-y -i` instead of `-i`) and its YAML emitter formats lists differently, which produces large whitespace-only diffs. Install Go yq: ```bash # macOS brew install yq # Linux (download the static binary) curl -L "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64" \ -o ~/.local/bin/yq && chmod +x ~/.local/bin/yq # Verify (must say "mikefarah" in the version output) yq --version ``` ## Code structure ### What is where? We have three crates, `lakekeeper`, `lakekeeper-bin` and `iceberg-ext`. The bulk of the code is in `lakekeeper`. The `lakekeeper-bin` crate contains the main entry point for the catalog. The `iceberg-ext` crate contains extensions to `iceberg-rust`. **lakekeeper** The `lakekeeper` crate contains the core of the catalog. It is structured into several modules: 1. `api` - contains the implementation of the REST API handlers as well as the `axum` router instantiation. 2. `catalog` - contains the core business logic of the REST catalog 3. `service` - contains various function blocks that make up the whole service, e.g., authn, authz and implementations of specific cloud storage backends. 4. `tests` - contains integration tests and some common test helpers, see below for more information. 5. `implementations` - contains the concrete implementation of the catalog backend, currently there's only a Postgres implementation and an alternative for Postgres as secret-store, `kv2`. **lakekeeper-bin** The main function branches out into multiple commands, amongst others, there's a health-check, migrations, but also serve which is likely the most relevant to you. In case you are forking us to implement your own AuthZ backend, you'll want to change the `serve` command to use your own implementation, just follow the call-chain. ### Where to put tests? We try to keep unit-tests close to the code they are testing. E.g., all tests for the database module of tables are located in `crates/lakekeeper/src/implementations/postgres/tabular/table/mod.rs`. While working on more complex features we noticed a lot of repetition within tests and started to put commonly used functions into `crates/lakekeeper/src/tests/mod.rs`. Within the `tests` module, there are also some higher-level tests that cannot be easily mapped to a single module or require a non-trivial setup. Depending on what you are working on, you may want to put your tests there. ### I need to add an endpoint You'll start at `api` and add the endpoint function to either `management` or `iceberg` depending on whether the endpoint belongs to official iceberg REST specification. The likely next step is to extend the respective `Service` trait so that there's a function to be called from the REST handler. Within the trait function, depending on your feature, you may need to store or fetch something from the storage backend. Depending on if the functionality already exists, you can do so via the respective function on the `C` generic and either the `state: ApiContext>` struct or by first getting a transaction via `C::Transaction::begin_(state.v1_state.catalog.clone()).await?;`. If you need to add a new function to the storage backend, extend the `Catalog` trait and implement it in the respective modules within `implementations`. Remember to do appropriate AuthZ checks within the function of the respective `Service` trait. ## Debugging complex issues and prototyping using our examples To debug more complex issues, work on prototypes or simply an initial manual test, you can use one of the `examples`. Unless you are working on AuthN or AuthZ, you'll most likely want to use the minimal example. All examples come with a `docker-compose-build.yaml` which will build the catalog image from source. The invocation looks like this: `docker compose -f docker-compose.yaml -f docker-compose-build.yaml up -d --build`. Aside from building the catalog, the `docker-compose-build.yaml` overlay also exposes the docker services to your host, so you can also use it as a development environment by e.g. pointing your env vars to the docker container to test against its minio instance. If you made changes to SQL queries, you'll have to run `just sqlx-prepare` before rebuilding the catalog image. This will update the sqlx queries in `.sqlx` to enable static checking of the queries without a migrated database. After spinning the example up, you may head to `localhost:8888` and use one of the notebooks. ## Working with SQLx This crate uses sqlx. For development and compilation a Postgres Database is required. This is part of the [Initial setup](#initial-setup). If your database credentials used differ, please modify the `.env` accordingly and run `source .env` again. Run: ```sh # Migrate db. Make sure you have sqlx-cli install with `cargo install sqlx-cli` # Run this locally if you change the db schema via `crates/lakekeeper-storage-postgres/migrations`, # e.g. after adding a table or dropping a column. sqlx database create sqlx migrate run --source crates/lakekeeper-storage-postgres/migrations # If you changed any of the SQL statements embedded in Rust code, run this before pushing to GitHub. just sqlx-prepare ``` This will update the sqlx queries in `.sqlx` to enable static checking of the queries without a migrated database. Remember to `git add .sqlx` before committing. If you forget, your PR will fail to build on GitHub. Be careful, if the command failed, `.sqlx` will be empty. But do not worry, it wouldn't build on GitHub so there's no way of really breaking things. ### ⚠️ Schema Qualification Warning **IMPORTANT**: When adding new migrations, do **NOT** schema qualify references to any database objects. Schema qualification will break deployments that place the application in a schema different than the public one. **❌ Incorrect - Do NOT do this:** ```sql -- This will break deployments in non-public schemas CREATE TABLE public.my_new_table ( id SERIAL PRIMARY KEY, name VARCHAR(255) ); INSERT INTO public.my_new_table (name) VALUES ('example'); ALTER TABLE public.existing_table ADD COLUMN new_column INTEGER; ``` **✅ Correct - Do this instead:** ```sql -- This will work in any schema CREATE TABLE my_new_table ( id SERIAL PRIMARY KEY, name VARCHAR(255) ); INSERT INTO my_new_table (name) VALUES ('example'); ALTER TABLE existing_table ADD COLUMN new_column INTEGER; ``` The migration system will automatically apply the migration in the correct schema context, so explicit schema qualification is unnecessary and will cause issues in deployments where Lakekeeper is deployed to a custom schema. ### Inspecting the db The db schema is the result of all migrations applied in order. To inspect it you can: ```shell # Assumes you set up the db as described above # Get a shell in the db's container docker exec -it postgres-16 /bin/bash # Then you can connect to the db psql "postgresql://postgres:postgres@localhost:5432/postgres" # And inspect it, for instance by describing views or tables \d+ active_tabulars # Or you can dump the entire schema pg_dump --schema-only "postgresql://postgres:postgres@localhost:5432/postgres" > /home/lakekeeper_schema.sql # Copy it out of the container and then inspect it or pass it as context to LLMs docker cp postgres-16:/home/lakekeeper_schema.sql . ``` ### Extension tables (`ext_*` prefix) Lakekeeper reserves the `ext_*` table-name prefix for downstream extensions that need to store their own state in the catalog database. The convention is an operational contract between upstream and any extension: | Rule | What it means | |---|---| | Reserved prefix | Upstream core migrations **never** create tables matching `ext_*`. Extensions own that namespace. The integration test `test_core_does_not_create_ext_objects` enforces this. | | FK direction | Extension tables FK *into* upstream tables. Upstream never FKs into `ext_*`. | | CASCADE required | Every FK from an `ext_*` table to an upstream table should be `ON DELETE CASCADE` or `ON DELETE SET NULL`. Enforce in the extension crate's own CI — upstream cannot inspect downstream migration sets. | | Scope of allowed objects | `ext_*` may name tables and objects owned by those tables (indexes, sequences). Triggers, functions, indexes, or views attached to upstream-owned objects are not permitted under the prefix — they would survive extension removal and could brick OSS. | | Tracker tables | Each registered extension migration source uses its own SQLx tracker, named `ext__sqlx_migrations`. Core's `_sqlx_migrations` is untouched. | #### Registering extension migrations Extensions call upstream's `migrate(pool, extensions)` with their own migration source. Every registered source runs **inside the same outer transaction** as core migrations — either every migration commits or the entire upgrade rolls back. Partial state is impossible. ```rust use lakekeeper_storage_postgres::migrations::{ExtensionMigrations, migrate}; // `name` must be 1–40 chars: first [a-z_], remaining [a-z0-9_]; rejected at // the start of `migrate()` otherwise. Derives `ext_my_extension_sqlx_migrations`. let extensions = vec![ExtensionMigrations::builder() .name("my_extension") .migrator(sqlx::migrate!("./migrations")) // embedded at compile time .build()]; let server_id = migrate(&pool, extensions).await?; ``` Optional fields not shown: `.data_hooks(map)` for Rust-side hooks tied to specific migration versions, and `.sha_patches(set)` for in-place edits to already-shipped migrations. The `data_hooks` field on `ExtensionMigrations` is a `HashMap>` keyed by the migration's version id. Each entry's `MigrationHook` runs immediately after the matching extension migration is applied, inside the same transaction — use it for Rust-side data backfills tied to a specific SQL migration. Pass `HashMap::new()` when no hooks are needed (the common case in the snippet above). Callers that don't register extensions use the back-compat shim `migrate_core_only(pool)`. Core upstream tooling and tests already do. #### Recovery: removing an extension's state Dropping the extension binary and removing its tables restores the database to a working OSS-only state. The SQL below scans every relation kind that the `ext_*` prefix may name and drops it: ```sql -- Run inside the catalog database. Drops every ext_* table and tracker -- (CASCADE handles dependent indexes, sequences, and constraints). DO $$ DECLARE r record; BEGIN -- Tables (covers extension state + per-source `_sqlx_migrations` trackers). FOR r IN SELECT c.relname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace WHERE n.nspname = current_schema() AND c.relkind IN ('r', 'p') AND c.relname LIKE 'ext\_%' ESCAPE '\' LOOP EXECUTE format('DROP TABLE %I CASCADE', r.relname); END LOOP; -- Defensive sweeps for object kinds the convention forbids extensions -- from creating on upstream-owned tables, but that may exist if a -- non-conforming extension was deployed. FOR r IN SELECT t.tgname, c.relname AS tbl FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid WHERE NOT t.tgisinternal AND t.tgname LIKE 'ext\_%' ESCAPE '\' LOOP EXECUTE format('DROP TRIGGER %I ON %I', r.tgname, r.tbl); END LOOP; FOR r IN SELECT typname FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace WHERE n.nspname = current_schema() AND typname LIKE 'ext\_%' ESCAPE '\' LOOP EXECUTE format('DROP TYPE %I CASCADE', r.typname); END LOOP; FOR r IN SELECT p.proname FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname = current_schema() AND p.proname LIKE 'ext\_%' ESCAPE '\' LOOP EXECUTE format('DROP FUNCTION %I CASCADE', r.proname); END LOOP; END $$; ``` After running this, the OSS binary boots cleanly against the remaining catalog state. ## KV2 / Vault This catalog supports KV2 as a backend for secrets. Tests for KV2 are disabled by default. To enable them, you need to run the following commands: ```shell docker run -d -p 8200:8200 --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' hashicorp/vault # append some more env vars to the .env file, it should already have PG related entries defined above. # the values below configure KV2 echo 'export ICEBERG_REST__KV2__URL="http://localhost:8200"' >> .env echo 'export ICEBERG_REST__KV2__USER="test"' >> .env echo 'export ICEBERG_REST__KV2__PASSWORD="test"' >> .env echo 'export ICEBERG_REST__KV2__SECRET_MOUNT="secret"' >> .env source .env # setup vault ./tests/vault-setup.sh http://localhost:8200 # Select kv2 tests cargo nextest run --all-features --all-targets \ --ignore-default-filter -E "test(::kv2_integration_tests::)" ``` ## Test cloud storage profiles Currently, we're not aware of a good way of testing cloud storage integration against local deployments. That means, to test against AWS S3, GCS and ADLS Gen2, you need to set the following environment variables. For more information, take a look at the [Storage Guide](storage.md). A sample `.env` could look like this: ```sh export LAKEKEEPER_TEST__AZURE_TENANT_ID= export LAKEKEEPER_TEST__AZURE_STORAGE_FILESYSTEM= export LAKEKEEPER_TEST__AZURE_STORAGE_ACCOUNT_NAME= # Auth Method 1: Client Credentials export LAKEKEEPER_TEST__AZURE_CLIENT_ID= export LAKEKEEPER_TEST__AZURE_CLIENT_SECRET= # Auth Method 2: Shared Key export LAKEKEEPER_TEST__AZURE_STORAGE_SHARED_KEY= export LAKEKEEPER_TEST__AWS_S3_BUCKET= export LAKEKEEPER_TEST__AWS_S3_REGION= export LAKEKEEPER_TEST__AWS_S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export LAKEKEEPER_TEST__AWS_S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export LAKEKEEPER_TEST__AWS_S3_STS_ROLE_ARN=arn:aws:iam::123456789012:role/role-name # the values below should work with the default minio in our docker-compose export LAKEKEEPER_TEST__S3_BUCKET=tests export LAKEKEEPER_TEST__S3_REGION=local export LAKEKEEPER_TEST__S3_ACCESS_KEY=minio-root-user export LAKEKEEPER_TEST__S3_SECRET_KEY=minio-root-password export LAKEKEEPER_TEST__S3_ENDPOINT=http://localhost:9000 export LAKEKEEPER_TEST__GCS_CREDENTIAL='{"type": "service_account","project_id": "..", ...}' export LAKEKEEPER_TEST__GCS_BUCKET=name-of-gcs-bucket-without-hns export LAKEKEEPER_TEST__GCS_HNS_BUCKET=name-of-gcs-bucket-with-hns ``` You may then run tests by ignoring the nextest's default filter and selecting the desired tests: ```sh source .example.env-from-above cargo nextest run --all-features --ignore-default-filter -E "test(::aws_integration_tests::)" # see .config/nextest.toml for all filters ``` ## Running integration test Our integration tests are written in Python and use pytest. They are located in the `tests` folder. The integration tests spin up Lakekeeper and all the dependencies via `docker compose`. Please check the [Integration Test Docs](https://github.com/lakekeeper/lakekeeper/tree/main/tests) for more information. ### Running Authorization unit tests Some authorization unit tests need to be run against an OpenFGA server. They are excluded by our nextest `default-filter`. The workflow for executing them is: ```bash # Start an OpenFGA server in a docker container docker rm --force openfga-client && docker run -d --name openfga-client -p 36080:8080 -p 36081:8081 -p 36300:3000 openfga/openfga:v1.14 run # Set Lakekeeper's OpenFGA endpoint export LAKEKEEPER_TEST__OPENFGA__ENDPOINT="http://localhost:36081" # Use a filterset to select the tests cargo nextest run --all-features --ignore-default-filter -E "test(::openfga_integration_tests::)" ``` ## Extending Authz When adding a new endpoint, you may need to extend the authorization model. Please check the [Authorization Docs](./authorization.md) for more information. For OpenFGA, perform the following steps: 1. Add the new action to the relevant enum in `crate::service::authz`, e.g. `CatalogViewAction::CanUndrop`. Actions that must carry request context for policy-based authorizers are parameterized variants — see `CatalogProjectAction::CreateWarehouse`. 1. In the `lakekeeper-authz-openfga` crate (`crates/authz-openfga/src/relations.rs`), add or reuse a relation on the resource enum (e.g. `RoleRelation::CanUndrop`) and map the action to it in the `ReducedRelation` impl (e.g. `CatalogViewAction::CanUndrop => ViewRelation::CanUndrop`). 1. Create the next authz schema version. For **backward-compatible** changes (no tuple rewrite, e.g. adding a relation) **rename** the latest folder — e.g. `authz/openfga/v4.6/` → `authz/openfga/v4.7/` — so existing stores re-migrate to the new model id. Only changes that require tuple migrations get a new major-version folder (keep the old one for the migration chain). Unreleased models are renamed in place rather than duplicated. 1. Edit the relevant component(s) under `authz/openfga//components/*.fga` (e.g. add `define can_undrop: modify` to `view.fga`), then regenerate and validate: ```bash just update-openfga # fga model transform /fga.mod > /schema.json just test-openfga # runs the /store.fga.yaml assertions ``` (Requires the `fga` CLI — download from the [OpenFGA repo](https://github.com/openfga/cli/releases/).) 1. In `crates/authz-openfga/src/migration.rs` bump `ACTIVE_MODEL_VERSION` to the new version. For backward-compatible changes, repoint the current `add_model_*_current` call (schema-path `include_str!` + version). For tuple-migrating changes, add another `add_model` call carrying the migration fn. 1. Record the change under the new version heading in `authz/openfga/README.md`. ## Building the docs locally ```bash cd site just serve ``` --- # Query Engines Source: https://docs.lakekeeper.io/docs/latest/engines/ # Query Engines In this page we document how query engines can be configured to connect to Lakekeeper. Please also check the documentation of your query engine to obtain additional information. All Query engines that support the Apache Iceberg REST Catalog (IRC) also support Lakekeeper. If Lakekeeper Authorization is enabled, Lakekeeper enforces permissions based on the `sub` field in the received tokens. For query engines used by a single user, the user should use its own credentials to log-in to Lakekeeper. For query engines shared by multiple users, Lakekeeper supports two architectures that allow a shared query engine to enforce permissions for individual users: 1. OAuth2 enabled query engines should use standard OAuth2 Token-Exchange to exchange the user's token of the query engine for a Lakekeeper token (RFC8693). The Catalog then receives a token that has the `sub` field set to the user using the query engine, instead of the technical user that is used to configure the catalog in the query engine itself. 2. Query engines flexible enough to connect to external permission management systems such as Open Policy Agent (OPA), can directly enforce the same permissions on Data that Lakekeeper uses. Please find more information and a complete docker compose example with trino in the [Open Policy Agent Guide](opa.md). Shared query engines must use the same Identity Provider as Lakekeeper in both scenarios unless user-ids are mapped, for example in OPA. We are tracking open issues and missing features in query engines in a [Tracking Issue on GitHub](https://github.com/lakekeeper/lakekeeper/issues/399). ## Generic Iceberg REST Clients All Apache Iceberg REST clients are compatible with Lakekeeper, as Lakekeeper fully implements the standard Iceberg REST Catalog API specification. This page only contains some exemplary tools and configurations to help you get started. For tools not listed here, please refer to their documentation for specific configuration details and best practices when connecting to an Iceberg REST Catalog. Always check with your tool provider for the most up-to-date information regarding supported features and configuration options. When using Lakekeeper with authentication enabled, remember that you can follow the approaches described at the beginning of this page: either use credentials specific to individual users or leverage OAuth2 token exchange for shared query engines. The authentication parameters typically include credential pairs, OAuth2 server URIs, and scopes as shown in the examples above. ## DuckDB WASM {#duckdb-wasm} DuckDB WASM allows you to query Lakekeeper directly from your browser. If you are using the Lakekeeper UI, DuckDB WASM is pre-configured. To use DuckDB WASM from the Lakekeeper UI, there are two important requirements due to browser security restrictions: **Requirements:** 1. **Same-Origin Access**: The S3 endpoint must be accessible from your browser at the same URL/origin that Lakekeeper uses to access it. For example, if Lakekeeper accesses S3 at `http://my-s3-endpoint:9000`, your browser must also be able to reach it at `http://my-s3-endpoint:9000`. This means the Docker Compose examples won't work with DuckDB WASM out of the box, as the S3 endpoint is typically only accessible within the Docker network, while your browser is not in this network. 2. **CORS Policy**: Your S3 storage must be configured with a CORS policy that allows requests from the Lakekeeper origin. See the [CORS Configuration guide](storage.md#cors-configuration) for setup instructions. ## DuckDB Basic setup in DuckDB: ```python import duckdb CATALOG_URL = "http://localhost:8181/catalog" WAREHOUSE = "my_warehouse" # Required if OAuth2 authentication is enabled for Lakekeeper CLIENT_ID = "your-client-id" CLIENT_SECRET = "your-client-secret" KEYCLOAK_TOKEN_ENDPOINT = "http://your-idp/realms/iceberg/protocol/openid-connect/token" # Install and load Iceberg extension duckdb.sql("INSTALL ICEBERG;") duckdb.sql("LOAD ICEBERG;") # Create secret for authentication duckdb.sql(f""" CREATE SECRET lakekeeper_secret ( TYPE ICEBERG, CLIENT_ID '{CLIENT_ID}', CLIENT_SECRET '{CLIENT_SECRET}', OAUTH2_SCOPE 'lakekeeper', OAUTH2_SERVER_URI '{KEYCLOAK_TOKEN_ENDPOINT}' ) """) # Attach catalog duckdb.sql(f""" ATTACH '{WAREHOUSE}' AS my_datalake ( TYPE ICEBERG, ENDPOINT '{CATALOG_URL}', SECRET lakekeeper_secret ) """) # Query tables duckdb.sql("SELECT * FROM my_datalake.my_namespace.my_table").show() ``` ## Trino The following docker compose examples are available for trino: - [`Minimal`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal): No authentication - [`Access-Control-Simple`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control-simple): Lakekeeper secured with OAuth2, single technical User for trino - [`Access-Control-Advanced`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control-advanced): Single trino instance secured by OAuth2 shared by multiple users. Lakekeeper Permissions for each individual user enforced by trino via the Open Policy Agent bridge. If [Soft-Deletion](./concepts.md#soft-deletion) is enabled in Lakekeeper, make sure to set `"iceberg.unique-table-location" = 'true'`, to ensure that tables can be recreated in new locations while their dropped counterparts are waiting for expiration. As Lakekeeper supports nesting of namespaces, we recommend to set `"iceberg.rest-catalog.nested-namespace-enabled" = 'true'`. Basic setup in trino: === "S3-Compatible" Trino supports vended-credentials from Iceberg REST Catalogs for S3, so that no S3 credentials are required when creating the Catalog. ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.rest-catalog.vended-credentials-enabled" = 'true', "iceberg.unique-table-location" = 'true', "s3.region" = '', "fs.s3.enabled" = 'true' -- Required for some S3-compatible storages: "s3.path-style-access" = 'true', "s3.endpoint" = '', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` === "Azure" Trino supports vended-credentials from Iceberg REST Catalogs for Azure, so that no Storage Account credentials are required when creating the Catalog. Please find additional configuration Options in the [Trino docs](https://trino.io/docs/current/object-storage/file-system-azure.html#object-storage-file-system-azure--page-root). ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.unique-table-location" = 'true', "fs.azure.enabled" = 'true', "azure.auth-type" = 'OAUTH', "azure.oauth.client-id" = '', "azure.oauth.secret" = '', "azure.oauth.tenant-id" = '', "azure.oauth.endpoint" = 'https://login.microsoftonline.com//v2.0', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', -- Client-ID used to access Lakekeeper. Typically different to `azure.oauth.client-id`. "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` === "GCS" Trino supports vended-credentials from Iceberg REST Catalogs for GCS, so that no GCS credentials are required when creating the Catalog. Please find additional configuration Options in the [Trino docs](https://trino.io/docs/current/object-storage/file-system-gcs.html). ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.unique-table-location" = 'true', "fs.gcs.enabled" = 'true', "gcs.project-id" = '', "gcs.json-key" = '', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', -- Client-ID used to access Lakekeeper. Typically different to `azure.oauth.client-id`. "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` ## Starburst Starburst If [Soft-Deletion](./concepts.md#soft-deletion) is enabled in Lakekeeper, make sure to set `"iceberg.unique-table-location" = 'true'`, to ensure that tables can be recreated in new locations while their dropped counterparts are waiting for expiration. As Lakekeeper supports nesting of namespaces, we recommend to set `"iceberg.rest-catalog.nested-namespace-enabled" = 'true'`. Basic setup in Starburst: === "S3-Compatible" Starburst supports vended-credentials from Iceberg REST Catalogs for S3, so that no S3 credentials are required when creating the Catalog. Please find additional configuration Options in the [Starburst docs](https://docs.starburst.io/latest/object-storage/file-system-s3.html). ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.rest-catalog.vended-credentials-enabled" = 'true', "iceberg.unique-table-location" = 'true', "s3.region" = '', "fs.s3.enabled" = 'true' -- Required for some S3-compatible storages: "s3.path-style-access" = 'true', "s3.endpoint" = '', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` === "Azure" Starburst supports vended-credentials from Iceberg REST Catalogs for Azure, so that no Storage Account credentials are required when creating the Catalog. Please find additional configuration Options in the [Starburst docs](https://docs.starburst.io/latest/object-storage/file-system-azure.html). ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.unique-table-location" = 'true', "fs.azure.enabled" = 'true', "azure.auth-type" = 'OAUTH', "azure.oauth.client-id" = '', "azure.oauth.secret" = '', "azure.oauth.tenant-id" = '', "azure.oauth.endpoint" = 'https://login.microsoftonline.com//v2.0', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', -- Client-ID used to access Lakekeeper. Typically different to `azure.oauth.client-id`. "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` === "GCS" Starburst supports vended-credentials from Iceberg REST Catalogs for GCS, so that no GCS credentials are required when creating the Catalog. Please find additional configuration Options in the [Starburst docs](https://docs.starburst.io/latest/object-storage/file-system-gcs.html). ```sql CREATE CATALOG lakekeeper USING iceberg WITH ( "iceberg.catalog.type" = 'rest', "iceberg.rest-catalog.uri" = '', "iceberg.rest-catalog.warehouse" = '', "iceberg.rest-catalog.nested-namespace-enabled" = 'true', "iceberg.unique-table-location" = 'true', "fs.gcs.enabled" = 'true', "gcs.project-id" = '', "gcs.json-key" = '', -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.security" = 'OAUTH2', "iceberg.rest-catalog.oauth2.credential" = ':', -- Client-ID used to access Lakekeeper. Typically different to `azure.oauth.client-id`. "iceberg.rest-catalog.oauth2.server-uri" = '', -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.rest-catalog.oauth2.scope" = '' ) ``` ## Spark The following docker compose examples are available for spark: - [`Minimal`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal): No authentication - [`Access-Control-Simple`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control-simple): Lakekeeper secured with OAuth2, single technical User for spark Basic setup in spark: === "S3-Compatible / Azure / GCS" Spark supports credential vending for all storage types, so that no credentials need to be specified in spark when creating the catalog. ```python import pyspark import pyspark.sql pyspark_version = pyspark.__version__ pyspark_version = ".".join(pyspark_version.split(".")[:2]) # Strip patch version iceberg_version = "1.10.1" # Disable the jars which are not needed spark_jars_packages = ( f"org.apache.iceberg:iceberg-spark-runtime-{pyspark_version}_2.12:{iceberg_version}," f"org.apache.iceberg:iceberg-aws-bundle:{iceberg_version}," f"org.apache.iceberg:iceberg-azure-bundle:{iceberg_version}," f"org.apache.iceberg:iceberg-gcp-bundle:{iceberg_version}" ) catalog_name = "lakekeeper" configuration = { "spark.jars.packages": spark_jars_packages, "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.defaultCatalog": catalog_name, f"spark.sql.catalog.{catalog_name}": "org.apache.iceberg.spark.SparkCatalog", f"spark.sql.catalog.{catalog_name}.catalog-impl": "org.apache.iceberg.rest.RESTCatalog", f"spark.sql.catalog.{catalog_name}.uri": "", # Required Parameters if OAuth2 authentication is enabled for Lakekeeper: f"spark.sql.catalog.{catalog_name}.credential": ":", # Client-ID used to access Lakekeeper f"spark.sql.catalog.{catalog_name}.oauth2-server-uri": "", f"spark.sql.catalog.{catalog_name}.warehouse": "", # Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: f"spark.sql.catalog.{catalog_name}.scope": "", # Optional Parameter to configure which kind of vended-credential to use for S3: f"spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation": "vended-credentials" # Alternatively "remote-signing" } spark_conf = pyspark.SparkConf().setMaster("local[*]") for k, v in configuration.items(): spark_conf = spark_conf.set(k, v) spark = pyspark.sql.SparkSession.builder.config(conf=spark_conf).getOrCreate() spark.sql(f"USE {catalog_name}") ``` ## PyIceberg ```python import pyiceberg.catalog import pyiceberg.catalog.rest import pyiceberg.typedef catalog = pyiceberg.catalog.rest.RestCatalog( name="my_catalog_name", uri="", warehouse="", # Required Parameters if OAuth2 authentication is enabled for Lakekeeper: credential=":", **{ "oauth2-server-uri": "http://localhost:30080/realms//protocol/openid-connect/token" }, # Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: scope="", ) print(catalog.list_namespaces()) ``` ## AWS Athena (Spark) Amazon Athena is a serverless query service that allows you to use SQL or PySpark to query data in Lakekeeper without provisioning infrastructure. The following steps demonstrate how to connect Athena PySpark with Lakekeeper. **1. Create an Apache Spark workgroup in the AWS Athena console:** * Go to the Athena console > Administration > Workgroups * Create a workgroup with Apache Spark as the analytics engine **2. Create a new PySpark notebook:** * Give your notebook a name * Select your Spark workgroup * Configure JSON properties with Lakekeeper catalog settings ```json { "spark.sql.catalog.lakekeeper": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.lakekeeper.type": "rest", "spark.sql.catalog.lakekeeper.uri": "", "spark.sql.catalog.lakekeeper.warehouse": "", "spark.sql.defaultCatalog": "lakekeeper", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.lakekeeper.credential": ":", "spark.sql.catalog.lakekeeper.oauth2-server-uri": "" } ``` **3. Verify the connection in your notebook:** ```python # Verify connectivity to your Lakekeeper catalog spark.sql("select count(*) from lakekeeper..").show() ``` Amazon Athena has Iceberg pre-installed, so no additional package installations are required. ## Starrocks Starrocks is improving the Iceberg REST support quickly. This guide is written for Starrocks 3.3, which does not support vended-credentials for AWS S3 with custom endpoints. The following docker compose examples are available for starrocks: - [`Minimal`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/minimal): No authentication - [`Access-Control`](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control): Lakekeeper secured with OAuth2, single technical user for starrocks **Note:** If you are using an IdP like Keycloak, in order for Starrocks to be able to authenticate with Lakekeeper you must ensure the client you are connecting to has "Standard Token Exchange" (or equivalent) enabled. Otherwise Starrocks will be unable to refresh access tokens and you will get authentication errors when the initial access token created by the `CREATE EXTERNAL CATALOG` command expires. === "S3-Compatible" ```sql CREATE EXTERNAL CATALOG rest_catalog PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.warehouse" = "", -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.catalog.security" = "OAUTH2", "iceberg.catalog.oauth2-server-uri" = "", "iceberg.catalog.credential" = ":", -- Optional Parameters if OAuth2 authentication is enabled for Lakekeeper: "iceberg.catalog.scope" = "", -- S3 specific configuration, probably not required anymore in version 3.4.1 and newer. "aws.s3.region" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", -- Required for some S3-compatible storages: "aws.s3.endpoint" = "", "aws.s3.enable_path_style_access" = "true" ) -- You must set your catalog in the current session before you can query Iceberg data SET CATALOG rest_catalog; -- Starrocks uses MySQL compatible terminology. This is equivalent to Namespaces SHOW DATABASES; -- Starrocks will let you create resources in Lakekeeper CREATE DATABASE testing; -- You must use your namespace like a SQL database USE `testing`; -- In this case Tables is the same between MySQL and Iceberg. SHOW TABLES; -- You can also create tables, INSERT INTO them, and query them just like you would any other SQL database. ``` ## OLake OLake is an open-source, quick and scalable tool for replicating Databases to Apache Iceberg or Data Lakehouses written in Go. Visit the [Olake Iceberg Documentation](https://olake.io/docs/writers/iceberg/catalog/rest#rest-catalog) for the full documentation, and additional information on Olake. === "S3-Compatible" ```json { "type": "ICEBERG", "writer": { "catalog_type": "rest", "normalization": false, "rest_catalog_url": "http://localhost:8181/catalog", "iceberg_s3_path": "warehouse", "iceberg_db": "ICEBERG_DATABASE_NAME" } } ``` ## RisingWave [RisingWave](https://www.risingwave.com/) is a distributed SQL streaming database that is wire-compatible with PostgreSQL, designed for real-time data ingestion, processing, and querying. Unlike many other query engines that use a `CATALOG` abstraction, RisingWave connects to Lakekeeper through a `CONNECTION` object, which allows it to use Iceberg tables for sources, sinks, and internal tables. For a hands-on example, a Docker Compose setup is available in the [RisingWave repository](https://github.com/risingwavelabs/risingwave). You can find detailed deployment instructions in the [official RisingWave documentation](https://docs.risingwave.com/iceberg/catalogs/lakekeeper#deploy-with-docker). Once you have both services running, you can create a `CONNECTION` in RisingWave to connect to Lakekeeper. The following is an example configuration. As parameters may change over time, please refer to the [official RisingWave documentation](https://docs.risingwave.com/iceberg/catalogs/lakekeeper) for the most up-to-date and complete configuration options. ```sql CREATE CONNECTION lakekeeper_catalog_conn WITH ( type = 'iceberg', catalog.type = 'rest', catalog.uri = 'http://lakekeeper:8181/catalog/', warehouse.path = 'risingwave-warehouse', s3.access.key = 'hummockadmin', s3.secret.key = 'hummockadmin', s3.path.style.access = 'true', s3.endpoint = 'http://minio-0:9301', s3.region = 'us-east-1' ); ``` After creating the connection, you must set it as the default for your session to create and query internal Iceberg tables. The `SET` command applies the change to the current session only, while `ALTER SYSTEM` makes it persistent across restarts. ```sql -- Set for the current session SET iceberg_engine_connection = 'public.lakekeeper_catalog_conn'; -- Set persistent for the system ALTER SYSTEM SET iceberg_engine_connection = 'public.lakekeeper_catalog_conn'; ``` ## Apache Fluss [Apache Fluss](https://fluss.apache.org/) is a streaming storage system that can tier streaming data into Iceberg tables via its [Streaming Lakehouse](https://fluss.apache.org/docs/streaming-lakehouse/overview/) feature. Lakekeeper can be used as the Iceberg REST catalog for this tiering, so that tiered data is immediately queryable by any Iceberg-compatible engine through Lakekeeper. For details on how Fluss integrates with Iceberg specifically, see the [Fluss Iceberg integration docs](https://fluss.apache.org/docs/streaming-lakehouse/integrate-data-lakes/iceberg/). To point Fluss at Lakekeeper, set the following properties in `server.yaml`. Fluss strips the `datalake.iceberg.` prefix and passes the remainder as native Iceberg REST catalog properties. The snippet below assumes Lakekeeper is running without authentication; if authentication is enabled, additional properties need to be set (see f. ex. [Spark](#spark)). ```yaml datalake.format: iceberg datalake.iceberg.type: rest datalake.iceberg.uri: http://:/catalog datalake.iceberg.warehouse: ``` A Docker Compose example including Fluss, the tiering service, and Lakekeeper is available in the `examples/fluss` directory of the Lakekeeper repository. ## Firebolt Firebolt [Firebolt](https://www.firebolt.io/) is a high-performance, scale-out analytical database. [Firebolt Core](https://github.com/firebolt-db/firebolt-core) is the free, self-hosted edition packaged as a single Docker image. Both connect to Lakekeeper through the same `CREATE LOCATION` syntax. Firebolt supports vended-credentials from Iceberg REST Catalogs for S3, so no S3 credentials need to be configured on Firebolt. This works for both AWS S3 and self-hosted S3-compatible object stores (e.g. MinIO, RustFS). === "S3-Compatible" ```sql CREATE LOCATION lakekeeper WITH SOURCE = ICEBERG CATALOG = REST CATALOG_OPTIONS = ( URL = '' WAREHOUSE = '' NAMESPACE = '' TABLE = '
' ) -- Required Parameters if OAuth2 authentication is enabled for Lakekeeper: CREDENTIALS = ( OAUTH_CLIENT_ID = '' OAUTH_CLIENT_SECRET = '' OAUTH_SERVER_URL = '' -- Optional: OAUTH_SCOPE = '' ); -- Read the table SELECT * FROM READ_ICEBERG(LOCATION => 'lakekeeper'); ``` Refer to the [Firebolt CREATE LOCATION (Iceberg) docs](https://docs.firebolt.io/reference-sql/commands/data-definition/create-location-iceberg) for additional options. --- # Apache Flink (Java) Source: https://docs.lakekeeper.io/docs/latest/generic-tables-flink/ # Apache Flink This guide shows how to stream data into Lakekeeper from [Apache Flink](https://flink.apache.org/) using the Lakekeeper Java client. Because Lakekeeper vends short-lived, prefix-scoped storage credentials, Flink writes directly to your object store (S3, GCS, Azure) without any long-lived keys in your job configuration. The companion [`flink` example](https://github.com/lakekeeper/lakekeeper-clients/tree/main/java/examples/flink) streams synthetic IoT sensor readings into a Lakekeeper **generic table** (`format: dataset`), rolling a new JSON file to the table's storage location on every checkpoint. ``` sensor-001 ┐ sensor-002 ┼─► Flink job ─► FileSink ─► s3://bucket/prefix/iot/sensor-readings/ ... ┘ ├── part-0-0.json ← 10 records ├── part-0-1.json ← 10 records └── ... ``` ## How it works The job runs in three stages: 1. **Register + vend** — the Lakekeeper Java client creates the generic table (idempotent — a `409 Conflict` on an existing table is ignored), then reloads it with `vended=true` to obtain short-lived STS credentials scoped to the table's storage prefix. 2. **Credential wiring** — the vended credentials are injected into Hadoop S3A config (`fs.s3a.*`) **before** the Flink stream graph is built. Flink's `flink-s3-fs-hadoop` plugin picks them up via `ServiceLoader`. 3. **Streaming sink** — a `FileSink` with a custom bulk writer emits each rolled file as a valid JSON array. Files roll on every Flink checkpoint (every `BATCH_INTERVAL_MS` ms) and commit with a `.json` suffix. !!! note "Generic tables, not Iceberg" This example uses Lakekeeper's [Generic Table API](generic-tables.md) — Lakekeeper catalogs the dataset for identity, governance, and credential vending, but does not commit format-specific metadata. Flink writes directly to storage. For Iceberg tables, use Flink's native Iceberg connector against the Iceberg REST endpoint instead. ## Prerequisites - Java 11+ - A running Lakekeeper instance with a warehouse and namespace already created - The warehouse's object-store backend reachable from where the job runs ## Configure Copy the example environment file and fill in your values: ```sh cp java/.env.local.example java/.env.local ``` **Required:** | Variable | Description | |---|---| | `LAKEKEEPER` | Lakekeeper base URL (default: `http://localhost:8181`) | | `WAREHOUSE_ID` | Warehouse UUID (used as the URL path prefix — use the UUID, not the warehouse name) | | `TOKEN` | Static bearer token **—or—** the `OAUTH_*` variables below | | `OAUTH_TOKEN_URL`, `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, `OAUTH_SCOPE` | OAuth2 client-credentials flow (alternative to `TOKEN`) | **Optional tuning:** | Variable | Default | Description | |---|---|---| | `NAMESPACE` | `iot` | Lakekeeper namespace | | `TABLE` | `sensor-readings` | Table name | | `NUM_SENSORS` | `5` | Number of virtual sensors | | `NUM_RECORDS` | `-1` | Total records to emit; `-1` streams forever | | `BATCH_SIZE` | `10` | Records written per file | | `BATCH_INTERVAL_MS` | `15000` | Milliseconds between batches (= file roll interval) | ## Run locally ```sh cd java ./gradlew :examples:flink:run ``` Gradle reads `java/.env.local` automatically. The job streams until you press ++ctrl+c++. Expected output: ``` Created iot.sensor-readings → s3://your-bucket/prefix/iot/sensor-readings Location: s3://your-bucket/prefix/iot/sensor-readings [Lakekeeper] vended credential keys: [s3.access-key-id, s3.secret-access-key, s3.session-token] Streaming 10 records/file every 15s from 5 sensors → s3://your-bucket/... ``` A new `.json` file appears at the storage location every ~15 seconds. ## Submit to a Flink cluster Build the self-contained fat JAR: ```sh cd java ./gradlew :examples:flink:shadowJar # → examples/flink/build/libs/flink--all.jar ``` Submit it: ```sh flink run examples/flink/build/libs/flink-*-all.jar ``` !!! warning "Passing credentials in production" The configuration is read from environment variables, which must be available to the **TaskManager** JVM. Use Flink's `env.java.opts` or cluster-level secret management to inject `TOKEN`/`OAUTH_*` — never pass secrets on the `flink run` command line in production. ## Credential vending details Lakekeeper returns Iceberg-style `s3.*` config keys, but the Hadoop S3A filesystem that Flink uses reads `fs.s3a.*`. The example translates between them: | Lakekeeper key | Hadoop S3A key | |---|---| | `s3.access-key-id` | `fs.s3a.access.key` | | `s3.secret-access-key` | `fs.s3a.secret.key` | | `s3.session-token` | `fs.s3a.session.token` | | `s3.endpoint` / `client.endpoint` | `fs.s3a.endpoint` | Two details matter when using **vended STS credentials**: - Because a session token is present, the job sets `fs.s3a.aws.credentials.provider` to `TemporaryAWSCredentialsProvider`. The default `SimpleAWSCredentialsProvider` ignores the session token, and AWS rejects the bare STS access key with `InvalidAccessKeyId`. - STS session policies typically omit `s3:DeleteObject`. S3A creates and then deletes directory-marker objects, so the job sets `fs.s3a.directory.marker.retention=keep` to avoid an `AccessDenied` error. ## Related - [Generic Tables](generic-tables.md) — the catalog API this example writes against - [Query Engines](engines.md) — connecting Iceberg-native engines to Lakekeeper - [Storage](storage.md) — configuring S3, GCS, and Azure warehouses --- # Python Client Source: https://docs.lakekeeper.io/docs/latest/generic-tables-pylakekeeper/ # Python Client (pylakekeeper) `pylakekeeper` is the official Python client for Lakekeeper's [Generic Table API](generic-tables.md). It is a small, standalone library — **not** a general Iceberg REST client — that handles two things you otherwise hand-roll: catalog CRUD for generic tables, and OAuth2 authentication (static token or client-credentials with automatic token refresh). Its main job is **credential vending**: you ask Lakekeeper to load a table with `vended=True`, and the client hands you short-lived, prefix-scoped storage credentials already translated into the shape your engine expects — Lance `storage_options`, `fsspec` kwargs, or plain AWS keys for `boto3`. !!! note "Generic tables only" This client covers the generic-tables surface (Lance, `dataset`, Parquet, CSV, any format). For Iceberg tables use a standard Iceberg REST client such as [PyIceberg](engines.md#pyiceberg) pointed at Lakekeeper's catalog endpoint. ## Install ```sh pip install pylakekeeper # with the Lance object-store helpers: pip install 'pylakekeeper[lance]' ``` Requires Python 3.10+. The core install depends only on `httpx` and `pydantic`. ## Connect Create a `Client` with your Lakekeeper URL, a **warehouse UUID** (it becomes the URL path prefix — use the UUID, not the warehouse name), and an auth strategy. Use it as a context manager to release the connection pool. ```python from pylakekeeper import Client, StaticToken, ClientCredentials # Static bearer token (e.g. the no-auth `minimal` stack accepts TOKEN=dev): client = Client( base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("my-token"), ) # ...or OAuth2 client credentials (the client refreshes the token automatically): client = Client( base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=ClientCredentials( token_url="http://keycloak/realms/iceberg/protocol/openid-connect/token", client_id="...", client_secret="...", scope="lakekeeper", ), ) ``` !!! info "Warehouse and namespace must already exist" The client creates *tables*, not warehouses or namespaces — warehouse administration is intentionally out of scope. Create the warehouse via the UI or [Management API](api/management.md) and the namespace via the [Catalog API](api/catalog.md) first. ## Generic-tables API All table operations hang off `client.generic_tables`: | Method | Description | |---|---| | `create(namespace, name, format=..., base_location=..., doc=..., properties=...)` | Create a table; returns the load response. Raises `ConflictError` if it exists. | | `load(namespace, name, vended=False)` | Load a table. With `vended=True`, the response carries inline STS credentials. | | `list(namespace, page_size=100)` | Iterate every table identifier in a namespace, following pagination. | | `drop(namespace, name)` | Drop a table. | Namespaces are given as dotted strings (`"ai.test"`) or lists (`["ai", "test"]`); multi-level namespace encoding is handled for you. The `load(..., vended=True)` response exposes ready-to-use credential shapes: - `resp.location` — the table's storage URI (e.g. `s3://bucket/prefix/...`) - `resp.lance_storage_options` — a dict of AWS-style keys for Lance / `boto3` - `resp.fsspec_kwargs` — kwargs to unpack into `fsspec.filesystem("s3", ...)` ## Example: Lance table The `lance` format stores a columnar dataset — here, vector embeddings. Create the table, load vended credentials, then write and read directly through Lance: ```python import lance from pylakekeeper import Client, StaticToken, GenericTableFormat, ConflictError with Client(base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("dev")) as c: # Create the table (idempotent — tolerate re-runs). try: c.generic_tables.create( "ai.test", "image_embeddings", format=GenericTableFormat.LANCE, # or just "lance" properties={"embedding-dim": "768"}, ) except ConflictError: pass # already exists # Load with vended credentials → base location + short-lived STS creds. t = c.generic_tables.load("ai.test", "image_embeddings", vended=True) # Write, then reload (STS creds are short-lived) and read back. lance.write_dataset(my_arrow_table, t.location, storage_options=t.lance_storage_options, mode="overwrite") t = c.generic_tables.load("ai.test", "image_embeddings", vended=True) ds = lance.dataset(t.location, storage_options=t.lance_storage_options) print("rows:", ds.count_rows()) ``` `t.lance_storage_options` maps Lakekeeper's Iceberg-style credential keys (`s3.access-key-id`, …) to the `aws_access_key_id`, `aws_secret_access_key`, `aws_session_token`, `aws_region`, and `aws_endpoint` names Lance expects (and adds `allow_http=true` for plaintext endpoints like MinIO). !!! tip "Every format shares the same catalog flow" Create → `load(vended=True)` → write → reload (STS creds are short-lived) → read. Only two things change per format: the **library** that does the I/O, and the **shape** the vended credentials must take. The client maps the Lance/`boto3` shape for you (`t.lance_storage_options`); Delta wants `UPPER_SNAKE` names, so you remap those yourself. ## Example: Delta table The `delta` format writes a Delta Lake table with [`deltalake`](https://pypi.org/project/deltalake/) (`pip install deltalake`). Delta reads storage options under **`UPPER_SNAKE`** names (`AWS_ACCESS_KEY_ID`, …) rather than Lance's lower-case shape, so remap the vended keys: ```python from deltalake import DeltaTable, write_deltalake from pylakekeeper import Client, StaticToken, GenericTableFormat, ConflictError # deltalake wants UPPER_SNAKE storage-option names; the client only maps the Lance shape. ICEBERG_TO_DELTA = { "s3.access-key-id": "AWS_ACCESS_KEY_ID", "s3.secret-access-key": "AWS_SECRET_ACCESS_KEY", "s3.session-token": "AWS_SESSION_TOKEN", "s3.region": "AWS_REGION", "client.region": "AWS_REGION", "s3.endpoint": "AWS_ENDPOINT_URL", } def delta_options(t): props = {**{k: v for cred in (t.storage_credentials or []) for k, v in cred.config.items()}, **(t.config or {})} opts = {ICEBERG_TO_DELTA[k]: v for k, v in props.items() if k in ICEBERG_TO_DELTA} opts.setdefault("AWS_S3_ALLOW_UNSAFE_RENAME", "true") # plain S3 has no atomic rename if opts.get("AWS_ENDPOINT_URL", "").startswith("http://"): opts["AWS_ALLOW_HTTP"] = "true" # MinIO/SeaweedFS over http return opts with Client(base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("dev")) as c: try: c.generic_tables.create("ai.test", "events", format=GenericTableFormat.DELTA) except ConflictError: pass t = c.generic_tables.load("ai.test", "events", vended=True) write_deltalake(t.location, my_arrow_table, storage_options=delta_options(t), mode="overwrite") t = c.generic_tables.load("ai.test", "events", vended=True) # refresh STS creds dt = DeltaTable(t.location, storage_options=delta_options(t)) print("rows:", dt.to_pyarrow_table().num_rows) ``` ## Example: Vortex table The [`vortex`](https://pypi.org/project/vortex-data/) writer (`pip install vortex-data`) targets a local path, so the pattern is **write-local-then-upload**: build a `boto3` client from the vended credentials (the Lance option names double as `boto3` kwargs), write the `.vortex` file to a temp dir, then upload it to the table location. ```python import os, tempfile from urllib.parse import urlparse import boto3 import vortex as vx from pylakekeeper import Client, StaticToken, GenericTableFormat, ConflictError with Client(base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("dev")) as c: try: c.generic_tables.create("ai.test", "signals", format=GenericTableFormat.VORTEX) except ConflictError: pass t = c.generic_tables.load("ai.test", "signals", vended=True) opts = t.lance_storage_options # lower_snake keys double as boto3 kwargs s3 = boto3.client("s3", aws_access_key_id=opts["aws_access_key_id"], aws_secret_access_key=opts["aws_secret_access_key"], aws_session_token=opts.get("aws_session_token"), region_name=opts.get("aws_region"), endpoint_url=opts.get("aws_endpoint")) parsed = urlparse(t.location) bucket, key = parsed.netloc, f"{parsed.path.strip('/')}/data.vortex" with tempfile.TemporaryDirectory() as tmp: # write local = os.path.join(tmp, "data.vortex") vx.io.write(vx.array(my_arrow_table), local) s3.upload_file(local, bucket, key) with tempfile.TemporaryDirectory() as tmp: # read back local = os.path.join(tmp, "data.vortex") s3.download_file(bucket, key, local) table = vx.open(local).scan().read_all().to_arrow_table() print("rows:", table.num_rows) ``` ## Example: Paimon table [`pypaimon`](https://pypi.org/project/pypaimon/) (`pip install pypaimon`, needs a JVM on `PATH`) also writes through a local filesystem catalog, so it uses the same **write-local-then-upload** approach as Vortex — except a Paimon table is a *directory tree*, so the whole tree is walked and uploaded. ```python import os, tempfile from urllib.parse import urlparse import boto3 from pypaimon import CatalogFactory, Schema from pylakekeeper import Client, StaticToken, GenericTableFormat, ConflictError with Client(base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("dev")) as c: try: c.generic_tables.create("ai.test", "orders", format=GenericTableFormat.PAIMON) except ConflictError: pass t = c.generic_tables.load("ai.test", "orders", vended=True) opts = t.lance_storage_options s3 = boto3.client("s3", aws_access_key_id=opts["aws_access_key_id"], aws_secret_access_key=opts["aws_secret_access_key"], aws_session_token=opts.get("aws_session_token"), region_name=opts.get("aws_region"), endpoint_url=opts.get("aws_endpoint")) parsed = urlparse(t.location) bucket, prefix = parsed.netloc, parsed.path.strip("/") with tempfile.TemporaryDirectory() as tmp: catalog = CatalogFactory.create({"warehouse": tmp}) catalog.create_database("default", True) schema = Schema.from_pyarrow_schema(my_arrow_table.schema, primary_keys=["id"], options={"bucket": "1"}) catalog.create_table("default.orders", schema, True) table = catalog.get_table("default.orders") wb = table.new_batch_write_builder() writer = wb.new_write() writer.write_arrow(my_arrow_table) commit = wb.new_commit() commit.commit(writer.prepare_commit()) writer.close(); commit.close() # Upload the local table tree to the vended location. table_dir = os.path.join(tmp, "default.db", "orders") for root, _dirs, files in os.walk(table_dir): for f in files: local = os.path.join(root, f) rel = os.path.relpath(local, table_dir).replace(os.sep, "/") s3.upload_file(local, bucket, f"{prefix}/{rel}") print("uploaded paimon table →", t.location) ``` Reading a Paimon table back is the mirror image: download the object tree under the prefix into a temp dir, then open it with a local `CatalogFactory(...).get_table("default.orders")` and a `new_read_builder()`. ## Example: dataset format (unstructured files) The `dataset` format catalogs unstructured data — raw files rather than a columnar table. Lakekeeper vends credentials; the actual upload is done with a backend-specific client (`boto3` for S3, `azure-storage-blob` for ADLS, `google-cloud-storage` for GCS). Only the upload step changes — the catalog flow is identical. ```python import boto3, mimetypes from pathlib import Path from urllib.parse import urlparse from pylakekeeper import Client, StaticToken, GenericTableFormat, ConflictError with Client(base_url="http://localhost:8181", warehouse="my-warehouse-uuid", auth=StaticToken("dev")) as c: try: c.generic_tables.create("ai.test", "product_images", format=GenericTableFormat.DATASET, doc="product images") except ConflictError: pass t = c.generic_tables.load("ai.test", "product_images", vended=True) # Build a boto3 client straight from the vended credentials. opts = t.lance_storage_options s3 = boto3.client("s3", aws_access_key_id=opts["aws_access_key_id"], aws_secret_access_key=opts["aws_secret_access_key"], aws_session_token=opts.get("aws_session_token"), region_name=opts.get("aws_region"), endpoint_url=opts.get("aws_endpoint"), # None on real AWS; set for MinIO/SeaweedFS ) parsed = urlparse(t.location) # s3:/// bucket, prefix = parsed.netloc, parsed.path.strip("/") for p in Path("images").iterdir(): s3.put_object(Bucket=bucket, Key=f"{prefix}/{p.name}", Body=p.read_bytes(), ContentType=mimetypes.guess_type(p.name)[0] or "application/octet-stream") ``` ## Authentication | Strategy | Class | Use when | |---|---|---| | Static token | `StaticToken(token)` | You already have a bearer token, or the target is a no-auth dev stack. | | Client credentials | `ClientCredentials(token_url=, client_id=, client_secret=, scope=)` | Production OAuth2 — the client fetches and refreshes the token for you. | See [Authentication](authentication.md) for how Lakekeeper validates tokens and maps them to identities. ## Related - [Generic Tables](generic-tables.md) — the catalog concept this client operates on - [Apache Flink](generic-tables-flink.md) — the same vending flow from Java/Flink - [Query Engines](engines.md) — Iceberg-native engines against Lakekeeper - Source & examples: [`lakekeeper-clients` on GitHub](https://github.com/lakekeeper/lakekeeper-clients/tree/main/python) --- # Apache Spark (PySpark) Source: https://docs.lakekeeper.io/docs/latest/generic-tables-spark/ # Apache Spark This page shows how to read and write a **Lance** [generic table](generic-tables.md) from **PySpark**. The Lakekeeper client resolves the table location and vends short-lived storage credentials; Spark (and `pylance`) then read/write the data directly. !!! note "For Iceberg tables, use the REST catalog instead" If you want plain Spark SQL over Iceberg tables (`spark.sql("SELECT ... FROM lakekeeper.ns.table")`), configure Lakekeeper as an Iceberg REST catalog — see [Query Engines → Spark](engines.md#spark). You do not need the client library or the credential wiring below for that. ## How it works The [`lakekeeper-client-spark`](https://github.com/lakekeeper/lakekeeper-clients/tree/main/java) JAR adds a small `LakekeeperSpark` helper on top of the Java client. From PySpark you reach it through `spark._jvm` (py4j) — no separate Python package is required. The helper: 1. Loads the generic table with `vended=true` to get short-lived STS credentials. 2. Translates the Iceberg-style `s3.*` credentials into Hadoop `fs.s3a.*` config and applies them to the active `SparkSession`. 3. Reads/writes the table's storage location with `spark.read().format()` / `df.write().format()`, where the format comes from the Lakekeeper table metadata. Because Lance also has a fast native Python writer (`pylance`), the example below **writes** with `pylance` and **reads back** through the Spark shim — but you can do both from Spark. ## Prerequisites ```sh pip install pyspark pylance pyarrow numpy ``` Download two JARs and point env vars at them: - `lakekeeper-client-spark-3.5_2.13-.jar` — from GitHub Packages - The [Lance Spark connector](https://github.com/lancedb/lance/releases) JAR | Variable | Description | |---|---| | `LAKEKEEPER` | Base URL (default `http://localhost:8181`) | | `WAREHOUSE_ID` | Warehouse **UUID** (used as the URL path prefix — not the name) | | `TOKEN` **or** `OAUTH_TOKEN_URL` / `OAUTH_CLIENT_ID` / `OAUTH_CLIENT_SECRET` / `OAUTH_SCOPE` | Auth | | `LAKEKEEPER_JAR` | Path to `lakekeeper-client-spark-*.jar` | | `LANCE_SPARK_JAR` | Path to the Lance Spark connector JAR | The warehouse and namespace must already exist; the client creates the table. ## Start Spark with the JARs ```python import os from pyspark.sql import SparkSession, DataFrame jars = ",".join(j for j in [os.environ["LAKEKEEPER_JAR"], os.environ.get("LANCE_SPARK_JAR", "")] if j) spark = ( SparkSession.builder .master("local[*]") .appName("lakekeeper-lance-demo") .config("spark.jars", jars) .getOrCreate() ) ``` ## Build the Lakekeeper client via py4j `spark._jvm` gives direct access to any class on the Spark JVM classpath: ```python jvm = spark._jvm # Auth — static token or OAuth2 client_credentials if token := os.environ.get("TOKEN"): auth = jvm.io.lakekeeper.client.auth.StaticToken(token) else: auth = jvm.io.lakekeeper.client.auth.ClientCredentials( os.environ["OAUTH_TOKEN_URL"], os.environ["OAUTH_CLIENT_ID"], os.environ["OAUTH_CLIENT_SECRET"], os.environ.get("OAUTH_SCOPE"), # scope (nullable) 60, # refreshMarginSeconds 30, # timeoutSeconds ) client = ( jvm.io.lakekeeper.client.LakekeeperClient.builder() .baseUrl(os.environ.get("LAKEKEEPER", "http://localhost:8181")) .warehouse(os.environ["WAREHOUSE_ID"]) .auth(auth) .build() ) ``` ## Create the Lance table ```python GenericTableFormat = jvm.io.lakekeeper.client.GenericTableFormat try: client.genericTables().create( "examples", "spark_lance_embeddings", GenericTableFormat.normalize(GenericTableFormat.LANCE), None, # base-location — let the server assign None, # doc {"embedding-dim": "128"}, # properties (Java Map via py4j) ) except Exception as e: # tolerate re-runs if "ConflictException" in type(e).__name__ or "409" in str(e): pass else: raise ``` ## Write with pylance, read back with Spark `LakekeeperSpark.read()` calls Lakekeeper, wires the vended credentials into `fs.s3a.*`, and returns a JVM `Dataset` — wrap it as a Python `DataFrame`: ```python import numpy as np, pyarrow as pa, lance # 1. Fresh vended credentials for the pylance write ICEBERG_TO_LANCE = { "s3.access-key-id": "aws_access_key_id", "s3.secret-access-key": "aws_secret_access_key", "s3.session-token": "aws_session_token", "s3.region": "aws_region", "client.region": "aws_region", "s3.endpoint": "aws_endpoint", } def lance_opts(resp): props = {} for cred in resp.getStorageCredentials(): props.update(cred.getConfig()) if resp.getConfig(): props.update(resp.getConfig()) opts = {ICEBERG_TO_LANCE[k]: v for k, v in props.items() if k in ICEBERG_TO_LANCE} if opts.get("aws_endpoint", "").startswith("http://"): opts["allow_http"] = "true" # MinIO/SeaweedFS over http return opts resp = client.genericTables().load("examples", "spark_lance_embeddings", True) rng = np.random.default_rng(42) data = pa.table({ "id": pa.array(range(1000), type=pa.int64()), "embedding": pa.FixedSizeListArray.from_arrays( pa.array(rng.standard_normal(1000 * 128).astype(np.float32)), 128), }) lance.write_dataset(data, resp.getLocation(), storage_options=lance_opts(resp), mode="overwrite") # 2. Read it back through the Spark shim LakekeeperSpark = jvm.io.lakekeeper.spark.LakekeeperSpark java_df = LakekeeperSpark.read(spark._jsparkSession, client, "examples", "spark_lance_embeddings") df = DataFrame(java_df, spark) print("rows:", df.count()) df.select("id").show(5) ``` ## Write back from Spark `LakekeeperSpark.write()` fetches fresh vended credentials, then saves the DataFrame to the table location. Pass write options (including `mode`) as a `java.util.HashMap`: ```python opts = jvm.java.util.HashMap() opts.put("mode", "overwrite") subset = df.filter("id < 100") LakekeeperSpark.write(spark._jsparkSession, client, "examples", "spark_lance_embeddings", subset._jdf, opts) ``` When you're done, release the client and Spark: ```python client.close() spark.stop() ``` ## Related - [Query Engines → Spark](engines.md#spark) — Iceberg tables via the REST catalog (no client library needed) - [Generic Tables](generic-tables.md) — the catalog concept - [Python Client](generic-tables-pylakekeeper.md) — the same vending flow in pure Python (no Spark) - [Apache Flink](generic-tables-flink.md) — streaming into generic tables from Java - Source & notebook: [`lakekeeper-clients` on GitHub](https://github.com/lakekeeper/lakekeeper-clients/tree/main/python/examples) --- # Overview Source: https://docs.lakekeeper.io/docs/latest/generic-tables/ # Generic Tables Lakekeeper's **Generic Table API** catalogs non-Iceberg tables — Lance, CSV, Parquet, or any other format — alongside Iceberg tables in the same Warehouse. Each generic table sits in a Namespace, has a name, a `format` string, an optional base `location`, `schema`, `statistics`, `properties`, and a free-form `doc` field. Engines handle writes against the underlying format; Lakekeeper handles **identity, governance, access control, and lifecycle**. ![generic tables in the Console](../../assets/generic-tables.png){ width="50%" } Unlike Iceberg tables, Lakekeeper does not commit format-specific metadata for generic tables — readers and writers go directly to the storage location after obtaining catalog metadata and credentials. This makes the API format-agnostic: any future or experimental format works without changes to the catalog. ## When to use generic tables - **Raw landing zones** — register CSV, JSON, or Parquet drops so they show up in the same catalog and inherit the same permissions as downstream Iceberg tables. - **Lance for multimodal AI** — store text, image embeddings, raw bytes, and scalar features in one table, then run vector + SQL queries via LanceDB, DuckDB, or Polars. See the example below. - **Experimental formats** — prototype new file/table formats behind the same `/credentials`, soft-delete, and authorization plumbing as production Iceberg tables. ## Capabilities Generic tables are first-class citizens. Most of Lakekeeper's table-side machinery applies: | Feature | Generic tables | Notes | |---------|:--------------:|-------| | Credentials vending (S3, GCS, Azure) | :white_check_mark: | `GET /lakekeeper/v1/{prefix}/namespaces/{ns}/generic-tables/{t}/credentials` | | [Soft-deletion](./concepts.md#soft-deletion) + undrop | :white_check_mark: | Respects per-warehouse soft-delete settings | | [Protection](./concepts.md#protection) flag | :white_check_mark: | `protected: bool` on load response; toggle via `GET/POST /management/v1/warehouse/{wh}/generic-table/{id}/protection`. Drops require `force=true` when set. | | Rename | :white_check_mark: | `POST /lakekeeper/v1/{prefix}/generic-tables/rename` | | Listing + pagination | :white_check_mark: | Same cursor scheme as Iceberg tables | | Per-action permissions | :white_check_mark: | 16 distinct actions (`drop`, `undrop`, `read_data`, `write_data`, `get_metadata`, `rename`, `change_ownership`, grant-* relations, ...) | | [Case-insensitive identifiers](./concepts.md#identifier-case-sensitivity) | :white_check_mark: | Cross-engine name resolution applies | | Name uniqueness across types | :white_check_mark: | A generic table cannot collide with an Iceberg table or view in the same Namespace | | Stored schema / statistics fields | :white_check_mark: | Free-form JSON; informational only | | Format-agnostic | :white_check_mark: | `format` is an opaque string | | Commit coordination | :x: | The catalog does not arbitrate writes — engines write directly | | Schema enforcement | :x: | Schema is stored, not validated against data files | ## Working with generic tables The flow is the same for every format: create the table, load it with *vended* credentials, then read or write directly with the format's own library. Lakekeeper only brokers metadata and short-lived storage credentials — never the data itself — so the same credentials path works for any format and only the reader library changes. You don't need to hand-roll these HTTP calls: - **Python** — the [Python client (`pylakekeeper`)](generic-tables-pylakekeeper.md) wraps create/load/list/drop and maps vended credentials into the keys Lance, `boto3`, and `fsspec` expect. See its [Lance example](generic-tables-pylakekeeper.md#example-lance-table). - **PySpark** — [Apache Spark](generic-tables-spark.md) reads and writes generic tables through the same vending flow. - **Java / Flink** — [Apache Flink](generic-tables-flink.md) shows the same vending flow. For a runnable end-to-end example (warehouse setup, STS credentials, create/load/drop, undrop, listing), see [`tests/integration-tests/lance/test_lance.py`](https://github.com/lakekeeper/lakekeeper/blob/main/tests/integration-tests/lance/test_lance.py). ## In the Console The Lakekeeper Console lists generic tables alongside Iceberg tables and views inside each Namespace, tagged with their `format` so you can tell them apart at a glance. Selecting one opens a detail view with its location, schema, statistics, properties, and the same actions (rename, protect, drop/undrop, permissions) available for Iceberg tables. Because the `format` field is an opaque string, any format shows up as a first-class entry — below are a few common ones. ### Lance Multimodal / vector tables written with Lance. The detail view surfaces the stored schema (embeddings, scalar features, raw bytes) and the storage location engines read from. ![Lance generic table in the Console](../../assets/generic-tables-lance.png){ width="100%" } ### Delta Delta Lake tables cataloged as generic tables — governed and permissioned in Lakekeeper while engines read/write the Delta log directly at the table location. ![Delta generic table in the Console](../../assets/generic-tables-delta.png){ width="100%" } ### Dataset A raw dataset (CSV / JSON / Parquet drop) registered so it appears in the catalog and inherits the surrounding Namespace's permissions. ![Dataset generic table in the Console](../../assets/generic-tables-dataset.png){ width="100%" } ## Authorization model Generic tables have an OpenFGA object type (`lakekeeper_generic_table`) parallel to `lakekeeper_table` and `lakekeeper_view`. Permissions inherit from the parent Namespace and Warehouse, and can be granted to users or roles via the standard `/management/v1/permissions/generic-table/{id}` endpoints. See [Authorization](./authorization.md). Because grants are per-action, you can — for example — give an ML platform service `read_data` and `get_metadata` on every generic table in a namespace without granting `drop` or `change_ownership`. ## Limits - The catalog does not coordinate concurrent writes. If your format requires commit coordination, the engine or the format library is responsible. - Schema and statistics fields are informational. Engines that need an authoritative schema should read it from the underlying files. - Generic tables and Iceberg tables share the namespace's identifier space — a name conflict across types is rejected at create time. --- # Gotchas Source: https://docs.lakekeeper.io/docs/latest/gotchas/ # Gotchas ## I got permissions but am still getting 403s Lakekeeper does not always return 404s for missing objects. If you are getting 403s while having correct grants, it is likely that the object you are trying to access does not exist. This is a security feature to prevent information leakage. ## I'm using Helm and the UI seems to hang forever Check out [our routing guide](./configuration.md#routing-and-base-url), both the catalog and UI create links pointing at the Lakekeeper instance. We use some heuristics by default and also offer a configuration escape hatch (`catalog.config.ICEBERG_REST__BASE_URI`). ### Examples ##### Local ```ssh k port-forward services/my-lakekeeper 7777:8181 ``` ```yaml catalog: # omitting the rest of the values config: # assuming that the catalog is forwarded to localhost:7777 ICEBERG_REST__BASE_URI: "http://localhost:7777" ``` ##### Public ```yaml catalog: # omitting the rest of the values config: # assuming that the catalog is reachable at https://lakekeeper.example.com ICEBERG_REST__BASE_URI: "https://lakekeeper.example.com" ``` ## Identifiers are case-insensitive All entity names (Warehouses, Namespaces, Tables, Views, Roles) are case-insensitive. If you create a table named `MyTable`, querying for `mytable` or `MYTABLE` will find it. Attempting to create `mytable` in the same namespace where `MyTable` already exists will fail with a conflict error. See [Identifier Case Sensitivity](./concepts.md#identifier-case-sensitivity) for details. ## I'm using Postgres <15 and the Lakekeeper database migrations fail with syntax error ``` Caused by: 0: error returned from database: syntax error at or near "NULLS" 1: syntax error at or near "NULLS" ``` Lakekeeper is currently compatible with Postgres >= 15 since we rely on `NULLS not distinct` which was added with PG 15. --- # Logging Source: https://docs.lakekeeper.io/docs/latest/logging/ # Logging ## Overview Lakekeeper emits structured JSON logs through the Rust `tracing` ecosystem. All logs include standard fields (`timestamp`, `level`, `message`, `target`) and can be filtered using the `RUST_LOG` environment variable. ## Controlling Log Output ### The RUST_LOG Environment Variable The `RUST_LOG` variable controls which logs are emitted based on their **level** and **target** (the Rust module that produced the log). This applies to all log types including audit logs, error responses, and general application logs. **Basic syntax:** ```bash # Set global minimum level RUST_LOG=info # Show INFO, WARN, ERROR RUST_LOG=debug # Show DEBUG and above RUST_LOG=warn # Show only WARN and ERROR # Filter by `target` RUST_LOG=lakekeeper=debug # Debug for lakekeeper, nothing else RUST_LOG=info,lakekeeper=debug # INFO globally, DEBUG for lakekeeper RUST_LOG=lakekeeper::service::events=trace # Trace only the events module ``` For production environments, use `RUST_LOG=info` to avoid excessive log volume while capturing all important operational events. You can optionally reduce noise from verbose dependencies (e.g., `RUST_LOG=info,sqlx=warn`). ### Audit Logs and RUST_LOG Audit logs are **enabled by default**. They will appear when `RUST_LOG` is set to `info` or higher (since audit logs are emitted at INFO level). To disable audit logs entirely: ```bash LAKEKEEPER__AUDIT__TRACING__ENABLED=false ``` **Note:** Audit logs contain PII. When disabling them, ensure you have alternative mechanisms for compliance and security monitoring. ## Log Types Lakekeeper produces three types of logs, distinguished by the `event_source` field: ### 1. Audit Logs {#audit-logs} Authorization events tracking access to catalog resources. **Contains PII** (user identities). **Identified by:** `"event_source": "audit"` Audit logs cover two distinct schemas depending on the source of the event: #### Authorization Events Emitted for every authz check. Always contain `action`/`actions`, `entity`/`entities`, `actor`, and `decision`. **Structure:** | Field | Type | Description | |------------------------|-----------------|-----------------------------------| | `event_source` | String | Always `"audit"` | | `action` or `actions` | Object or Array | Operation(s) attempted. Each action is an object with an `action_name` field (e.g., `"read_data"`, `"drop"`, `"create_namespace"`) and optional context fields (e.g., `properties`, `updated-properties`, `removed-properties`). See format below. | | `entity` or `entities` | Object or Array | Resource(s) accessed, containing `entity_type` and type-specific fields (e.g., `warehouse-id`, `namespace`, `table`) | | `actor` | Object | Who performed the action (see format below) | | `privilege_source` | String | Request-level classification of the caller's privilege: `"authorizer"` (no special privileges — all decisions come from the configured Authorizer backend), `"instance_admin"` (caller listed in `LAKEKEEPER__INSTANCE_ADMINS` — control-plane actions are auto-approved, data-plane actions still go through the Authorizer), or `"internal"` (in-process call — full bypass). This is a property of the request, not of individual entries in the `authorizations` array. See [Instance Admins](./authorization.md#instance-admins). | | `decision` | String | `"allowed"` or `"denied"` — the rollup decision for the whole event | | `authorizations` | Array | Per-decision breakdown. Always present and non-empty. Each entry is self-contained — see [Per-decision breakdown](#per-decision-breakdown-authorizations) below | | `context` | Object | Optional. Additional operation context (e.g., `project-id`, `warehouse-name`) | | `failure_reason` | Object | Only on failed events. Single-key object identifying the variant — one of `{"ActionForbidden": []}`, `{"ResourceNotFound": []}`, `{"CannotSeeResource": []}`, `{"InternalAuthorizationError": []}`, `{"InternalCatalogError": []}`, `{"InvalidRequestData": []}`. The empty array is the variant payload. | | `error` | Object | Only on failed events. Contains `type`, `message`, `code`, `error_id`, `stack` | **Note:** Empty arrays and objects are omitted from the output. For example, if `stack` is empty, the field will not appear in the log. **Actor Types:** ```json // Anonymous {"actor_type": "anonymous"} // Authenticated user {"actor_type": "principal", "principal": "oidc~user@example.com"} // Assumed role {"actor_type": "assumed-role", "principal": "oidc~user@example.com", "assumed_role": "role-id"} // Internal system {"actor_type": "lakekeeper-internal"} ``` **Action Format:** Each action is a structured object containing the operation name and optional context about the operation: ```json // Simple action (no context) {"action_name": "read_data"} // Action with properties context (e.g., create_namespace) {"action_name": "create_namespace", "properties": {"location": "s3://bucket/ns", "owner": "alice"}} // Action with update context (e.g., commit with property changes) {"action_name": "commit", "updated-properties": {"retention-days": "30"}, "removed-properties": ["staging"]} ``` When only a single action is involved, it appears as the `action` field. When multiple actions are checked the `actions` field contains an array. #### Per-decision breakdown (`authorizations`) Every authorization event carries an `authorizations` array with **at least one entry**. For ordinary single-check API calls the array has exactly one entry, synthesised from the event's top-level fields. For batch-style endpoints (e.g. `/management/v1/action/batch-check` and the various `get_*_actions` introspection endpoints) the array contains one entry per inner check, in request order. This means audit consumers can use **one query path** for both single and batch events: iterate `authorizations[]` and read the per-entry `allowed` flag, instead of switching between top-level `decision` and a per-batch breakdown. Each entry is **self-contained** — it does not require zipping with the top-level fields: | Field | Type | Description | |-----------------|---------|--------------------------------------------------------------------------------------| | `id` | String | Stable identifier for this entry. When the client supplies an `id` on a batch-check input it appears verbatim here, and the API response echoes the same value so the two can be correlated 1:1. When the client omits `id`, the API response omits it too; the audit log instead substitutes the request item's zero-based index as an internal bookkeeping fallback so individual decisions can still be pinpointed in the logs. **Do not assume the API response carries index-based ids — that fallback exists only in audit entries.** Absent on synthesised single-check entries. | | `for-principal` | Object | Optional. The principal whose permission was evaluated, when different from the request actor. Shape: `{"user": "..."}` or `{"role": "..."}`. Absent means the request actor itself. | | `action` | Object | Same shape as the top-level `action` field. | | `entity` | Object | Same shape as the top-level `entity` field. | | `allowed` | Boolean | The decision for *this* tuple. Absent when no definitive verdict was reached — e.g. on `InternalAuthorizationError`, `InternalCatalogError`, or `InvalidRequestData` failures, where the system never actually evaluated the request. Definitive denials (`ActionForbidden`, `ResourceNotFound`, `CannotSeeResource`) are recorded as `false`. | **Top-level vs. per-entry semantics.** The top-level `actor` always reflects the *API caller* (the bearer token holder); `authorizations[].for-principal` reflects *whose permissions were checked*. For most calls these are the same and `for-principal` is omitted. For introspection endpoints like `GET /lakekeeper/v1/permissions/...?for-user=X` the actor is the caller while every entry's `for-principal` is `X` — both facts are recorded structurally on the same event, no `context.for-user` string needed. **Examples:**
Authorization Succeeded ```json { "timestamp": "2026-02-15T14:20:50.758690Z", "level": "INFO", "event_source": "audit", "action": { "action_name": "create_warehouse", "name": "demo" }, "entity": { "entity_type": "project", "project-id": "00000000-0000-0000-0000-000000000000" }, "actor": { "actor_type": "principal", "principal": "oidc~94eb1d88-7854-43a0-b517-a75f92c533a5" }, "privilege_source": "authorizer", "decision": "allowed", "authorizations": [ { "action": { "action_name": "create_warehouse", "name": "demo" }, "entity": { "entity_type": "project", "project-id": "00000000-0000-0000-0000-000000000000" }, "allowed": true } ], "message": "Authorization succeeded event", "target": "lakekeeper::service::events::backends::audit" } ```
Authorization Failed ```json { "timestamp": "2026-02-15T14:21:10.123456Z", "level": "INFO", "event_source": "audit", "action": { "action_name": "drop" }, "entity": { "entity_type": "table", "warehouse-id": "414b18f0-0a6d-11f1-b2d7-f31430431ca0", "namespace": "production", "table": "sensitive_data" }, "actor": { "actor_type": "principal", "principal": "oidc~user@example.com" }, "privilege_source": "authorizer", "decision": "denied", "authorizations": [ { "action": { "action_name": "drop" }, "entity": { "entity_type": "table", "warehouse-id": "414b18f0-0a6d-11f1-b2d7-f31430431ca0", "namespace": "production", "table": "sensitive_data" }, "allowed": false } ], "failure_reason": { "ActionForbidden": [] }, "error": { "type": "Forbidden", "message": "Insufficient permissions", "code": 403, "error_id": "01234567-89ab-cdef-0123-456789abcdef" }, "message": "Authorization failed event", "target": "lakekeeper::service::events::backends::audit" } ```
Batch check (introspect_permissions) — multiple inner decisions A single `POST /management/v1/action/batch-check` call from `oidc~94eb1d88-…` asking whether `oidc~cfb55bf6-…` may `delete` a warehouse and `read_data` from a table. Top-level `actor` is the caller; each `authorizations[]` entry records the on-behalf-of principal and its individual decision. ```json { "timestamp": "2026-04-07T17:58:34.358975Z", "level": "INFO", "event_source": "audit", "action": { "action_name": "introspect_permissions" }, "entities": [ { "entity_type": "warehouse", "warehouse-id": "255a8f5c-32ab-11f1-889e-4706b6f66241" }, { "entity_type": "table", "warehouse-id": "255a8f5c-32ab-11f1-889e-4706b6f66241", "namespace": "production", "table": "events" } ], "actor": { "actor_type": "principal", "principal": "oidc~94eb1d88-7854-43a0-b517-a75f92c533a5" }, "privilege_source": "authorizer", "decision": "allowed", "authorizations": [ { "id": "warehouse-delete", "for-principal": { "user": "oidc~cfb55bf6-fcbb-4a1e-bfec-30c6649b52f8" }, "action": { "action_name": "delete" }, "entity": { "entity_type": "warehouse", "warehouse-id": "255a8f5c-32ab-11f1-889e-4706b6f66241" }, "allowed": true }, { "id": "1", "for-principal": { "user": "oidc~cfb55bf6-fcbb-4a1e-bfec-30c6649b52f8" }, "action": { "action_name": "read_data" }, "entity": { "entity_type": "table", "warehouse-id": "255a8f5c-32ab-11f1-889e-4706b6f66241", "namespace": "production", "table": "events" }, "allowed": false } ], "message": "Authorization succeeded event", "target": "lakekeeper::service::events::backends::audit" } ```
#### Operational Audit Events Emitted for non-authz operations that touch user identity (PII) — such as LDAP/directory role resolution and user enrichment. Use these to audit *what the system fetched on behalf of a user*, rather than *whether the user was allowed to do something*. **Structure:** | Field | Type | Description | |----------------|--------|----------------------------------------------------| | `event_source` | String | Always `"audit"` | | `operation` | String | Machine-readable name of the operation (e.g., `"ldap_resolve_roles"`) | | `actor` | Object | Same shape as authorization events: `{"actor_type": "principal", "principal": "oidc~…"}` | | `outcome` | String | Result of the operation. Component-specific; see individual operation docs below | | `context` | Object | Optional. Operation-specific metadata (e.g., `provider_id`, `role_count`) | **Outcomes are not binary allow/deny** — they describe the result of the system operation. No `decision` field is present. **LDAP role resolution (`operation = "ldap_resolve_roles"`):** | `outcome` | When emitted | |------------------|-----------------------------------------------------------| | `success` | User found and role list resolved (possibly empty after mapping) | | `user_not_found` | No LDAP entry matched the search filter for this subject | | `no_roles` | User entry exists but the group-membership attribute is absent | | `ambiguous_user` *(since 0.12.2)* | LDAP search matched more than one entry for the subject; request errors out | | `dn_no_match` *(since 0.12.2)* | `Branching` mode with `else.mode = none`: the user DN did not match `branch_if_user_dn_matches`; empty role list returned | *Since 0.12.2*, every `ldap_resolve_roles` context carries a `mode` field describing which resolution path was active. Possible values: | `mode` | Meaning | |-------------------------|---------------------------------------------------------------------------------------------------------| | `search` | Stand-alone Search-mode resolution | | `attribute` | Stand-alone Attribute-mode resolution | | `branching` | Branching mode, but no branch decision was reached (e.g. `user_not_found`) | | `branch_then` | Branching mode `then` branch ran (DN matched the regex) | | `branch_else_attribute` | Branching mode `else.mode = attribute` ran (DN did not match) | | `branch_else_none` | Branching mode `else.mode = none` (only emitted alongside `outcome = "dn_no_match"`) | **PII in context fields.** `filter` (substituted with the user's subject), `user_dn`, and `principal` are PII. `provider_id`, `attribute`, `pattern`, `role_count`, `count`, and `mode` are not. **Examples:**
Roles resolved successfully ```json { "timestamp": "2026-03-05T09:12:34.000000Z", "level": "INFO", "event_source": "audit", "operation": "ldap_resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~j791840@corp.example.com" }, "outcome": "success", "context": { "provider_id": "my-ldap", "role_count": 3, "mode": "search" }, "message": "LDAP role resolution complete", "target": "lakekeeper_role_provider::role_provider::ldap" } ```
User not found in LDAP ```json { "timestamp": "2026-03-05T09:12:34.000000Z", "level": "INFO", "event_source": "audit", "operation": "ldap_resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~unknown@corp.example.com" }, "outcome": "user_not_found", "context": { "provider_id": "my-ldap", "filter": "(&(objectClass=person)(uid=unknown))", "mode": "search" }, "message": "LDAP user not found; returning empty role list", "target": "lakekeeper_role_provider::role_provider::ldap" } ```
Ambiguous user (multiple matches) ```json { "timestamp": "2026-03-05T09:12:34.000000Z", "level": "INFO", "event_source": "audit", "operation": "ldap_resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~alice@corp.example.com" }, "outcome": "ambiguous_user", "context": { "provider_id": "my-ldap", "filter": "(&(objectClass=person)(uid=alice))", "count": 2, "mode": "attribute" }, "message": "LDAP search matched multiple entries; cannot resolve principal unambiguously", "target": "lakekeeper_role_provider::role_provider::ldap" } ```
Branching mode: user DN did not match (else.mode = none) ```json { "timestamp": "2026-03-05T09:12:34.000000Z", "level": "INFO", "event_source": "audit", "operation": "ldap_resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~svc-account@corp.example.com" }, "outcome": "dn_no_match", "context": { "provider_id": "my-ldap", "user_dn": "CN=svc-account,OU=Services,DC=corp,DC=example,DC=com", "pattern": "OU=(?[^,]+),OU=Tenants,", "mode": "branch_else_none" }, "message": "branching DN regex did not match; explicit no-roles outcome", "target": "lakekeeper_role_provider::role_provider::ldap" } ```
**Role resolution (`operation = "resolve_roles"`):** | `outcome` | When emitted | |--------------------------|---------------------------------------------------| | `no_provider_applicable` | No configured role provider matched this user. | | `roles_resolved` | At least one role was resolved. Disabled by default — enable with `LAKEKEEPER__ROLE_PROVIDER_CHAIN__LOG_ROLE_ASSIGNMENTS=true`. The `context` contains `role_count`, the full `roles` list, and `sources` showing where each provider's roles came from (`fresh`, `cache_hit`, `stale_fallback`, or `in_request`). | | `error` | A matched provider failed to resolve roles (e.g. LDAP connection error). The request proceeds with an empty role set. | The `no_provider_applicable` outcome is enabled by default and can be controlled via `LAKEKEEPER__ROLE_PROVIDER_CHAIN__LOG_UNHANDLED_USERS`. A `no_provider_applicable` outcome for a user that you expect to be covered indicates a misconfigured domain filter or a missing provider. Set the variable to `false` to suppress these events if some users are intentionally not covered. The `roles_resolved` outcome is **disabled by default** because it fires on every authenticated request and contains the full list of resolved role names. Enable it temporarily to debug role-provider configuration — do not leave it on in production. The `error` outcome always fires when role resolution fails. It is accompanied by a general application warning in the non-audit log stream (without PII).
No provider applicable ```json { "timestamp": "2026-03-07T10:00:00.000000Z", "level": "INFO", "event_source": "audit", "operation": "resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~unknown@other-domain.com" }, "outcome": "no_provider_applicable", "context": { "providers_checked": ["ldap-prod"] }, "message": "No role provider handled user; user will have no provider-assigned roles" } ```
Roles resolved (debug) ```json { "timestamp": "2026-03-07T10:00:01.000000Z", "level": "INFO", "event_source": "audit", "operation": "resolve_roles", "actor": { "actor_type": "principal", "principal": "oidc~alice@corp.example.com" }, "outcome": "roles_resolved", "context": { "role_count": 2, "roles": ["my-ldap~devs", "my-ldap~admins"], "sources": {"my-ldap": "cache_hit", "oidc": "in_request"} }, "message": "Resolved role assignments for user" } ```
**Role assignment cache (`operation = "cached_role_provider"`):** | `outcome` | When emitted | |-----------------------|-------------------------------------------------------------------------| | `stale_cache_fallback` | One or more providers failed to refresh; stale DB-cached roles are returned instead. The `context.provider_ids` field lists the affected providers. | This outcome is always accompanied by a WARN-level general log (without PII) and indicates a transient connectivity issue with the role provider (e.g. LDAP unavailable). The user receives their last-known roles rather than an error.
Stale cache fallback ```json { "timestamp": "2026-03-07T11:30:00.000000Z", "level": "INFO", "event_source": "audit", "operation": "cached_role_provider", "actor": { "actor_type": "principal", "principal": "oidc~user@corp.example.com" }, "outcome": "stale_cache_fallback", "context": { "provider_ids": ["ldap-prod"] }, "message": "stale provider(s) failed to refresh; serving cached roles" } ```
**jq filters for operational audit events:** ```bash # All LDAP resolution events cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .operation == "ldap_resolve_roles")' # Users not found in LDAP (misconfigured filter or unknown principals) cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .outcome == "user_not_found")' # Successful resolutions for a specific user cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .operation == "ldap_resolve_roles" and .actor.principal == "oidc~user@example.com")' # Users not matched by any role provider cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .outcome == "no_provider_applicable")' # Stale cache fallbacks (role provider unreachable, last-known roles served) cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .outcome == "stale_cache_fallback")' ``` ### 2. Error Response Logs HTTP error responses returned to clients. **Does not contain PII.** **Identified by:** `"event_source": "error_response"` **Structure:** | Field | Type | Description | |----------------|--------|----------------------------------------------------| | `event_source` | String | Always `"error_response"` | | `error` | Object | Contains `type`, `code`, `message`, `error_id`, `stack`, `source` | **Note:** Empty arrays are omitted. If `stack` or `source` are empty, they will not appear in the log. **Example:** ```json { "timestamp": "2026-02-15T14:22:15.456789Z", "level": "ERROR", "event_source": "error_response", "error": { "type": "TableNotFound", "code": 404, "message": "Table 'my_table' not found in namespace 'production'", "error_id": "01234567-89ab-cdef-0123-456789abcdef", "stack": ["Additional context here"], "source": ["Caused by: ..."] }, "message": "Internal server error response", "target": "iceberg_ext::catalog::rest::error" } ``` **Note:** For 5xx errors, the `stack` and `source` fields are logged but hidden from the HTTP response body for security. ### 3. General Application Logs Standard operational and debug logs from Lakekeeper. No `event_source` field. **Example:** ```json { "timestamp": "2026-02-15T14:20:42.425131Z", "level": "INFO", "message": "Authorization model for version 4.3 found in OpenFGA store lakekeeper. Model ID: 01KHGMK6TQKN1AVMWX16E37AD1", "target": "openfga_client::migration" } ``` ## Additional Configuration ### Extended Debug Logs Include source file locations and line numbers in logs: ```bash LAKEKEEPER__DEBUG__EXTENDED_LOGS=true ``` This is useful for debugging but increases log size. ## Filtering Logs Use `jq` to filter structured JSON logs. Lakekeeper outputs non-JSON content during startup (ASCII art banner, version info), so standard `jq` will fail. Use `jq -R 'fromjson?'` to handle mixed output: - `-R` reads each line as raw text instead of expecting JSON - `fromjson?` attempts to parse each line as JSON, silently skipping non-JSON lines (the `?` suppresses errors) ```bash # Only audit logs cat logs.json | jq -R 'fromjson? | select(.event_source == "audit")' # Failed authorizations cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .decision == "denied")' # Error responses cat logs.json | jq -R 'fromjson? | select(.event_source == "error_response")' # Specific user activity cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .actor.principal == "oidc~user@example.com")' # Specific table access cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and .entity.table == "my_table")' # Any individual denied decision (single-check OR a denied entry inside a batch event) cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and any((.authorizations // [])[]; .allowed == false))' # Permissions checked on behalf of a specific user (introspection / batch-check) cat logs.json | jq -R 'fromjson? | select(.event_source == "audit" and any((.authorizations // [])[]; .["for-principal"].user == "oidc~cfb55bf6-fcbb-4a1e-bfec-30c6649b52f8"))' ``` ## Best Practices 1. **Separate Audit Logs**: Route logs with `event_source=audit` to a secure, long-term storage system for compliance. 2. **PII Handling**: Audit logs contain user identities. Apply appropriate access controls and retention policies. 3. **Error IDs**: Every error has a unique `error_id`. Use this to correlate client-side errors with server logs. 4. **Log Aggregation**: In production, use a centralized logging system (ELK, Loki, Splunk) to collect and analyze logs from all Lakekeeper instances. 5. **Alerts**: Set up alerts for: - Multiple `decision=denied` events from the same principal - High rates of `event_source=error_response` with 5xx codes - Access to sensitive resources outside business hours ## Related Topics - [Authentication](./authentication.md) - Configure identity providers - [Authorization](./authorization.md) - Set up permission management - [Configuration](./configuration.md) - Complete configuration reference --- # Monitoring Lakekeeper Source: https://docs.lakekeeper.io/docs/latest/monitoring/ # Monitoring Lakekeeper Lakekeeper exposes Prometheus metrics and per-project endpoint statistics. We recommend integrating these into your Kubernetes/Grafana/Prometheus stack. ## Key Metrics ### HTTP Request Metrics Three metrics cover all HTTP traffic: | Metric | Labels | Description | |--------------------------------------------------------------------------|--------------------------------------|-----| | axum_http_requests_total | `method`, `status`, `endpoint` | Request count broken down by HTTP method, status code, and endpoint path | | axum_http_requests_pending | `method`, `endpoint` | Requests currently in-flight per endpoint and method | | axum_http_requests_duration_seconds | `method`, `status`, `endpoint`, `le` | Response time histogram; use the `le=1` bucket as a baseline health indicator | !!! tip "Interpreting HTTP request metrics" Visualize `axum_http_requests_total` by status code for overall API health. Rising 4XX rates indicate client-side issues; rising 5XX rates indicate server or database problems requiring urgent attention. High `axum_http_requests_pending` counts signal backend bottlenecks — consider scaling Lakekeeper horizontally. For latency, monitor the `le=1` bucket of `axum_http_requests_duration_seconds` as a baseline; spikes typically point to Postgres or upstream service issues. ### Tokio Metrics Lakekeeper emits all default [Tokio Runtime Metrics](https://github.com/tokio-rs/tokio-metrics?tab=readme-ov-file#runtime-metrics), including "unstable" metrics. A detailed description of these metrics, including how they are derived, can be found in the [tokio_metrics crate documentation](https://docs.rs/tokio-metrics/latest/tokio_metrics/struct.RuntimeMetrics.html#fields). ### Cache Metrics Lakekeeper maintains in-memory caches for Short-Term Credentials, Warehouses, Namespaces, Secrets, Roles, User Assignments, and Role Members. All caches share three metric names, differentiated by the `cache_type` label: | Metric | Type | Labels | Description | |--------------------------------------------------------------------|---------|--------------|-----| | lakekeeper_cache_size | Gauge | `cache_type` | Current number of entries in the cache | | lakekeeper_cache_hits_total | Counter | `cache_type` | Total cache hits | | lakekeeper_cache_misses_total | Counter | `cache_type` | Total cache misses | `cache_type` values: `stc`, `warehouse`, `namespace`, `secrets`, `role`, `user_assignments`, `role_members`. A persistently low hit rate signals the cache capacity should be increased. See [Configuration > Caching](./configuration.md#caching) for details. Role-membership cache invalidation emits one additional metric: | Metric | Type | Labels | Description | |---------------------------------------------------------------------------------------|-----------|-------------|-----| | lakekeeper_role_membership_edge_fanout_users | Histogram | `operation` | Users whose cached role assignments were invalidated by a single role-to-role membership edge change (`operation`: `add` / `remove`) | The user-assignments cache stores a fully-expanded transitive closure, so one role-membership edge change can invalidate many users at once. A high p99 means a single edit fans out widely; Lakekeeper also logs a `warn` when one change invalidates more than 1000 users. ### Role Provider Metrics When a Role Provider (e.g. LDAP) is configured, Lakekeeper emits the following metrics, each labelled by `provider_id`: | Metric | Type | Labels | Description | |----------------------------------------------------------------------------------------------------|-----------|--------------------------|-----| | lakekeeper_role_provider_up | Gauge | `provider_id` | `1` when the provider is reachable, `0` when unreachable. Updated by the periodic health-check loop. Emitted only for providers with an external backend (e.g. LDAP); the OIDC token provider has no external dependency and reports no series. | | lakekeeper_role_provider_get_roles_duration_seconds | Histogram | `provider_id`, `outcome` | Duration of each role-lookup call. The `outcome` label reflects how the request was served (see table below). Emitted by external-backed providers (LDAP). | | lakekeeper_role_provider_sync_errors_total | Counter | `provider_id` | Number of failures writing fresh roles back to the Postgres catalog cache. Emitted by LDAP providers and by the OIDC token provider when `persist_token_roles` is enabled. | | lakekeeper_role_provider_ldap_reconnects_total | Counter | `provider_id`, `outcome` | LDAP reconnect attempts (LDAP providers only), labelled `success` or `error`. | **`outcome` values for `lakekeeper_role_provider_get_roles_duration_seconds`** (histogram label): | Value | Meaning | |-------------------------------|----------------------------------------------| | `cache_hit` | All applicable providers were fresh; the external provider was not contacted. | | `success` | Fresh roles were fetched from the external provider and synced to Postgres. | | `stale_fallback` | The external provider was unreachable, but previously cached roles from Postgres were served instead. Authorization continues to work. | | `error` | Unrecoverable error — the provider failed and no cached roles were available. | **Health probe behavior.** Role provider health is intentionally *excluded* from the `/health` endpoint. The periodic health-check loop still calls `update_health` on every cycle (to drive reconnection attempts and keep `lakekeeper_role_provider_up` current), but an unreachable provider does **not** cause the pod to fail its liveness or readiness probe. Lakekeeper continues serving the roles it last synced to Postgres (`stale_fallback`), so authorization keeps working during a provider outage — at the cost of potentially stale group memberships. This contrasts with the Postgres connection: if Postgres becomes unreachable, the pod **will** fail its health check (see [Database Monitoring](#database-postgres-monitoring) below). `/health` returns `200 OK` only when the aggregate health state is `ok`; it returns `503 Service Unavailable` when the aggregate state is `error` or `unknown`. !!! tip "Alerting on role provider health" Alert on `lakekeeper_role_provider_up == 0 or absent(lakekeeper_role_provider_up{provider_id=""})` to detect provider outages early. The `== 0` clause alone misses a provider that never reported — the series exists only for external-backed providers (LDAP) and only after the first health-check cycle, so pin the `absent()` clause to the `provider_id`s you expect. A sustained `stale_fallback` rate in `lakekeeper_role_provider_get_roles_duration_seconds` confirms that Lakekeeper is actively falling back to cached roles. Rising `lakekeeper_role_provider_sync_errors_total` indicates failures writing roles back to Postgres — for an LDAP provider a database connectivity/permissions problem; for the OIDC token provider (`persist_token_roles`) a failure persisting token roles for definer-view reuse. ## Prometheus Integration Lakekeeper listens on `LAKEKEEPER__BIND_IP:LAKEKEEPER__METRICS__PORT` (defaults: `0.0.0.0:9000`). The bind address `0.0.0.0` means "listen on all interfaces" — it is not a valid scrape target. Configure Prometheus to scrape a reachable address such as `http://localhost:9000/metrics` or `http://:9000/metrics`. | Variable | Description | |---------------------------------------------------------------|--------------| | LAKEKEEPER__METRICS__PORT | Port Lakekeeper listens on for the metrics endpoint (default `9000`) | | LAKEKEEPER__BIND_IP | Listener bind address for metrics, REST API, and Management API (default `0.0.0.0`; use a specific IP to restrict access) | ```yaml title="Example Prometheus scrape configuration" scrape_configs: - job_name: "lakekeeper" static_configs: - targets: ["lakekeeper-host:9000"] ``` ## Database (Postgres) Monitoring Postgres is Lakekeeper's primary backend. Use [postgres_exporter](https://github.com/prometheus-community/postgres_exporter) for database-internal signals — kube-state-metrics covers Kubernetes API object state (pods, deployments, nodes) but not Postgres internals. | Signal | Recommended tool | |---------------------------------------|--------------------------------------| | Free connection pool slots | `postgres_exporter` | | Connection failures / pool exhaustion | `postgres_exporter` | | Query latency | `postgres_exporter` | | Replication lag | `postgres_exporter` | | Disk usage and IOPS | Cloud provider metrics or `node_exporter` | | Pod restarts, deployment health | kube-state-metrics | If you run Postgres via the [CloudNativePG](https://cloudnative-pg.io/) operator, its built-in per-instance exporter (port `9187`, metrics prefixed `cnpg_collector_*`) covers WAL file counts and size, archive status, sync replica state, and basic liveness — complementing `postgres_exporter` for those signals. Connection pool slots, query latency, and replication lag are available as [user-defined custom queries](https://cloudnative-pg.io/documentation/current/monitoring/#user-defined-metrics) in CloudNativePG; disk and IOPS still require `node_exporter` or cloud provider metrics. ### Connection Pool (client-side) `postgres_exporter` reports the Postgres *server's* connection slots. Lakekeeper additionally exposes its own *client-side* pools — the separate read and write pools each replica holds, sized by `LAKEKEEPER__PG_READ_POOL_CONNECTIONS` and `LAKEKEEPER__PG_WRITE_POOL_CONNECTIONS`. A client pool can saturate even when the server has free slots, so monitor both. The read and write pools are reported separately via the `pool` label. | Metric | Type | Labels | Description | |---------------------------------------------------------------------------------|---------|---------------------------------------------------|-----| | lakekeeper_catalog_pg_pool_connections | Gauge | `pool` (`read`/`write`), `state` (`in_use`/`idle`) | Live connections held by the pool | | lakekeeper_catalog_pg_pool_max_connections | Gauge | `pool` | Configured pool ceiling | | lakekeeper_catalog_pg_pool_acquire_timeouts_total | Counter | `pool` | Connection acquisitions that timed out — direct evidence of pool exhaustion | !!! tip "Alerting on pool saturation" Utilization `lakekeeper_catalog_pg_pool_connections{state="in_use"} / lakekeeper_catalog_pg_pool_max_connections` approaching `1` is the leading edge of exhaustion. Any nonzero rate on `lakekeeper_catalog_pg_pool_acquire_timeouts_total` means requests are already being delayed or failing — alert on it. The gauges are sampled every 15s, so brief spikes may be smoothed; the timeout counter captures every occurrence. The counter covers transaction acquisition (the path catalog reads and writes use), not ad-hoc direct-pool queries. !!! warning Lakekeeper's `/health` endpoint checks the database connection. If Postgres becomes unreachable or runs out of connections, `/health` returns `503 Service Unavailable`, so standard Kubernetes HTTP probes fail and the pod is marked unhealthy or unready. Use `/health` for readiness probes when traffic should only be sent to pods with a healthy Postgres connection. It is also suitable as a liveness probe if your deployment wants Kubernetes to restart pods whose database-dependent health remains unhealthy. ```yaml title="Example Kubernetes probes" livenessProbe: httpGet: path: /health port: 8181 readinessProbe: httpGet: path: /health port: 8181 ``` ## Kubernetes and Resource Monitoring Monitor pod CPU, memory, and restart counts with kube-state-metrics or equivalent tooling. ## Endpoint Statistics Lakekeeper aggregates per-request statistics in memory and flushes them to the database periodically (default every 30 s). Each record captures the HTTP method, endpoint path, response status code, project, and warehouse (where applicable). This data is stored internally by Lakekeeper and is accessible without a Prometheus setup. These statistics can be viewed in the UI under the Project View's **Statistics** tab. The Management API also exposes them directly: - `POST /management/v1/endpoint-statistics` — query endpoint-level usage data, filterable by warehouse, status code, and time window. - `GET /management/v1/warehouse/{warehouse_id}/statistics` — query warehouse-level table and view counts. For real-time traffic visibility, the [HTTP request metrics](#http-request-metrics) expose per-second counters and latency histograms via Prometheus — but only with `method`, `status`, and `endpoint` labels. They carry no project or warehouse dimensions, so they cannot be used for tenant-scoped analysis. Endpoint statistics are the only source of per-project and per-warehouse breakdowns, making them the right tool for chargeback, abuse detection, and per-customer analytics in multi-tenant deployments. The flush interval is controlled by `LAKEKEEPER__ENDPOINT_STAT_FLUSH_INTERVAL` (supports `s` and `ms` units): ```env LAKEKEEPER__ENDPOINT_STAT_FLUSH_INTERVAL=60s ``` See [Configuration - Endpoint Statistics](./configuration.md#endpoint-statistics) for details. ## Best Practices Split Grafana dashboards by concern: API health (status codes, pending, latency), database health, cache hit/miss ratios, role provider health, and Kubernetes resource utilization. Alert on sustained 5XX/4XX spikes, high pending request counts, low cache hit rates, and `lakekeeper_role_provider_up == 0` (combined with `absent(...)` to catch a provider that never reported). ## Troubleshooting If Grafana shows stale or missing metrics, verify that Prometheus can reach the metrics endpoint and that the bind IP and port match your scrape configuration. For historical analysis beyond Prometheus retention, query endpoint statistics from the database. --- # Open Policy Agent (OPA) Source: https://docs.lakekeeper.io/docs/latest/opa/ # Open Policy Agent (OPA) [Lakekeeper's Open Policy Agent bridge](https://github.com/lakekeeper/lakekeeper/tree/main/authz/opa-bridge) enables compute engines that support fine-grained access control via Open Policy Agent (OPA) as authorization engine to respect privileges in Lakekeeper. We have also prepared a self-contained [Docker Compose Example](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control-advanced) to get started quickly. Let's imagine we have a trusted multi-user query engine such as trino, in addition to single-user query engines like pyiceberg or daft in Jupyter Notebooks. Managing permissions in trino independently of the other tools is not an option, as we do not want to duplicate permissions across query engines. Our multi-user query engine has two options: 1. **Catalog enforces permissions**: The engine contacts the Catalog on behalf of the user. To achieve this, the engine must be able to impersonate the user for the catalog application. In OAuth2 settings, this can be accomplished through downscoping tokens or other forms of Token Exchange. 2. **Compute enforces permissions**: After contacting the catalog with a god-like "I can do everything!" user (e.g. `project_admin`), the query engine then contacts the permission system, retrieves, and enforces those permissions. Note that this requires the engine to run in a trusted environment, as whoever has root access to the engine also has access to the god-like credential. The Lakekeeper OPA Bridge enables solution 2, by exposing all permissions in Lakekeeper via OPA. The Bridge itself is a collection of OPA files in the `authz/opa-bridge` folder of the Lakekeeper GitHub repository. The bridge also comes with a translation layer for trino to translate trino to Lakekeeper permissions and thus serve trinos OPA queries. Currently trino is the only iceberg query engine we are aware of that is flexible enough to honor external permissions via OPA. Please [let us know](https://github.com/lakekeeper/lakekeeper/issues/new/choose) if you are aware of other engines, so that we can add support. ## Configuration Lakekeeper's OPA bridge needs to access the permissions API of Lakekeeper. As such, we need a technical user for OPA (Client ID, Client Secret) that OPA can use to authenticate to Lakekeeper. Please check the [Authentication guide](./authentication.md) for more information on how to create technical users. We recommend to use the same user for creating the catalog in trino to ensure same access. In most scenarios, this user should have the `project_admin` role. The plugin can be customized by either editing the `configuration.rego` file or by setting environment variables. By editing the `configuration.rego` files you can also easily connect multiple lakekeeper instances to the same trino instance. Please find all available configuration options explained in the file. ### Lakekeeper Connection If configuration is done via environment variables, the following settings are available: | Variable | Example | Description | |------------------------------------------|---------------------------------------------------------------------|-----| | `LAKEKEEPER_URL` | `https://lakekeeper.example.com` | URL where lakekeeper is externally reachable. Default: `https://localhost:8181` | | `LAKEKEEPER_TOKEN_ENDPOINT` | `http://keycloak:8080/realms/iceberg/protocol/openid-connect/token` | Token endpoint of the IdP used to secure Lakekeeper. This endpoint is used to exchange OPAs client credentials for an access token. | | `LAKEKEEPER_CLIENT_ID` | `trino` | Client ID used by OPA to access Lakekeeper's permissions API. | | `LAKEKEEPER_CLIENT_SECRET` | `abcd` | Client Secret for the Client ID. | | `LAKEKEEPER_SCOPE` | `lakekeeper` | Scopes to request from the IdP. Defaults to `lakekeeper`. Please check the [Authentication Guide](./authentication.md) for setup. | | `LAKEKEEPER_MAX_BATCH_CHECK_SIZE` | `1000` | Maximum number of checks per `batch-check` HTTP request. Larger values mean fewer HTTP calls but more load on the authorization backend. Default: `1000` | ### Catalog Mapping All above mentioned configuration options refer to a specific Lakekeeper instance. What is missing is a mapping of trino catalogs to Lakekeeper warehouses. By default we support 4 catalogs in trino, but more can easily be added in the `configuration.rego`. | Variable | Example | Description | |------------------------------------------------|---------------------------|-----| | `TRINO_DEV_CATALOG_NAME` | `dev` | Name of the development catalog in trino. Default: `dev` | | `LAKEKEEPER_DEV_WAREHOUSE` | `development` | Name of the development warehouse in lakekeeper that corresponds to the `TRINO_DEV_CATALOG_NAME` catalog in trino. Default: `development` | | `TRINO_PROD_CATALOG_NAME` | `prod` | Name of the production catalog in trino. Default: `prod` | | `LAKEKEEPER_PROD_WAREHOUSE` | `production` | Name of the production warehouse in lakekeeper that corresponds to the `TRINO_PROD_CATALOG_NAME` catalog in trino. Default: `production` | | `TRINO_DEMO_CATALOG_NAME` | `demo` | Name of the demo catalog in trino. Default: `demo` | | `LAKEKEEPER_DEMO_WAREHOUSE` | `demo` | Name of the demo warehouse in lakekeeper that corresponds to the `TRINO_DEMO_CATALOG_NAME` catalog in trino. Default: `demo` | | `TRINO_LAKEKEEPER_CATALOG_NAME` | `lakekeeper` | Name of the lakekeeper catalog in trino. Default: `lakekeeper` | | `LAKEKEEPER_LAKEKEEPER_WAREHOUSE` | `lakekeeper` | Name of the lakekeeper warehouse in lakekeeper that corresponds to the `TRINO_LAKEKEEPER_CATALOG_NAME` catalog in trino. Default: `lakekeeper` | ### Unmanaged Catalogs | Variable | Example | Description | |------------------------------------------------|---------|-----| | `TRINO_ALLOW_UNMANAGED_CATALOGS` | `true` | Blanket-allow access to all catalogs not listed in the `trino_catalog` array. When trino has multiple authorizers configured, ALL authorizers must allow an action for it to succeed. If trino uses catalogs managed by other authorizers (e.g. a connected PostgreSQL catalog), set this to `true` so the OPA bridge does not block access to those catalogs. Default: `false`. For fine-grained control over unmanaged catalogs, use the `allow_unmanaged` extension point instead (see below). | ### Admin Users Admin users get full access to Trino system schemas and tables across all catalogs (including `system.metadata`, `system.runtime`, etc.) and can view queries owned by any user (`FilterViewQueryOwnedBy`, `ViewQueryOwnedBy`). Non-admin users can only view their own queries. Note that this only affects Trino-level authorization — access to data in Lakekeeper-managed catalogs is still governed by Lakekeeper's own authorization. | Variable | Example | Description | |-------------------------------------------|----------------------------------|-----| | `TRINO_ADMIN_USERS` | `user-id-1,user-id-2` | Comma-separated list of Trino user IDs (typically OIDC subject identifiers) that receive admin access. Default: empty (no admins). | Admin users can also be configured directly in the `trino_admin_users` list in `configuration.rego`. ### Trusted Engines and View Security If you use Lakekeeper's [DEFINER view security model](./view-security.md), Lakekeeper protects the engine's owner property (e.g. `trino.run-as-owner`) so that only trusted engines may set or remove it. Both the **Trino catalog client** and the **OPA bridge client** must be matched as trusted engines — otherwise `CREATE VIEW ... SECURITY DEFINER` and related operations will be rejected with `403 ProtectedPropertyModification`. We recommend giving the OPA bridge its **own** Keycloak client, separate from the Trino catalog client. The OPA bridge only needs permission-check access on Lakekeeper, which is a narrower privilege scope than what the Trino catalog client needs — keeping them separate limits blast radius if either credential leaks. Register both clients as identities of the same trusted engine. Tokens don't usually carry a distinguishing audience in this case, so list each client's service-account subject under `SUBJECTS`: ```bash LAKEKEEPER__TRUSTED_ENGINES__TRINO__TYPE=trino LAKEKEEPER__TRUSTED_ENGINES__TRINO__OWNER_PROPERTY=trino.run-as-owner LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__OIDC__SUBJECTS=["",""] ``` Alternatively, if your IdP mints a distinguishing audience for each client, use `AUDIENCES` instead. See [View Security](./view-security.md) for the full configuration reference. ### Trino Configuration When OPA is running and configured, set the following configurations for trino in `access-control.properties`: ```yaml access-control.name=opa opa.policy.uri=http:///v1/data/trino/allow opa.log-requests=true opa.log-responses=true opa.policy.batched-uri=http:///v1/data/trino/batch ``` ## System Schema Handling The OPA bridge distinguishes between user-created schemas (namespaces) and system schemas. User schemas are authorized via Lakekeeper's permission system, while system schemas are handled locally by the bridge. ### Trino `system` Catalog The following schemas in the trino `system` catalog are accessible to all authenticated users: | Schema | Allowed Tables | Description | |--------|----------------|-------------| | `jdbc` | all | Required by JDBC clients for metadata discovery. | | `information_schema` | `columns`, `schemata`, `tables`, `views` | Standard SQL metadata tables. | | `metadata` | `analyze_properties`, `catalogs`, `column_properties`, `materialized_views`, `schema_properties`, `table_comments`, `table_properties` | Catalog metadata. Tables like `*_authorization` are excluded for non-admins. | | `runtime` | `queries` | Query monitoring. Non-admins can only see their own queries. | Admin users have unrestricted access to all tables in all system schemas. ### Lakekeeper Catalog System Schemas Within Lakekeeper-managed catalogs, the following schemas are treated as system schemas and require only catalog-level (`get_config`) access instead of namespace-level permissions: | Schema | Allowed Tables | Description | |--------|----------------|-------------| | `information_schema` | `columns`, `schemata`, `tables`, `views` | Standard SQL metadata. | | `schema_discovery` | `discovery`, `shallow_discovery` | Schema discovery for UI tools. | | `system` | `iceberg_tables` | Iceberg table metadata. | User-created schemas are authorized through Lakekeeper's permission system as usual. ## Extension Points The OPA bridge provides two extension points for adding custom authorization rules without modifying the built-in policies. Both default to `false` and can be extended by creating `.rego` files in the `policies/trino/` directory. ### `allow_managed` — Managed Catalog Extensions Use `allow_managed` for additional rules on catalogs listed in `trino_catalog`. These rules run alongside Lakekeeper's permission checks (OR logic). For example, granting UDF management permissions on a Lakekeeper-managed catalog. To use it, create `policies/trino/allow_managed.rego` in the `trino` package and define `allow_managed` rules for the Trino operations you need. These rules bypass Lakekeeper's permission system, so review them carefully. ### `allow_unmanaged` — Unmanaged Catalog Extensions Use `allow_unmanaged` for catalogs **not** listed in `trino_catalog` (e.g., external database catalogs like PostgreSQL or Exasol). These rules are evaluated on a fast path that never triggers Lakekeeper HTTP calls, including in batch operations. For a simple blanket allow, set `TRINO_ALLOW_UNMANAGED_CATALOGS=true` (see above). For fine-grained control, create `policies/trino/allow_unmanaged.rego` in the `trino` package. For example, to grant read-only access to a specific external catalog for certain users: ```rego package trino allow_unmanaged if { input.action.operation in ["AccessCatalog", "FilterCatalogs", "ShowSchemas", "FilterSchemas", "SelectFromColumns", "FilterTables", "FilterColumns"] input.action.resource.catalog.name == "my_external_catalog" } ``` ## Batch Optimization For Trino filter operations (`FilterTables`, `FilterColumns`, `SelectFromColumns`, `FilterSchemas`), the OPA bridge optimizes authorization checks on Lakekeeper-managed catalogs by batching resource checks into Lakekeeper `batch-check` HTTP requests, instead of making one HTTP call per resource. For table/column/select operations, each resource generates two checks (table + view) since Trino does not distinguish between tables and views in filter requests. A resource is allowed if either check passes. Schema filter operations generate one namespace check per resource. When the number of checks exceeds the configured batch size, the bridge automatically splits them into multiple `batch-check` requests (chunking) and concatenates the results. The batch size can be configured per Lakekeeper instance via the `LAKEKEEPER_MAX_BATCH_CHECK_SIZE` environment variable or the `max_batch_check_size` field in `configuration.rego` (default: 1000, matching Lakekeeper's server limit). When using OpenFGA as the authorization backend, this value should be tuned to match the OpenFGA and Lakekeeper batch check settings to avoid overloading the authorization backend. System schemas (`information_schema`, `schema_discovery`, `system`) and metadata tables (e.g., `foo$snapshots`) within managed catalogs are excluded from this batch optimization: they are not included in the `batch-check` request and are instead evaluated per-resource by the OPA policies. These per-resource evaluations may still trigger Lakekeeper calls (for example, catalog-level `get_config` checks), but they do not participate in the batched table/schema authorization request. This optimization is transparent — it produces the same results as per-resource evaluation but with significantly fewer HTTP round-trips to Lakekeeper for non-system resources. ## Context Forwarding The OPA bridge forwards resource names to Lakekeeper's batch-check API for create actions. This enables Lakekeeper's authorizer (e.g. Cedar) to make authorization decisions based on the name of the resource being created: | Trino Operation | Lakekeeper Action | Name Forwarded | |-----------------|-------------------|----------------| | `CreateSchema` (top-level) | `create_namespace` (warehouse) | Schema name | | `CreateSchema` (nested) | `create_namespace` (parent namespace) | Child schema name | | `CreateTable` | `create_table` (namespace) | Table name | | `CreateView` / `CreateMaterializedView` | `create_view` (namespace) | View name | Properties specified during creation (e.g. `WITH (format='PARQUET')`) are also forwarded. A full self-contained example is [available on GitHub](https://github.com/lakekeeper/lakekeeper/tree/main/examples/access-control-advanced). --- # Production Checklist Source: https://docs.lakekeeper.io/docs/latest/production/ # Production Checklist Lakekeeper is the heart of your data platform and needs to integrate deeply with your existing infrastructure such as IdPs. The easiest way to get Lakekeeper to production is our enterprise support. Please find more information on our commercial offerings at [lakekeeper.io](https://lakekeeper.io) Please find following some general recommendations for productive setups: * Use an external high-available database as a catalog backend. We recommend using a managed service in your preferred Cloud or host a high available cluster on Kubernetes yourself using your preferred operator. We are using the amazing [CloudNativePG](https://cloudnative-pg.io) internally. Make sure the Database is backed-up regularly. * Ensure sure both `LAKEKEEPER__PG_DATABASE_URL_READ` and `LAKEKEEPER__PG_DATABASE_URL_WRITE` are set for ideal load distribution. Most postgres deployments specify separate URLs for reading and writing to channel writes to the master while distributing reads across replicas. * For medium or large deployments, ensure the `LAKEKEEPER__PG_READ_POOL_CONNECTIONS` and `LAKEKEEPER__PG_WRITE_POOL_CONNECTIONS` are set to a higher value to allow Lakekeeper to use more connections to the database. * For high-available setups, ensure that multiple Lakekeeper instances are running on different nodes. We recommend our [helm chart](https://github.com/lakekeeper/lakekeeper-charts/tree/main/charts/lakekeeper) for production deployments. * Ensure that Authentication is enabled, typically by setting `LAKEKEEPER__OPENID_PROVIDER_URI` and / or `LAKEKEEPER__ENABLE_KUBERNETES_AUTHENTICATION`. Check our [Authentication Guide](./authentication.md) for more information. * If `LAKEKEEPER__OPENID_PROVIDER_URI` is set, we recommend to set `LAKEKEEPER__OPENID_AUDIENCE` as well. * If Authorization is desired, follow our [Authorization Guide](./authorization.md). Ensure that OpenFGA is hosted in close proximity to Lakekeeper - ideally on the same VM or Kubernetes node. In our Helm-Chart we use `PodAffinity` to achieve this. * When using OpenFGA, make sure that Caching is enabled. Check the [OpenFGA in Production](./authorization-openfga.md#openfga-in-production) section for more information. * If the default Postgres secret backend is used, ensure that `LAKEKEEPER__PG_ENCRYPTION_KEY` is set to a long random string. * Ensure that all Warehouses use distinct storage locations / prefixes and distinct credentials that only grant access to the prefix used for a Warehouse. * Ensure that SSL / TLS is enabled. Lakekeeper does not terminate connections natively. Please use a reverse proxy like Nginx or Envoy to secure the connection to Lakekeeper. On Kubernetes, any Ingress controller can be used. For high-availability, failover should be handled by the reverse proxy. Lakekeeper exposes a database-dependent `/health` endpoint that returns `200 OK` when healthy and `503 Service Unavailable` when unhealthy or unknown. Use it for Kubernetes readiness probes, and for liveness probes when pods should restart after sustained database-dependent health failures. If you are using our helm-chart, probes are already built-in. * When using our helm-chart with the default postgres secret store, we recommend to set `secretBackend.postgres.encryptionKeySecret` to use a pre-created secret to reduce the risk of overwriting the secret created by the helm-chart. * If a trusted query engine, such as a centrally managed trino, uses Lakekeeper's OPA bridge, ensure that no users have root access to trino or OPA as those contain credentials to Lakekeeper with very high permissions. * Specify the `LAKEKEEPER__OPENID_SUBJECT_CLAIM` configuration value if `LAKEKEEPER__OPENID_PROVIDER_URI` is set. Accepts a single claim name or a comma-separated list (e.g. `oid,sub`); the first claim present in the token is used. By default Lakekeeper tries `oid` first, then falls back to `sub`. We strongly recommend setting this configuration explicitly in production deployments. Entra-ID users want to use `oid`; users from all other IdPs most likely want to use `sub`. * Create regular Backups of your Lakekeeper database (Postgres) and OpenFGA (if used). Test your backup and restore process regularly. Always backup the Lakekeeper database before upgrading Lakekeeper or OpenFGA. * Set up monitoring and observability. See the [Monitoring Guide](./monitoring.md) for details. --- # Storage Source: https://docs.lakekeeper.io/docs/latest/storage/ # Storage Storage in Lakekeeper is bound to a Warehouse. Each Warehouse stores data in a location defined by a `StorageProfile` attached to it. Currently, we support the following storages: - S3 (tested with AWS & Minio) - Azure Data Lake Storage Gen 2 - OneLake (Microsoft Fabric) - Google Cloud Storage (with and without Hierarchical Namespaces) When creating a Warehouse or updating storage information, Lakekeeper validates the configuration. By default, Lakekeeper Warehouses enforce specific URI schemas for tables and views to ensure compatibility with most query engines: - **S3 / AWS Warehouses**: Must start with `s3://` - **Azure / ADLS Warehouses**: Must start with `abfss://` - **GCP Warehouses**: Must start with `gs://` When a new table is created without an explicitly specified location, Lakekeeper automatically assigns the appropriate protocol based on the storage type. If a location is explicitly provided by the client, it must adhere to the required schema. ## Disabling Credential Vending & Remote Signing Lakekeeper provides multiple ways to control how credentials and remote signing information are provided to clients. You can disable credential vending and remote signing on a per-warehouse basis using storage profile settings. For S3 warehouses, set `remote-signing-enabled` to `false` to disable remote signing and `sts-enabled` to `false` to disable STS vended credentials. For Azure ADLS warehouses, set `sas-enabled` to `false` to disable SAS token generation. For GCS warehouses, set `sts-enabled` to `false` to disable STS token generation. When these options are disabled at the storage profile level, clients will not receive the corresponding credentials or signing information for that warehouse, regardless of the request headers. Lakekeeper downscopes vended credentials for all supported storages to the location of the table being accessed and ensures that there are no overlapping table locations within a warehouse. Clients can also control credential delegation per request using the `X-Iceberg-Access-Delegation` header. Lakekeeper supports the standard Iceberg REST spec values (`vended-credentials` and `remote-signing`), plus a special `client-managed` value. When set to `client-managed`, no credentials or signing information are returned, regardless of storage profile configuration. This allows clients to use their own credentials for direct storage access. ## Allowing Alternative Protocols (s3a, s3n, wasbs) For S3 / AWS and Azure / ADLS Warehouses, Lakekeeper optionally supports additional protocols. To enable these, activate the "Allow Alternative Protocols" flag in the storage profile of the Warehouse. When enabled, the following additional protocols are accepted for table creation or registration: - **S3 / AWS Warehouses**: Supports `s3a://` and `s3n://` in addition to `s3://` - **Azure Warehouses**: Supports `wasbs://` in addition to `abfss://` ## Storage Layout The storage layout controls how namespace and tabular directories are structured under the warehouse base location. It is configured via the `storage-layout` field inside the `storage-profile` when creating or updating a warehouse. The layout applies to all new tabulars created in the warehouse; existing tabular locations are not changed. ### Layout Types | Type | JSON `"type"` value | Description | |----------------------------|--------------------------------|----------------| | Default | `"default"` | Flat: no namespace directories; all tabulars are placed directly under the base location with a `{uuid}` segment. Used when `storage-layout` is omitted. **Changed in 0.13** — see [Default](#default). | | Full hierarchy | `"full-hierarchy"` | One directory per namespace level in the full ancestry, one for the tabular. | | Tabular-only (flat) | `"tabular-only"` | No namespace directories; all tabulars are placed directly under the base location. | !!! note "OneLake supports only the default layout" The [OneLake](#onelake-microsoft-fabric) storage profile currently rejects `tabular-only` and `full-hierarchy` at warehouse-creation time because OneLake silently percent-decodes `%XX` in blob paths, which would alias `{name}` segments that differ only by URL-encoding. See the [OneLake storage-layout note](#onelake-microsoft-fabric) for details. ### Default The default layout is **flat**: tabulars are placed directly under the warehouse base location with no namespace directories, using a `{uuid}` segment. For a tabular `orders` in any namespace the path is: ```text / ``` With the default `{uuid}` template this looks like: ```text s3://my-bucket/warehouse// ``` To use the default layout explicitly: ```json { "storage-profile": { "type": "s3", "storage-layout": { "type": "default" } } } ``` !!! warning "The default layout changed in 0.13" Before 0.13, the default layout emitted a directory for the **direct parent namespace** (`//`). As of 0.13 the default is flat (`/`). The change is **not retroactive** — storage paths are assigned once, at creation time, and are never recomputed: - **Existing tabulars** keep their current locations. - **Existing namespaces** keep their persisted `location` property, so new tabulars created in them remain nested under the old `/` directory (see [Namespace Location Property](#namespace-location-property)). - Only **namespaces created on or after 0.13** use the flat default. A warehouse spanning the upgrade can therefore hold a mix of nested (pre-0.13 namespaces) and flat (new namespaces) tabular paths. If you want namespace directories in new tabular paths, set `storage-layout` to [`full-hierarchy`](#full-hierarchy) — note that this nests *every* ancestor level, whereas the pre-0.13 default nested only the direct parent. ### Full Hierarchy The full hierarchy layout creates one path segment for **every namespace level** in the ancestry, followed by the tabular segment. For a tabular `orders` in namespace `europe` / `production` the path is: ```text /// ``` With `{name}-{uuid}` templates: ```text s3://my-bucket/warehouse/europe-/production-/orders-/ ``` Configuration: ```json { "storage-profile": { "type": "s3", "storage-layout": { "type": "full-hierarchy", "namespace": "{name}-{uuid}", "tabular": "{name}-{uuid}" } } } ``` ### Tabular-Only (Flat) The flat layout places all tabulars directly under the warehouse base location with no namespace directories. For a tabular `orders` in any namespace the path is: ```text / ``` !!! note The `tabular` template **must** contain `{uuid}` in the flat layout. Without it, tabulars with the same name in different namespaces would map to the same storage path, causing data corruption. Configuration: ```json { "storage-profile": { "type": "s3", "storage-layout": { "type": "tabular-only", "tabular": "{name}-{uuid}" } } } ``` ### Template Placeholders Namespace and tabular templates support two placeholders: | Placeholder | Description | |-------------|----------------------------------------------------------------| | `{uuid}` | UUID of the namespace or tabular, inserted without any encoding. | | `{name}` | Name of the namespace or tabular, URL percent-encoded. For example, `my tabular` becomes `my%20tabular` and `中文` becomes `%E4%B8%AD%E6%96%87`. | Both placeholders can be combined with each other and with literal text, e.g. `{name}-{uuid}`. When `storage-layout` is omitted from the storage profile, the `default` layout is used. !!! warning "Always include `{uuid}` in templates" **We strongly recommend including `{uuid}` in every namespace and tabular template.** Storage paths are assigned once at creation time and are never updated when a tabular or namespace is renamed. If a template relies solely on `{name}` and a tabular or namespace is later renamed and re-created with the same name, Lakekeeper will reject the creation because the path is already in use by the old (now-renamed) object. Because UUIDs are unique and stable for the lifetime of an object, using `{uuid}` (alone or combined with `{name}`) guarantees that each object always has a distinct, collision-free storage path. This is the reason Lakekeeper defaults to pure `{uuid}` templates. ### Namespace Location Property Namespaces have a `location` property that determines where their tabulars are stored: - **With location property**: New tabulars always use the namespace's persisted location, regardless of storage layout changes. - **Without location property**: These namespaces compute locations from the current storage layout. Layout changes affect new tabular placement. ## S3 We support remote signing and vended-credentials with S3-compatible storages & AWS. Both provide a secure way to access data on S3: - **Remote Signing**: The client prepares an S3 request and sends its headers to the sign endpoint of Lakekeeper. Lakekeeper checks if the request is allowed, if so, it signs the request with its own credentials, creating additional headers during the process. These additional signing headers are returned to the client, which then contacts S3 directly to perform the operation on files. - **Vended Credentials**: Lakekeeper uses the "STS" Endpoint of S3 to generate temporary credentials which are then returned to clients. Remote signing works natively with all S3 storages that support the default `AWS Signature Version 4`. This includes almost all S3 solutions on the market today, including Rook Ceph Rados, NetApp StorageGRID 12.0 or newer, Minio and others. Vended credentials in turn depend on an additional "STS" Endpoint, that is not supported by all S3 implementations. We run our integration tests for vended credentials against Minio and AWS. We recommend to setup vended credentials for all supported stores, remote signing is not supported by all clients. When a client requests table configuration, Lakekeeper selects between remote signing and vended credentials based on the `X-Iceberg-Access-Delegation` header and storage profile settings: - If the header is set to `client-managed`, neither credentials nor signing information are returned - If the header specifies `vended-credentials` or `remote-signing`, that method is used if enabled in the storage profile - If both methods are requested or neither is specified, Lakekeeper attempts to provide vended credentials first (if STS is enabled), then falls back to remote signing (if enabled) - If both methods are disabled at the storage profile level, no credentials are returned regardless of the header value For maximum client compatibility, we recommend enabling both STS and remote signing when your S3 storage supports it. For some older remote signing clients that cannot handle table-specific remote signing endpoint locations, Lakekeeper needs to identifying a table by its location in the storage. Since there are multiple canonical ways to specify S3 resources (virtual-host & path), Lakekeeper warehouses by default use a heuristic to determine which style is used. For some setups these heuristics may not work, or you may want to enforce a specific style. In this case, you can set the `remote-signing-url-style` field to either `path` or `virtual-host` in your storage profile. `path` will always use the first path segment as the bucket name. `virtual-host` will use the first subdomain if it is followed by `.s3` or `.s3-`. The default mode is `auto` which first tries `virtual-host` and falls back to `path` if it fails. ### Configuration Parameters The following table describes all configuration parameters for an S3 storage profile: | Parameter | Type | Required | Default | Description | |-------------------------------|---------|----------|----------------------------|-----| | `bucket` | String | Yes | - | Name of the S3 bucket. Must be between 3-63 characters, containing only lowercase letters, numbers, dots, and hyphens. Must begin and end with a letter or number. | | `region` | String | Yes | - | AWS region where the bucket is located. For S3-compatible storage, any string can be used (e.g., "local-01"). | | `sts-enabled` | Boolean | Yes | - | Whether to enable STS for vended credentials. Not all S3 compatible object stores support "AssumeRole" via STS. We strongly recommend to enable sts if the storage system supports it. | | `remote-signing-enabled` | Boolean | No | `true` | Whether to enable remote signing for S3 requests. When disabled, clients cannot use remote signing for this storage profile even if STS is disabled. Defaults to `true`. | | `key-prefix` | String | No | None | Subpath in the bucket to use for this warehouse. | | `endpoint` | URL | No | None | Optional endpoint URL for S3 requests. If not provided, the region will be used to determine the endpoint. If both are provided, the endpoint takes precedence. Example: `http://s3-de.my-domain.com:9000` | | `sts-endpoint` | URL | No | Value of `endpoint` | Optional separate endpoint URL for STS requests. Use this when your S3-compatible storage exposes STS on a different endpoint than S3. If not provided, the S3 `endpoint` is used for STS requests as well. | | `flavor` | String | No | `aws` | S3 flavor to use. Options: `aws` (Amazon S3) or `s3-compat` (for S3-compatible solutions like MinIO). | | `path-style-access` | Boolean | No | `false` | Whether to use path style access for S3 requests. If the underlying S3 supports both virtual host and path styles, we recommend not setting this option. | | `assume-role-arn` | String | No | None | Optional ARN to assume when accessing the bucket from Lakekeeper. This is also used as the default for `sts-role-arn` if that is not specified. | | `sts-role-arn` | String | No | Value of `assume-role-arn` | Optional role ARN to assume for STS vended-credentials. Either `assume-role-arn` or `sts-role-arn` must be provided if `sts-enabled` is true and `flavor` is `aws`. | | `sts-token-validity-seconds` | Integer | No | `3600` | The validity period of STS tokens in seconds. Controls how long the vended credentials remain valid before they need to be refreshed. | | `sts-session-tags` | Object | No | `{}` | An optional JSON object containing key-value pairs of session tags to apply when assuming roles via STS. These tags are attached to the temporary credentials and can be used for access control, auditing, or cost allocation. Each key and value must be a string. Example: `{"Environment": "production", "Team": "data-engineering"}` | | `allow-alternative-protocols` | Boolean | No | `false` | Whether to allow `s3a://` and `s3n://` in locations. This is disabled by default and should only be enabled for migrating legacy Hadoop-based tables via the register endpoint. Tables with `s3a` paths are not accessible outside the Java ecosystem. | | `remote-signing-url-style` | String | No | `auto` | S3 URL style detection mode for remote signing. Options: `auto`, `path-style`, or `virtual-host`. When set to `auto`, Lakekeeper tries virtual-host style first, then path style. | | `push-s3-delete-disabled` | Boolean | No | `true` | Controls whether the `s3.delete-enabled=false` flag is sent to clients. Only has an effect if "soft-deletion" is enabled for this Warehouse. This prevents clients like Spark from directly deleting files during operations like `DROP TABLE xxx PURGE`, ensuring soft-deletion works properly. However, it also affects operations like `expire_snapshots` that require file deletion. For more information, please check the [Soft Deletion Documentation](./concepts.md#soft-deletion). | | `aws-kms-key-arn` | String | No | None | ARN of the AWS KMS Key that is used to encrypt the bucket. Vended Credentials is granted `kms:Decrypt` and `kms:GenerateDataKey` on the key. | | `legacy-md5-behavior` | Boolean | No | `false` | A flag to enable the legacy behavior of using MD5 checksums for operations that require checksums. | | `storage-layout` | Object | No | `{"type": "default"}` | Controls how namespace and tabular directories are structured under the warehouse base location. See [Storage Layout](#storage-layout) for details. | ### AWS ###### Direct File-Access with Access Key First create a new S3 bucket for the warehouse. Buckets can be re-used for multiple Warehouses as long as the `key-prefix` is different. We recommend to block all public access. Secondly we need to create an AWS role that can access and delegate access to the bucket. We start by creating a new Policy that allows access to data in the bucket. We call this policy `LakekeeperWarehouseDev`: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "ListBuckets", "Action": [ "s3:ListAllMyBuckets", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::*" ] }, { "Sid": "ListBucketContent", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": "arn:aws:s3:::lakekeeper-aws-demo" }, { "Sid": "DataAccess", "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::lakekeeper-aws-demo/*" ] } ] } ``` Now create a new user, we call the user `LakekeeperWarehouseDev`, and attach the previously created policy. When the user is created, click on "Security credentials" and "Create access key". Note down the access key and secret key for later use. We are done if we only rely on remote signing. For vended credentials, we need to perform one more step. Create a new role that we call `LakekeeperWarehouseDevRole`. This role needs to be trusted by the user, which is achieved via with the following trust policy: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "TrustLakekeeperWarehouseDev", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/LakekeeperWarehouseDev" }, "Action": "sts:AssumeRole" } ] } ``` Also attach the `LakekeeperWarehouseDev` policy created earlier. We are now ready to create the Warehouse via the UI or REST-API using the following values (make sure to replace everything in `<>`): ```json { "warehouse-name": "aws_docs", "storage-credential": { "type": "s3", "aws-access-key-id": "", "aws-secret-access-key": "", "credential-type": "access-key" }, "storage-profile": { "type": "s3", "bucket": "", "region": "", "sts-enabled": true, "flavor": "aws", "key-prefix": "lakekeeper-dev-warehouse", "sts-role-arn": "arn:aws:iam:::role/LakekeeperWarehouseDevRole" }, "delete-profile": { "type": "hard" } } ``` As part of the `storage-profile`, the field `assume-role-arn` can optionally be specified. If it is specified, this role is assumed for every IO Operation of Lakekeeper. It is also used as `sts-role-arn`, unless `sts-role-arn` is specified explicitly. If no `assume-role-arn` is specified, whatever authentication method / user os configured via the `storage-credential` is used directly for IO Operations, so needs to have S3 access policies attached directly (as shown in the example above). ##### System Identities / Managed Identities Since Lakekeeper version 0.8, credentials for S3 access can also be loaded directly from the environment. Lakekeeper integrates with the AWS SDK to support standard environment-based authentication, including all common configuration options through AWS_* environment variables. !!! note When using system identities, we **strongly recommend** configuring external-id values. This prevents unauthorized cross-account role access and ensures roles can only be assumed by authorized Lakekeeper warehouses. Without external IDs, any user with warehouse creation permissions in Lakekeeper could potentially access any role the system identity is allowed to assume. For more information, see [AWS's documentation on external IDs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html). Below is a step-by-step guide for setting up a secure system identity configuration: Firstly, create a dedicated AWS user to serve as your system identity. Do not attach any direct permissions or trust policies to this user. This user will only have the ability to assume specific roles with the proper external ID Secondly, configure Lakekeeper with this identity by setting the following environment variables. ```bash AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_DEFAULT_REGION=... # Required for System Credentials to work: LAKEKEEPER__S3_REQUIRE_EXTERNAL_ID_FOR_SYSTEM_CREDENTIALS=true ``` In addition to the standard `AWS_*` environment variables, Lakekeeper supports all authentication methods available in the AWS SDK, including instance profiles, container credentials, and SSO configurations. For enhanced security, Lakekeeper enforces that warehouses using system identities must specify both an `external-id` and an `assume-role-arn` when configured. This implementation follows AWS security best practices by preventing unauthorized role assumption. These default requirements can be adjusted through settings described in the [Configuration Guide](./configuration.md#storage). For this example, assume the system identity has the ARN `arn:aws:iam::123:user/lakekeeper-system-identity`. When creating a warehouse, users must configure an IAM role with an appropriate trust policy. The following trust policy template enables the Lakekeeper system identity to assume the role, while enforcing external ID validation: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123:user/lakekeeper-system-identity" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` The role also needs S3 access, so attach a policy like this: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAllAccessInWarehouseFolder", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3::://*" ], "Effect": "Allow" }, { "Sid": "AllowRootAndHomeListing", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] } ] } ``` We are now ready to create the Warehouse using the system identity: ```json { "warehouse-name": "aws_docs_managed_identity", "storage-credential": { "type": "s3", "credential-type": "aws-system-identity", "external-id": "" }, "storage-profile": { "type": "s3", "assume-role-arn": "", "bucket": "", "region": "", "sts-enabled": true, "flavor": "aws", "key-prefix": "" }, "delete-profile": { "type": "hard" } } ``` The specified `assume-role-arn` is used for Lakekeeper's reads and writes of the object store. It is also used as a default for `sts-role-arn`, which is the role that is assumed when generating vended credentials for clients (with an attached policy for the accessed table). ##### CORS Configuration For browser-based access to S3 buckets (required for [DuckDB WASM](engines.md#duckdb-wasm)), you need to configure CORS (Cross-Origin Resource Sharing) on your S3 bucket. To configure CORS for your S3 bucket: 1. In the AWS S3 Configuration Menu, click on the name of your bucket 2. Choose **Permissions** Tab 3. In the **Cross-origin resource sharing (CORS)** section, choose **Edit** 4. In the CORS configuration editor text box, type or copy and paste a new CORS configuration, or edit an existing configuration. The CORS configuration is a JSON file. The text that you type in the editor must be valid JSON. See below for an example. 5. Choose **Save changes** Example CORS policy: ```json [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "POST", "PUT", "DELETE", "HEAD" ], "AllowedOrigins": [ "https://lakekeeper.example.com" ], "ExposeHeaders": [ "ETag", "x-amz-version-id" ] } ] ``` Replace `https://lakekeeper.example.com` with the origin where your Lakekeeper instance is hosted. ##### STS Session Tags The optional `sts-session-tags` setting can be used to provide Session Tags when assuming roles via STS. Doing so requires that the IAM Role's Trust Relationship also allow `sts:TagSession`. Here's the above example with this addition: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAssumeRole", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123:user/lakekeeper-system-identity" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Sid": "AllowSessionTagging", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123:user/lakekeeper-system-identity" }, "Action": "sts:TagSession" } ] } ``` If wanting to use a session tag in an ABAC policy, one can reference that tag via `${aws:PrincipalTag/}`. For example, here's a policy that dynamically sets the S3 path based on a `tenant` tag: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAllAccessInTenantWarehouse", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::/${aws:PrincipalTag/tenant}/*" ], "Effect": "Allow" }, { "Sid": "AllowListingInTenantWarehouse", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": [ "${aws:PrincipalTag/tenant}/*" ] } } } ] } ``` ### S3 Compatible Unlike for AWS, we do not need any special trust-setup for vended credentials / STS with most S3 compatible solutions like Minio. Instead, we just need a bucket and an access key / secret key combination that is able to read and write from it. If `sts-role-arn` is provided, it will be sent as part of the request to the STS service. Keep in mind that the specific S3 compatible solution may ignore the parameter. Conversely, if `sts-role-arn` is not specified, the request to the STS service will not contain it. Make sure to select `flavor` to have the value `s3-compat`! This setting should work for most self-hosted S3 solutions. An warehouse create call could look like this: ```json { "warehouse-name": "minio_dev", "storage-credential": { "type": "s3", "aws-access-key-id": "", "aws-secret-access-key": "", "credential-type": "access-key" }, "storage-profile": { "type": "s3", "bucket": "", "region": "local-01", "sts-enabled": true, "flavor": "s3-compat", "key-prefix": "lakekeeper-dev-warehouse", }, "delete-profile": { "type": "hard" } } ``` ### Cloudflare R2 Lakekeeper supports Cloudflare R2 storage with all S3 compatible clients, including vended credentials via the `/accounts/{account_id}/r2/temp-access-credentials` Endpoint. First we create a new Bucket. In the cloudflare UI, Select "R2 Object Storage" -> "Overview" and select "+ Create Bucket". We call our bucket `lakekeeper-dev`. Click on the bucket, select the "Settings" tab, and note down the "S3 API" displayed. Secondly, we create an API Token for Lakekeeper as follows: 1. Go back to the Overview Page ("R2 Object Storage" -> "Overview") and select "Manage API tokens" in the "{} API" dropdown. 1. In the R2 token page select "Create Account API token". Give the token any name. Select the "Admin Read & Write" permission, this is unfortunately required at the time of writing, as the `/accounts/{account_id}/r2/temp-access-credentials` does not accept other tokens. Click "Create Account API Token". 1. Note down the "Token value", "Access Key ID" and "Secret Access Key" Finally, we can create the Warehouse in Lakekeeper via the UI or API. A POST request to `/management/v1/warehouse` expects the following body: ```json { "warehouse-name": "r2_dev", "delete-profile": { "type": "hard" }, "storage-credential": { "credential-type": "cloudflare-r2", "account-id": " ", "access-key-id": "access-key-id-from-above", "secret-access-key": "secret-access-key-from-above", "token": "token-from-above", }, "storage-profile": { "type": "s3", "bucket": "", "region": "", "key-prefix": "path/to/my/warehouse", "endpoint": ".eu.r2.cloudflarestorage.com>" }, } ``` For cloudflare R2 credentials, the following parameters are automatically set: - `assume-role-arn` is set to None, as this is not supported - `sts-enabled` is set to `true` - `flavor` is set to `s3-compat` It is required to specify the `endpoint`. Use a [Data Location Hint](https://developers.cloudflare.com/r2/reference/data-location/#available-hints) as region. ## Azure Data Lake Storage Gen 2 To add a Warehouse backed by ADLS, we need two Azure objects: The Storage Account itself and an App Registration which Lakekeeper can use to access it and delegate access to compute engines. ### Configuration Parameters The following table describes all configuration parameters for an ADLS storage profile: | Parameter | Type | Required | Default | Description | |-------------------------------|---------|----------|-------------------------------------|-----| | `account-name` | String | Yes | - | Name of the Azure storage account. | | `filesystem` | String | Yes | - | Name of the ADLS filesystem, in blob storage also known as container. | | `sas-enabled` | Boolean | No | `true` | Whether to enable SAS (Shared Access Signature) token generation for Azure Data Lake Storage. When disabled, clients cannot use vended credentials for this storage profile. Defaults to `true`. | | `key-prefix` | String | No | None | Subpath in the filesystem to use. | | `allow-alternative-protocols` | Boolean | No | `false` | Whether to allow `wasbs://` in locations in addition to `abfss://`. This is disabled by default and should only be enabled for migrating legacy Hadoop-based tables via the register endpoint. | | `host` | String | No | `dfs.core.windows.net` | The host to use for the storage account. | | `authority-host` | URL | No | `https://login.microsoftonline.com` | The authority host to use for authentication. | | `sas-token-validity-seconds` | Integer | No | `3600` | The validity period of the SAS token in seconds. | | `storage-layout` | Object | No | `{"type": "default"}` | Controls how namespace and tabular directories are structured under the warehouse base location. See [Storage Layout](#storage-layout) for details. | Lets start by creating a new "App Registration": 1. Create a new "App Registration" - **Name**: choose any, for this example we choose `Lakekeeper Warehouse (Development)` - **Redirect URI**: Leave empty 2. When the App Registration is created, select "Manage" -> "Certificates & secrets" and create a "New client secret". Note down the secrets "Value". 3. In the "Overview" page of the "App Registration" note down the `Application (client) ID` and the `Directory (tenant) ID`. Next, we create a new Storage Account. Make sure to select "Enable hierarchical namespace" in the "Advanced" section. For existing Storage Accounts make sure "Hierarchical namespace: Enabled" is shown in the "Overview" page. There are no specific requirements otherwise. Note down the name of the storage account. When the storage account is created, we need to grant the correct permissions to the "App Registration" and create the filesystem / container where the data is stored: 1. Open the Storage Account and select "Data storage" -> Containers. Add a new Container, we call it `warehouse-dev`. 2. Next, select "Access Control (IAM)" in the left menu and "Add role assignment". Grant the `Storage Blob Data Contributor` and `Storage Blob Delegator` roles to the `Lakekeeper Warehouse (Development)` App Registration that we previously created. We are now ready to create the Warehouse via the UI or the REST API. Use the following information: - **client-id**: The `Application (client) ID` of the `Lakekeeper Warehouse (Development)` App Registration. - **client-secret**: The "Value" of the client secret that we noted down previously. - **tenant-id**: The `Directory (tenant) ID` from the Applications Overview page. - **account-name**: Name of the Storage Account - **filesystem**: Name of the container (that Azure also calls filesystem) previously created. In our example its `warehouse-dev`. A POST request to `/management/v1/warehouse` would expects the following body: ```json { "warehouse-name": "azure_dev", "delete-profile": { "type": "hard" }, "storage-credential": { "client-id": "...", "client-secret": "...", "credential-type": "client-credentials", "tenant-id": "...", "type": "az", }, "storage-profile": { "account-name": "...", "filesystem": "warehouse-dev", "type": "adls", }, } ``` ##### Azure System Identity !!! warning Enabling Azure system identities allows Lakekeeper to access any storage location that the managed identity has permissions for. To minimize security risks, ensure the managed identity is restricted to only the necessary resources. Additionally, limit Warehouse creation permission in Lakekeeper to users who are authorized to access all locations that the system identity can access. Azure system identities can be used to authenticate Lakekeeper to ADLS Gen 2, without specifying credentials explicitly on Warehouse creation. This feature is disabled by default and must be explicitly enabled system-wide by setting the following environment variable: ```bash LAKEKEEPER__ENABLE_AZURE_SYSTEM_CREDENTIALS=true ``` When enabled, Lakekeeper will use the managed identity of the virtual machine or application it is running on to access ADLS. Ensure that the managed identity has the necessary permissions to access the storage account and container. For example, assign the `Storage Blob Data Contributor` and `Storage Blob Delegator` roles to the managed identity for the relevant storage account as described above. ## OneLake (Microsoft Fabric) *Available since Lakekeeper 0.12.4.* Microsoft Fabric exposes its OneLake data lake through ADLS Gen2-compatible APIs, so Lakekeeper can back warehouses directly with a Fabric lakehouse using a dedicated `onelake` storage profile. The OneLake profile is a convenience layer over the generic ADLS profile: instead of asking you to encode the OneLake URL conventions yourself (`account_name = "onelake"`, container = workspace ID, key prefix = `/Files/`), it derives those from the workspace and lakehouse UUIDs you provide. It also knows how to compute the workspace-scoped private-link endpoint host. !!! note The generic `adls` profile also works against OneLake if you set the fields manually (`account-name: "onelake"`, `host: "dfs.fabric.microsoft.com"`, `filesystem: `, `key-prefix: /Files/`). The `onelake` profile is the recommended path because it validates the OneLake-specific constraints (SAS lifetime cap, supported credentials, endpoint shapes) for you and computes private-link FQDNs automatically. ### Client compatibility OneLake's blob surface is API-compatible with regular ADLS Gen2 for the operations Lakekeeper's vended-credentials path uses, but client libraries need to be OneLake-aware (specifically: they have to honour `adls.account-host` so they target `*.fabric.microsoft.com` instead of defaulting to `.blob.core.windows.net`). Lakekeeper emits the right property in the catalog response; the engine still has to be on a version that consumes it. | Client | Minimum version | Notes | |---|---|---| | **PyIceberg** | `0.10.0` | First release that ships both [`adls.account-host`](https://github.com/apache/iceberg-python/pull/2016) and [`adls.credential`](https://github.com/apache/iceberg-python/pull/2299). Earlier versions don't construct OneLake URLs correctly even when Lakekeeper hands them the right properties. Transitively requires `adlfs >= 2024.7.0`. | | **Spark + Iceberg (Java)** | `iceberg-spark-runtime` ≥ `1.5` on Spark 3.5 *or* ≥ `1.10` on Spark 4 | The Java Iceberg ADLS file IO parses the host from `abfss://@/...` directly, so it's transparently OneLake-compatible for any version that supports vended ADLS credentials. | Lakekeeper's own OneLake integration tests are run against: - **Spark** `4.0.2` (`apache/spark:4.0.2-scala2.13-java21-python3-ubuntu`) with `iceberg-spark-runtime` `1.10.1` - **PyIceberg** `0.10.0` (with the `adlfs` extra) Older Spark 3 / Iceberg < 1.10 combinations are exercised by other suites in the same harness (`apache/spark:3.5.6-java17-python3`); the OneLake-specific paths in vended credentials don't depend on the Spark major version. ### Configuration Parameters | Parameter | Type | Required | Default | Description | |--------------------------------|---------|----------|-------------------------------------|-------------| | `workspace-id` | UUID | Yes | - | UUID of the Fabric workspace this warehouse lives in. | | `lakehouse-id` | UUID | Yes | - | UUID of the lakehouse within the workspace. | | `directory-rel-path` | String | Yes | - | Subpath beneath `/` inside the lakehouse — the root directory under which Lakekeeper writes all warehouse data. | | `top-level-folder` | String | No | `"Files"` | Top-level managed folder. Either `"Files"` (recommended, Iceberg-managed data area) or `"Tables"`. Writing Iceberg metadata under `Tables/` conflicts with Fabric's automatic Delta/Iceberg virtualization — choose only with care. | | `endpoint-mode` | Object | No | `{"type": "default"}` | OneLake endpoint selection. See [Endpoint modes](#endpoint-modes) below. | | `sas-enabled` | Boolean | No | `true` | Enable SAS-token vending. Disable to force clients to use their own credentials. | | `sas-token-validity-seconds` | Integer | No | `3600` | SAS-token validity in seconds. **Max: 3600 (OneLake hard cap).** Lakekeeper rejects values above 3600 and below 1; values below 60 are accepted with a warning and floored at mint time. | | `authority-host` | URL | No | `https://login.microsoftonline.com` | Microsoft Entra authority host. | | `storage-layout` | Object | No | `{"type": "default"}` | **Must be `{"type": "default"}` or omitted.** OneLake silently percent-decodes `%XX` sequences in blob paths, so layouts that embed `{name}` segments (`tabular-only`, `full-hierarchy`) would alias to the same blob after server-side decoding. Only the default `{uuid}`-only layout is currently supported. See [Storage Layout](#storage-layout). | ### Endpoint modes OneLake exposes three DFS endpoint shapes; the `endpoint-mode` field picks one. | Type | JSON | Resulting host | |------------------------|---------------------------------------------------------|-------------------------------------------------------------------------| | Default | `{"type": "default"}` | `onelake.dfs.fabric.microsoft.com` | | Regional | `{"type": "regional", "region": "westus"}` | `westus-onelake.dfs.fabric.microsoft.com` | | Workspace private link | `{"type": "workspace-private-link"}` | `.z.dfs.fabric.microsoft.com` (host derived from `workspace-id` automatically; `` is the first two hex characters of the un-dashed workspace UUID) | Use `regional` when data residency requires the request to stay within a specific Azure region. Use `workspace-private-link` when the workspace is fronted by a Fabric workspace-level private endpoint. !!! info "Tenant-level vs workspace-level private link" Fabric supports two distinct private-link scopes, and only one of them needs a dedicated `endpoint-mode`: - **Tenant-level private link**: traffic to the global host `onelake.dfs.fabric.microsoft.com` is routed privately via DNS that points the global FQDN at a tenant-PE NIC. From Lakekeeper's perspective this is indistinguishable from public traffic — use `default`. (Same shape as a private endpoint sitting in front of a regular ADLS Gen2 storage account: the URL Lakekeeper builds doesn't change, only DNS does.) - **Workspace-level private link**: each workspace gets its own `.z.dfs.fabric.microsoft.com` FQDN routed via a workspace-scoped PE. Lakekeeper has to build that FQDN — use `workspace-private-link`. !!! note Even when `endpoint-mode` is set to `workspace-private-link`, the Lakekeeper server itself must retain DNS resolution and outbound TLS connectivity to the global host `onelake.dfs.fabric.microsoft.com`. SAS token minting (the `Get User Delegation Key` call) is not served by the workspace-FQDN private-link endpoint — Fabric returns `DeniedByPolicy` there — so Lakekeeper issues that single call against the global OneLake host. Vended client traffic (read/write of table data) still flows through the workspace private link. !!! warning "OneLake path names cannot contain `%`" OneLake's request pipeline silently collapses any `%XX` percent-escape in a blob path to its decoded character before SAS validation, so a path that stores the *literal* three-character sequence `%3F` is indistinguishable from one that stores the single character `?`. Lakekeeper otherwise treats every byte in a path literally (`%41bc` is a different blob from `Abc`); on OneLake that guarantee can't hold, so the OneLake storage profile **rejects any table location whose path segments contain a literal `%`** at create time. Use a different character or strip the `%` before submitting the location. !!! warning "Only the `default` storage layout is supported" The OneLake profile currently rejects `tabular-only` and `full-hierarchy` storage layouts at warehouse-creation time. These layouts can embed namespace and table **names** into the storage path via `{name}` templates, and Lakekeeper URL-percent-encodes those segments — but OneLake silently decodes `%XX` sequences server-side (see the percent-encoding warning above), so two distinct names whose encoded form differs only by a `%XX` would land at the same blob and overwrite each other. Until we land a layout that side-steps this encoding mismatch, only the default `{uuid}`-only layout is supported. Set `storage-layout` to `{"type": "default"}` or omit it. Since 0.13 the default layout is **flat** — it has no `{name}` segments to percent-decode, so it is OneLake-safe. OneLake warehouses therefore place new tabulars directly under the base location (`/`); see [Default](#default), including the note on how this affects namespaces created before 0.13. ### Credentials OneLake does not have a storage-account key. Only Microsoft Entra credentials are accepted: - `client-credentials` (service principal): the standard option. - `azure-system-identity` (managed identity): if `LAKEKEEPER__ENABLE_AZURE_SYSTEM_CREDENTIALS=true` is set server-wide. Supplying `shared-access-key` to a OneLake warehouse is rejected at validation time. The OneLake tenant setting **"Authenticate with OneLake user-delegated SAS tokens"** must be enabled for the workspace before vended credentials work. This is a Fabric-side setting and cannot be configured from Lakekeeper. ### Example A POST request to `/management/v1/warehouse` to create a OneLake-backed warehouse: ```json { "warehouse-name": "onelake_dev", "delete-profile": { "type": "hard" }, "storage-credential": { "type": "az", "credential-type": "client-credentials", "client-id": "...", "client-secret": "...", "tenant-id": "..." }, "storage-profile": { "type": "onelake", "workspace-id": "0388d6cb-27fd-4dc5-948b-32ab7aab9577", "lakehouse-id": "eb2b7644-2ae4-43ed-ad08-8cc295ffa7ac", "directory-rel-path": "my_warehouse", "endpoint-mode": { "type": "default" } } } ``` This produces the abfss base location: ```text abfss://0388d6cb-27fd-4dc5-948b-32ab7aab9577@onelake.dfs.fabric.microsoft.com/eb2b7644-2ae4-43ed-ad08-8cc295ffa7ac/Files/my_warehouse/ ``` ### Immutability Most fields are immutable on `update-storage-profile` because changing them would orphan every table previously written to the warehouse: `workspace-id`, `lakehouse-id`, `top-level-folder`, `directory-rel-path`, and `endpoint-mode`. The following fields can be updated: `sas-token-validity-seconds`, `sas-enabled`, `authority-host`, `storage-layout`. ## Google Cloud Storage Google Cloud Storage can be used to store Iceberg tables through the `gs://` protocol. ### Configuration Parameters The following table describes all configuration parameters for a GCS storage profile: | Parameter | Type | Required | Default | Description | |------------------|---------|----------|-----------------------|--------------| | `bucket` | String | Yes | - | Name of the GCS bucket. | | `key-prefix` | String | No | None | Subpath in the bucket to use for this warehouse. | | `sts-enabled` | Boolean | No | `true` | Whether to enable STS (Security Token Service) downscoped token generation for GCS. When disabled, clients cannot use vended credentials for this storage profile. Defaults to `true`. | | `storage-layout` | Object | No | `{"type": "default"}` | Controls how namespace and tabular directories are structured under the warehouse base location. See [Storage Layout](#storage-layout) for details. | The service account should have appropriate permissions (such as Storage Admin role) on the bucket. Since Lakekeeper Version 0.8.2, hierarchical Namespaces are supported. ### Authentication Options Lakekeeper supports two primary authentication methods for GCS: ##### Service Account Key You can provide a service account key directly when creating a warehouse. This is the most straightforward way to give Lakekeeper access to your GCS bucket: ```json { "warehouse-name": "gcs_dev", "storage-profile": { "type": "gcs", "bucket": "...", "key-prefix": "..." }, "storage-credential": { "type": "gcs", "credential-type": "service-account-key", "key": { "type": "service_account", "project_id": "example-project-1234", "private_key_id": "....", "private_key": "-----BEGIN PRIVATE KEY-----\n.....\n-----END PRIVATE KEY-----\n", "client_email": "abc@example-project-1234.iam.gserviceaccount.com", "client_id": "123456789012345678901", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abc%example-project-1234.iam.gserviceaccount.com", "universe_domain": "googleapis.com" } } } ``` The service account key should be created in the Google Cloud Console and should have the necessary permissions to access the bucket (typically Storage Admin role on the bucket). ##### GCP System Identity !!! warning Enabling GCP system identities grants Lakekeeper access to any storage location the service account has permissions for. Carefully review and limit the permissions of the service account to avoid unintended access to sensitive resources. Additionally, limit Warehouse creation permissions in Lakekeeper to users who are authorized to access all locations that the system identity can access. GCP system identities allow Lakekeeper to authenticate using the service account that the application is running as. This can be either a Compute Engine default service account or a user-assigned service account. To enable this feature system-wide, set the following environment variable: ```bash LAKEKEEPER__ENABLE_GCP_SYSTEM_CREDENTIALS=true ``` When using system identity, Lakekeeper will use the service account associated with the application or virtual machine to access Google Cloud Storage (GCS). Ensure that the service account has the necessary permissions, such as the Storage Admin role on the target bucket. --- # Table Maintenance Source: https://docs.lakekeeper.io/docs/latest/table-maintenance/ # Table Maintenance ## Metadata File Cleanup Lakekeeper honors the Iceberg table properties `write.metadata.delete-after-commit.enabled` and `write.metadata.previous-versions-max`. Starting with Lakekeeper v0.10.0, `delete-after-commit` is enabled by default (it was disabled in earlier versions). On each table commit, when `delete-after-commit` is enabled, Lakekeeper keeps the current table metadata file plus up to `write.metadata.previous-versions-max` previous metadata files (default: 100) and deletes the oldest tracked metadata file from the metadata log once that limit is exceeded. This cleanup applies only to metadata files tracked in the metadata log; it does not remove orphaned metadata files. For example: if `write.metadata.previous-versions-max=20`, Lakekeeper retains 21 files in total (the current plus 20 previous); committing a 22nd version deletes the oldest tracked metadata file. Link to [Expire Snapshots](#expire-snapshots) ## Expire Snapshots {#expire-snapshots} Lakekeeper automatically expires old table snapshots based on configurable age and retention policies. This helps manage storage costs and performance by removing outdated snapshot metadata and associated data files. Expire snapshots can be configured per warehouse and optionally overridden at the table level using Iceberg table properties. ### Configuration Configuration can be set via the Management UI or REST API endpoints: - **GET** `/management/v1/warehouse/{warehouse_id}/task-queue/expire_snapshots/config` - **POST** `/management/v1/warehouse/{warehouse_id}/task-queue/expire_snapshots/config` | Parameter | Type | Default | Description | |---------------------------|---------|----------------------------------|-----| | `enable-expire-snapshots` | boolean | `false` | Enable automatic snapshot expiration for all tables in the warehouse. Can be overridden per table with `lakekeeper.history.expire.enabled` | | `max-snapshot-age-ms` | integer | `432000000` (5 days) | Maximum age of snapshots in milliseconds before expiration. Override per table with `history.expire.max-snapshot-age-ms` | | `min-snapshots-to-keep` | integer | `1` | Minimum snapshots to retain on each table branch. Override per table with `history.expire.min-snapshots-to-keep` | | `min-snapshots-to-expire` | integer | `20` | Minimum snapshots required before expiration job is scheduled (prevents expensive jobs for few snapshots). Override per table with `lakekeeper.history.expire.min-snapshots-to-expire` | | `max-ref-age-ms` | integer | `9223372036854775807` (no limit) | Maximum age for snapshot references (except main branch). Main branch references never expire | ### Table-Level Overrides Individual tables can override warehouse settings using these Iceberg table properties: - `lakekeeper.history.expire.enabled` - Enable/disable for specific table - `history.expire.max-snapshot-age-ms` - Custom max age for table snapshots - `history.expire.min-snapshots-to-keep` - Custom minimum retention for table - `lakekeeper.history.expire.min-snapshots-to-expire` - Custom threshold for table !!! note Tables with `gc.enabled=false` are excluded from automatic expiration regardless of other settings. !!! note Tables using Iceberg native encryption (`encryption.key-id` set) are skipped. Expiration must read manifest lists and manifests to determine which files to remove, and Lakekeeper cannot decrypt them. Skipped tables are recorded as a successful (skipped) run, not a failure. ### Production Deployment For production workloads, we recommend running expire snapshots workers in dedicated pods to avoid impacting REST API performance. This can be achieved by: 1. **API pods**: Set `LAKEKEEPER__TASK_EXPIRE_SNAPSHOTS_WORKERS=0` to disable workers 2. **Worker pods**: Use default worker configuration (2 workers) to handle expire snapshots tasks or set `LAKEKEEPER__TASK_EXPIRE_SNAPSHOTS_WORKERS` to desired number of workers ### Task Scheduling Expire snapshots tasks are intelligently scheduled immediately after table commits when needed, eliminating the overhead of cron-based polling. This ensures timely cleanup while maintaining optimal performance. ## Remove Orphan Files {#remove-orphan-files} Lakekeeper can detect and remove orphan files — files in a table's storage location that are no longer referenced by any snapshot, manifest, statistics file, or metadata log entry. Orphans typically come from failed writes (optimistic-concurrency conflicts) or incomplete maintenance jobs. The queue is **opt-in per warehouse** and disabled by default. Once enabled, tables are scheduled adaptively: the next run is timed to the observed rate at which orphans accumulate, with a 1-day floor and a configurable ceiling. Idle tables get a periodic safety check at the ceiling; busy tables get reclaimed more often. No cron-based polling is involved. ### How It Works 1. **Scan referenced files**: All files referenced by the current table metadata are collected (manifest lists, manifests, data files, statistics files, partition statistics files, and metadata log entries). 2. **List storage**: The table's storage location is listed recursively. 3. **Identify orphans**: Files present in storage but not in the referenced set are orphan candidates. 4. **Age filter**: Only files older than the configured threshold are deleted. Recently created files are preserved to avoid deleting data from in-progress writes. 5. **Delete**: Orphans are deleted in micro-batches (using cloud-native batch-delete APIs where available). 6. **Reschedule**: The worker self-schedules the next run based on the observed orphan-bytes-per-second rate. ### Configuration Configuration can be set via the Management UI or REST API endpoints: - **GET** `/management/v1/warehouse/{warehouse_id}/task-queue/remove_orphan_files/config` - **POST** `/management/v1/warehouse/{warehouse_id}/task-queue/remove_orphan_files/config` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `enable-remove-orphan-files` | boolean | `false` | Master switch for the warehouse. When `false`, no tables are scheduled unless they set `lakekeeper.remove-orphan-files.enabled=true`. | | `default-older-than-ms` | integer | `604800000` (7 days) | Minimum file age before deletion. **24-hour safety floor enforced at task pickup** (see below). Override per table with `lakekeeper.remove-orphan-files.older-than-ms`. | | `target-reclaim-bytes` | integer | `1073741824` (1 GiB) | Adaptive scheduler target: the next run is timed to reclaim roughly this many bytes based on the last run's observed rate. Clamped to [1 day, `maximum-interval-seconds`]. | | `maximum-interval-seconds` | integer | `7776000` (90 days) | Upper bound on the adaptive next-run interval. Idle tables get a safety check at this cadence. Must be ≥ 86400 (1 day). | | `max-run-time-seconds` | integer | `3600` (1 hour) | Wall-clock timeout for a single task attempt. Must be ≥ 60. | | `dry-run` | boolean | `false` | Identify orphans but do not delete. Useful for previewing what a run would remove. | | `disable-min-older-than-check` | boolean | `false` | Bypass the 24h floor on `default-older-than-ms`. Set only for dev/test setups that genuinely need sub-day retention. | ### Table-Level Overrides Individual tables can override warehouse settings using these Iceberg table properties: - `lakekeeper.remove-orphan-files.enabled` — `true` includes a table even when the warehouse master switch is off; `false` excludes it. - `lakekeeper.remove-orphan-files.older-than-ms` — per-table deletion-age threshold. Not subject to the 24h safety floor; use only for known-stale tables. - `lakekeeper.remove-orphan-files.dry-run` — per-table dry-run. Safer mode wins: if the warehouse is dry-run, the table cannot downgrade to live. ### Safety Mechanisms - **`gc.enabled` check**: Tables with `gc.enabled=false` are excluded. Attempting to remove orphan files on such a table returns an error immediately. - **Encrypted tables skipped**: Tables using Iceberg native encryption (`encryption.key-id` set) are skipped. Identifying the referenced-file set requires reading manifests, which Lakekeeper cannot decrypt; running anyway risks deleting live data. Skipped tables are recorded as a successful (skipped) run; manually scheduling one returns an error immediately. - **24h safety floor on `default-older-than-ms`**: At task pickup, the worker refuses to run with a queue-config retention shorter than 24 hours, returning `RemoveOrphanFilesRetentionTooShort`. A typo like `default-older-than-ms: 6000` (six seconds) would otherwise delete in-flight writer uploads. Mirrors Spark/Iceberg's `retentionDurationCheck.enabled=false` escape — set `disable-min-older-than-check=true` to override. - **Unknown age preservation**: Files where the storage backend does not report a last-modified timestamp are skipped (counted as `skipped_unknown_age_count` in the result). ### Recommended Usage Run remove orphan files **after** expire snapshots. Expiring snapshots first ensures that files from removed snapshots are no longer referenced, so they can be correctly identified as orphans. ### Production Deployment For production workloads, we recommend running remove orphan files workers in dedicated pods, similar to expire snapshots: 1. **API pods**: Set `LAKEKEEPER__TASK_REMOVE_ORPHAN_FILES_WORKERS=0` to disable workers. 2. **Worker pods**: Use the default (2 workers) or set `LAKEKEEPER__TASK_REMOVE_ORPHAN_FILES_WORKERS` to the desired number. !!! warning Every run performs a full recursive listing of the table's storage location, which can be expensive for tables with many files. The adaptive scheduler stretches the cadence on tables that produce few orphans, but each run still pays the listing cost. --- # UI Branding Source: https://docs.lakekeeper.io/docs/latest/ui-branding/ # UI Branding {#ui-branding} Lakekeeper Plus lets you **customize the built-in UI** for a deployment: replace the color scheme and place a partner logo next to the Lakekeeper+ logo — with **no rebuild**. Branding is applied at container start from a single environment variable, so the same image can be re-skinned per customer or per environment. Branding is a **Lakekeeper Plus** feature. The open-source console ships the standard Lakekeeper theme and cannot be re-skinned. ![Custom-branded Lakekeeper UI](../../assets/ui-branding-overview.png) ## How it works You describe the branding as a small JSON document, **base64-encode** it, and pass it to Lakekeeper Plus in the `LAKEKEEPER__UI__BRANDING` environment variable. At serve time Lakekeeper substitutes the value into the UI, which decodes it **synchronously before the app mounts** — so there is no flash of the default theme and no extra network request. ```text branding.json --base64--> LAKEKEEPER__UI__BRANDING --serve--> themed UI ``` Everything in the document is **optional**. Omit a color and the Lakekeeper default is kept; omit the logos and only the colors change. ## The branding document ```json title="branding.json" { "themes": { "light": { "primary": "#EE0000", "primary-darken-1": "#BE0000", "on-primary": "#FFFFFF", "secondary": "#212121", "background": "#FFFFFF", "surface": "#FFFFFF", "error": "#DC3545", "warning": "#FFC107", "success": "#198754", "info": "#1565C0" }, "dark": { "primary": "#EE0000", "on-primary": "#FFFFFF", "secondary": "#D9D9D9", "background": "#212121", "surface": "#212121", "error": "#FF4D5E", "info": "#4AA3DF" } }, "logoLight": "data:image/svg+xml;base64,PHN2Zy4uLg==", "logoDark": "data:image/svg+xml;base64,PHN2Zy4uLg==" } ``` `themes.light` and `themes.dark` are applied on top of the corresponding Lakekeeper base theme, so you only list the tokens you want to override. ### Color tokens | Token | Controls | |-------|----------| | `primary` | The brand color: primary buttons and call-to-action controls (for example **Add Warehouse**), active states, links, highlights. | | `primary-darken-1` | Hover/pressed shade of `primary`. | | `on-primary` | Text/icon color drawn **on** a `primary` surface — set it to whatever stays readable on your primary (usually `#FFFFFF` or `#000000`). | | `secondary` | Secondary buttons and accents. | | `secondary-darken-1` | Hover/pressed shade of `secondary`. | | `background` | Page background. | | `surface` | Cards, dialogs, navigation surfaces. | | `surface-bright`, `surface-light`, `surface-variant` | Elevated / alternate surface shades. | | `on-surface`, `on-surface-variant` | Text/icon color on surfaces. | | `error`, `warning`, `success`, `info` | Semantic status colors (alerts, chips, icons). | !!! tip "`info` is not always blue" Several informational controls use the `info` token. If your palette has no blue, set `info` to a color that fits your brand so those controls don't stand out — for example a red brand can set `info` to its red instead of the default blue. ### Logos Two logos are shown next to the Lakekeeper+ mark (AppBar, login page, and home page). The naming follows the theme they appear **on**: | Field | Shown on | Use | |-------|----------|-----| | `logoDark` | the **light** theme (white bar) | your **dark-colored** logo | | `logoLight` | the **dark** theme (dark bar) | your **light / white** logo | Use **SVG** logos: they stay crisp at every size the UI renders them (AppBar, login, home), are small once base64-encoded, and never blur. Provide each logo as a **`data:` URI** (recommended — self-contained, works in air-gapped deployments) or an `https://` URL. Raster formats (PNG, etc.) are technically accepted via their own MIME type, but are not recommended. For a tight fit next to the Lakekeeper+ logo, use a **horizontal** logo (wordmark) with little surrounding whitespace in its `viewBox`. ## Building the base64 string 1. **Write `branding.json`** with your colors, following the shape above. 2. **Inline each SVG logo as a `data:` URI** and paste it into `logoLight` / `logoDark`: ```bash printf 'data:image/svg+xml;base64,%s' "$(base64 -w0 logo.svg)" ``` On macOS use `base64 -i logo.svg` (no `-w0`). 3. **Base64-encode the whole document** — this is the value for the env var: === "Linux" ```bash base64 -w0 branding.json ``` === "macOS" ```bash base64 -i branding.json ``` 4. **Verify** it decodes back to your JSON: ```bash echo "" | base64 -d | jq . ``` ## Applying the branding Set the resulting string on the Lakekeeper Plus server: ```bash LAKEKEEPER__UI__BRANDING="eyJ0aGVtZXMiOnsibGlnaHQiOns..." ``` === "Docker Compose" ```yaml services: lakekeeper: image: quay.io/lakekeeper/lakekeeper-plus:latest environment: LAKEKEEPER__UI__BRANDING: "eyJ0aGVtZXMiOnsibGlnaHQiOns..." ``` === "Kubernetes" ```yaml env: - name: LAKEKEEPER__UI__BRANDING valueFrom: secretKeyRef: name: lakekeeper-branding key: branding ``` The string can be long (logos are embedded), so storing it in a `Secret` / `ConfigMap` and referencing it — as above — is usually cleaner than inlining it. Restart the server and reload the UI to see the new theme. ![Partner logo in the AppBar and Home page](../../assets/ui-branding-appbar.png) ## Local development When you run the UI with the Vite dev server (instead of the packaged binary), set the same base64 value in `VITE_UI_BRANDING` in your `.env` instead of `LAKEKEEPER__UI__BRANDING`. The dev server inlines env values **at startup**, so restart the dev server after changing it — a browser refresh alone won't pick it up. ## See also - [Configuration -> UI](./configuration.md#ui) — all `LAKEKEEPER__UI__*` variables. - [Logos](/about/logos/) — the official Lakekeeper marks. --- # View Security Source: https://docs.lakekeeper.io/docs/latest/view-security/ # View Security Lakekeeper supports **DEFINER** and **INVOKER** security models for views, enabling catalogs to make context-aware authorization decisions when query engines load tables through view chains. ## Background When a query engine executes a query that references a view, the engine sends a `loadTable` (or `loadView`) request to the catalog with a `referenced-by` query parameter. This parameter contains the chain of views through which the table is being accessed. Lakekeeper uses this chain to decide **which user's permissions** to check at each step. This feature is based on the [Apache Iceberg referenced-by specification](https://github.com/apache/iceberg/pull/13810). ## INVOKER vs DEFINER ### INVOKER (default) With the INVOKER security model, the **calling user's** permissions are checked at every step in the view chain. This is the default behavior — if a view does not have an owner property set, it is treated as INVOKER. **Example:** User Alice queries a view that references a table. ```text Alice --> View (INVOKER) --> Table | | Check: Alice Check: Alice ``` Alice must have permission to access both the view and the underlying table. ### DEFINER With the DEFINER security model, the **view owner's** permissions are used for resources downstream of the DEFINER view. This allows view owners to grant access to underlying data without granting direct table access. **Example:** User Alice queries a DEFINER view owned by Bob that references a table. ```text Alice --> View (DEFINER, owner=Bob) --> Table | | Check: Alice Check: Bob ``` Alice needs permission to access the view, but the **table** is checked against **Bob's** permissions. Alice does not need direct access to the table. ### Chained Views Views can reference other views, creating chains. The security model is evaluated at each step, and DEFINER views switch the "current user" for all subsequent checks. **Example:** A chain with mixed security models. ```text Alice --> View1 (DEFINER, owner=Bob) --> View2 (INVOKER) --> View3 (DEFINER, owner=Carol) --> Table | | | | Check: Alice Check: Bob Check: Bob Check: Carol ``` - **View1** is checked as Alice (the calling user). - **View2** is INVOKER, but we are already in Bob's delegated context (from View1 being DEFINER), so it is checked as Bob. - **View3** is DEFINER with owner Carol — from this point on, Carol's permissions are used. - The **Table** is checked as Carol. ## How It Works When a trusted engine sends a `loadTable` or `loadView` request with the `referenced-by` query parameter, Lakekeeper: 1. Resolves all views and tables in the chain. 2. Determines the security model (DEFINER or INVOKER) for each view by checking the configured owner property (e.g. `trino.run-as-owner`). 3. Walks the chain from entry point to target, switching the "current user" at each DEFINER boundary. 4. Checks permissions for the correct user at each step in a single batch authorization call. 5. Returns the result only if all checks pass. Without a trusted engine, the `referenced-by` parameter is ignored and only the calling user's permissions on the target resource are checked (standard behavior). ## Configuration ### Prerequisites - [Authentication](./authentication.md) must be enabled — Lakekeeper needs token information to identify engines and resolve owners. - An [authorization backend](./authorization.md) must be configured — DEFINER views are only useful when permissions are actually enforced. ### Setting Up Trusted Engines Configure one or more trusted engines so that Lakekeeper knows which query engines to trust. See [Trusted Engines](./configuration.md#trusted-engines) for the full configuration reference. **Minimal example for Trino:** ```bash LAKEKEEPER__TRUSTED_ENGINES__TRINO__TYPE=trino LAKEKEEPER__TRUSTED_ENGINES__TRINO__OWNER_PROPERTY=trino.run-as-owner LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__OIDC__AUDIENCES=[trino] ``` Matching is **scoped to the IdP** the token was issued by. Each engine's `IDENTITIES` block is keyed per IdP (e.g. `IDENTITIES__OIDC__...`), and a token is only tested against the block matching its own IdP. Within that block, a request is matched when **either** its audience appears in `AUDIENCES` **or** its subject appears in `SUBJECTS`. Use `SUBJECTS` when the IdP does not mint a distinguishing audience — for example, to trust a specific service-account's subject UUID directly: ```bash LAKEKEEPER__TRUSTED_ENGINES__TRINO__IDENTITIES__OIDC__SUBJECTS=[""] ``` **What happens when a request is not matched as a trusted engine:** - `loadTable` / `loadView` requests that include a `referenced-by` parameter are **silently ignored** with respect to that parameter — the load still succeeds, but the DEFINER chain is not resolved and permissions are evaluated against the caller only. This is logged at debug level; no error is returned. - Only **commits that actually attempt to set or remove a protected owner property** (`create-view` or `commit-view` writing `trino.run-as-owner`) are rejected with `403 ProtectedPropertyModification`. An ignored `referenced-by` on a load does **not** trigger this error. !!! note "When using the OPA bridge" The [OPA bridge](./opa.md) authenticates to Lakekeeper with its own client credentials to evaluate permission checks. We recommend it runs under a **dedicated** Keycloak client with narrower privileges than the Trino catalog client — not a shared one. If the OPA bridge issues `loadTable` / `loadView` requests with `referenced-by` on behalf of Trino, its client must be matched as a trusted engine so DEFINER chains are resolved rather than ignored. Add its service-account subject under `SUBJECTS` (or its audience under `AUDIENCES` if your IdP mints distinguishing audiences). Permission-check traffic itself is not gated by trusted-engine status — the `ProtectedPropertyModification` rejection only applies to DDL that writes a protected owner property, not to the OPA bridge's routine check calls. ### Creating DEFINER Views Once trusted engines are configured, only matched engines can set the owner property on views. In Trino, DEFINER views are created with: ```sql CREATE VIEW my_schema.my_view SECURITY DEFINER AS SELECT * FROM my_schema.my_table; ``` Trino automatically sets the `trino.run-as-owner` property on the view with the creating user as the owner. !!! warning "Enabling trusted engines with existing views" When you enable trusted engines in an existing deployment, any views that **already** have the owner property set (e.g. `trino.run-as-owner`) will **immediately** be treated as DEFINER views. Lakekeeper will start checking permissions against the owner specified in that property. **Before enabling**, audit your existing views: ```sql -- In Trino, check for views with the owner property SELECT * FROM my_catalog.information_schema.views; ``` Ensure that: 1. The owner values in existing view properties are valid users in your identity provider. 2. Those owners have appropriate permissions on the underlying tables. 3. You have tested the authorization chain in a non-production environment first. If an owner referenced in a view property does not exist or lacks permissions, queries through that view will fail once trusted engines are enabled. ### Property Protection Once a trusted engine is configured, the owner property (e.g. `trino.run-as-owner`) becomes **protected**: only requests from a matched engine can set or remove it. This prevents privilege escalation — without this protection, any user who can commit to a view could set themselves as the DEFINER owner and gain access to tables they shouldn't see. Non-engine requests that attempt to modify a protected property receive a `403 Forbidden` error with type `ProtectedPropertyModification`. Protection is **case-insensitive on the match, case-sensitive on the accepted value**: a property key that differs from the configured owner property only in casing (e.g. `Trino.Run-As-Owner` when the admin configured `trino.run-as-owner`) is also rejected. Most engines read the owner property with fixed casing, so a case variant would silently have no effect on the security model while appearing to set it — only the exact key configured by the admin is accepted. ## Security Considerations ### Delegated Execution When a user accesses a table through a DEFINER view, the table load happens with the **owner's** permissions. This is flagged as "delegated execution" in authorization checks and audit logs. Authorization backends can use this flag to apply different policies — for example, skipping permission-inspection rights that would normally be required. ### Metadata vs. execution: `get_metadata` and `select` Views expose two authorization checkpoints: - `get_metadata` (control-plane) — listing the view, reading its definition, returning it from `loadView`. - `select` (data-plane) — executing the view to produce rows, including traversing a DEFINER chain into the underlying table. `get_metadata` and `select` are **distinct actions** that an authorizer evaluates independently. By convention, a principal who can `select` a view can also `get_metadata` on it (someone allowed to query must also be allowed to inspect); the reverse does not hold. How you grant each is authorizer-specific — see the [OpenFGA](./authorization-openfga.md) or [Cedar](./authorization-cedar.md) docs for the concrete grant names. **When does the split matter in practice?** - **View with no downstream objects** (e.g. `SELECT 1`). No referenced-by chain is built, so `select` is never checked via the chain. `get_metadata` on the view is enough for `loadView` (reading the SQL). `select` is only required if the engine actually executes the view (which it does via the OPA bridge, see below). - **INVOKER view referencing other objects.** The referenced-by chain emits `select` on the view. Downstream objects are checked against the **caller**. The caller needs `select` on the view to traverse the chain plus `read_data` / `write_data` on downstream tables. - **DEFINER view.** Same `select`-on-view requirement for the caller, but downstream objects are checked against the **owner**. `select` on the view is what gates the caller from entering the owner's context without explicit permission. This is also the boundary where the instance-admin `get_metadata` bypass stops short — admins cannot traverse a DEFINER chain they weren't granted `select` on. - **OPA bridge.** When the engine is fronted by the [OPA bridge](./opa.md), the bridge issues a `select` check on the view for every query that returns data (via Trino's `SelectFromColumns`). Under the OPA bridge, `select` on the view is always required to read data. The data-plane / control-plane split lets policies differentiate the two dimensions — for example, the instance-admin bypass applies to `get_metadata` but not to `select`, matching the carve-out that already excludes `read_data` / `write_data` on tables. Operator-style identities can list and manage views they have no explicit access to, but cannot execute them through a referenced-by chain without `select` on that view. Every view in a referenced-by chain is checked for *both* actions. The target of `loadView` is checked only for `get_metadata`. ### Owner Property Integrity The owner property on a view is critical for security. Lakekeeper ensures that: - Only matched trusted engines can **set or remove** the owner property. - The owner string is resolved to a user in the engine's Identity Provider. - If the owner cannot be resolved, the request fails with a clear error. ### Audit Trail All authorization decisions in the referenced-by chain are logged when [audit logging](./configuration.md#audit-logging) is enabled. The audit log includes: - Which user was checked at each step. - Whether the check was a delegated execution. - The full view chain that was evaluated. --- # Getting Started Source: https://docs.lakekeeper.io/getting-started/ # Getting Started There are multiple ways to deploy Lakekeeper. Our [self-contained examples](#option-1-examples) are the easiest way to get started and deploy everything you need (including S3, Query Engines, Jupyter, ...). By default, compute outside of the docker network cannot access the example Warehouses due to docker networking. If you have your own Storage (e.g. S3) available, you can deploy Lakekeeper using [docker compose](#option-2-docker-compose), deploy on [Kubernetes](#option-3-kubernetes), deploy the pre-build [Binary](#option-4-binary) directly or [compile Lakekeeper yourself](#option-5-build-from-sources). Lakekeeper is currently compatible with Postgres >= 15. ## Deployment ### Option 1: 🐳 Examples !!! note Our docker compose examples are not designed to be used with compute outside of the docker network (e.g. external Spark). All docker compose examples come with batteries included (Identity Provider, Storage (S3), Query Engines, Jupyter) but are not accessible (by default) for compute outside of the docker network. To use Lakekeeper with external tools outside of the docker network, please check [Option 2: Docker Compose](#option-2-docker-compose) === "🐳 With Authentication & Authorization - Advanced" The advanced examples contains multiple query engines, including query engines that are shared between users while still enforcing single-user permissions. ```bash git clone https://github.com/lakekeeper/lakekeeper cd lakekeeper/examples/access-control-advanced docker compose up -d ``` === "🐳 With Authentication & Authorization - Simple" The simple access control example contains multiple query engines, each used by a single user. ```bash git clone https://github.com/lakekeeper/lakekeeper cd lakekeeper/examples/access-control-simple docker compose up -d ``` === "🐳 Unsecured" ```bash git clone https://github.com/lakekeeper/lakekeeper cd lakekeeper/examples/minimal docker compose up -d ``` Then open your browser and head to `localhost:8888` to load the example Jupyter notebooks or head to `localhost:8181` for the Lakekeeper UI. ### Option 2: 🐳 Docker Compose For a Docker-Compose deployment that is used with external object storage, and external Identity Providers, you can use the [`docker-compose` Setup](https://github.com/lakekeeper/lakekeeper/tree/main/docker-compose). Please also check the [Examples](#option-1-examples) and our [User Guides](./docs/nightly/configuration.md) for additional information on customization. While you can start the "🐳 Unsecured" variant without any external dependencies, you will need at least an external object store (S3, ADLS, GCS) to create a Warehouse. === "🐳 Unsecured" ```bash git clone https://github.com/lakekeeper/lakekeeper cd lakekeeper/docker-compose docker compose up -d ``` === "🐳 Authentication & Authorization" Please follow the [Authentication Guide](./docs/nightly/authentication.md) to prepare your Identity Provider. Additional environment variables might be required. ```bash git clone https://github.com/lakekeeper/lakekeeper cd lakekeeper/docker-compose export LAKEKEEPER__OPENID_PROVIDER_URI=... (required) export LAKEKEEPER__OPENID_AUDIENCE=... (recommended) export LAKEKEEPER__UI__OPENID_CLIENT_ID=... (required if UI is used) export LAKEKEEPER__UI__OPENID_SCOPE=... (typically required if UI is used) docker compose -f docker-compose.yaml -f openfga-overlay.yaml up ``` ### Option 3: ☸️ Kubernetes We recommend deploying the catalog on Kubernetes using our [Helm Chart](https://github.com/lakekeeper/lakekeeper-charts/tree/main/charts/lakekeeper). Please check the Helm Chart's documentation for possible values. To enable Authentication and Authorization, an external identity provider is required. A community driven [Kubernetes Operator](https://github.com/lakekeeper/lakekeeper-operator) is currently in development. ### Option 4: ⚙️ Binary For single node deployments, you can also download the Binary for your architecture from [GitHub Releases](https://github.com/lakekeeper/lakekeeper/releases). A basic configuration via environment variables would look like this: ```bash export LAKEKEEPER__PG_DATABASE_URL_READ="postgres://postgres_user:postgres_urlencoded_password@hostname:5432/catalog_database" export LAKEKEEPER__PG_DATABASE_URL_WRITE="postgres://postgres_user:postgres_urlencoded_password@hostname:5432/catalog_database" export LAKEKEEPER__PG_ENCRYPTION_KEY="MySecretEncryptionKeyThatIBetterNotLoose" ./lakekeeper migrate ./lakekeeper serve ``` To expose Lakekeeper behind a reverse proxy, most deployments also set: ```bash export LAKEKEEPER__BASE_URI="http://" ``` The default `LAKEKEEPER__BASE_URI` is `http://localhost:8181`. ### Option 5: 👨‍💻 Build from Sources To customize Lakekeeper, for example to connect to your own Authorization system, you might want to build the binary yourself. Please check the [Developer Guide](./docs/nightly/developer-guide.md) for more information. ## First Steps Now that the catalog is up-and-running, the following endpoints are available: 1. `/ui/` - the UI - by default: [http://localhost:8181/ui/](http://localhost:8181/ui/) 1. `/catalog` is the Iceberg REST API 1. `/management` contains the management API 1. `/lakekeeper` is the Data API (e.g. generic, non-Iceberg tables) 1. `/swagger-ui` hosts Swagger to inspect the API specifications ### Bootstrapping Our self-contained docker compose examples are already bootstrapped and require no further actions. After the initial deployment, Lakekeeper needs to be bootstrapped. This can be done via the UI or the bootstrap endpoint. Among others, bootstrapping sets the initial administrator of Lakekeeper and creates the first project. Please find more information on bootstrapping in the [Bootstrap Docs](docs/nightly/bootstrap.md). ### Creating a Warehouse Now that the server is running, we need to create a new warehouse. We recommend to do this via the UI.
![Create a Warehouse](assets/create-warehouse-v1.png){ width="100%" }
Create a Warehouse via UI

Alternatively, we can use the REST-API directly. For an S3 backed warehouse, create a file called `create-warehouse-request.json`: ```json { "warehouse-name": "my-warehouse", "storage-profile": { "type": "s3", "bucket": "my-example-bucket", "key-prefix": "optional/path/in/bucket", "region": "us-east-1", "sts-role-arn": "arn:aws:iam::....:role/....", "sts-enabled": true, "flavor": "aws" }, "storage-credential": { "type": "s3", "credential-type": "access-key", "aws-access-key-id": "...", "aws-secret-access-key": "..." } } ``` We now create a new Warehouse by POSTing the request to the management API: ```sh curl -X POST http://localhost:8181/management/v1/warehouse -H "Content-Type: application/json" -d @create-warehouse-request.json ``` If you want to use a different storage backend, see the [Storage Guide](docs/nightly/storage.md) for example configurations. ### Connect Compute That's it - we can now use the catalog. The example below targets the **unsecured** deployment: it sends the literal token `dummy` because no Authentication is configured, so any bearer token is accepted. If you enabled Authentication, replace `"dummy"` with a real OAuth2 token (or configure your engine's OAuth2 flow) — see the [Authentication Guide](./docs/nightly/authentication.md). !!! tip `ICEBERG_VERSION` below pins a known-good Iceberg release. Check [Iceberg releases](https://iceberg.apache.org/releases/) for the latest version. ```python import pandas as pd import pyspark SPARK_VERSION = pyspark.__version__ SPARK_MINOR_VERSION = '.'.join(SPARK_VERSION.split('.')[:2]) ICEBERG_VERSION = "1.10.1" # if you use adls as storage backend, you need iceberg-azure instead of iceberg-aws-bundle configuration = { "spark.jars.packages": f"org.apache.iceberg:iceberg-spark-runtime-{SPARK_MINOR_VERSION}_2.12:{ICEBERG_VERSION},org.apache.iceberg:iceberg-aws-bundle:{ICEBERG_VERSION},org.apache.iceberg:iceberg-azure-bundle:{ICEBERG_VERSION},org.apache.iceberg:iceberg-gcp-bundle:{ICEBERG_VERSION}", "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.defaultCatalog": "demo", "spark.sql.catalog.demo": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.demo.catalog-impl": "org.apache.iceberg.rest.RESTCatalog", "spark.sql.catalog.demo.uri": "http://localhost:8181/catalog/", "spark.sql.catalog.demo.token": "dummy", # unsecured deployment only; use a real token if Authentication is enabled "spark.sql.catalog.demo.warehouse": "my-warehouse", } spark_conf = pyspark.SparkConf() for k, v in configuration.items(): spark_conf = spark_conf.set(k, v) spark = pyspark.sql.SparkSession.builder.config(conf=spark_conf).getOrCreate() spark.sql("USE demo") spark.sql("CREATE NAMESPACE IF NOT EXISTS my_namespace") print(f"\n\nCurrently the following namespace exist:") print(spark.sql("SHOW NAMESPACES").toPandas()) print("\n\n") sdf = spark.createDataFrame( pd.DataFrame( [[1, 1.2, "foo"], [2, 2.2, "bar"]], columns=["my_ints", "my_floats", "strings"] ) ) spark.sql("DROP TABLE IF EXISTS demo.my_namespace.my_table") spark.sql( "CREATE TABLE demo.my_namespace.my_table (my_ints INT, my_floats DOUBLE, strings STRING) USING iceberg" ) sdf.writeTo("demo.my_namespace.my_table").append() spark.table("demo.my_namespace.my_table").show() ``` --- # Production Source: https://docs.lakekeeper.io/production/ # Running Lakekeeper in Production Lakekeeper is the heart of your data platform. The open-source catalog is built to run in production — it holds no local state, scales horizontally behind a load balancer, and upgrades stay online while database migrations run. Enterprise and regulated production demands more than uptime: governed and provable access control, audited identity, and maintenance run for you. That is what **Lakekeeper+** delivers — the production-grade path for the enterprise, backed by [Vakamo](https://vakamo.com). For these environments, Lakekeeper+ is what you need. ## Production checklist (open source) Everything you need to run Lakekeeper OSS safely is covered in the docs. In short: - **High-availability Postgres** as the catalog backend, with separate read/write URLs and regular, tested backups. - **Authentication** enabled through your existing OpenID provider (or Kubernetes auth). - **Authorization** configured for your access model. - **Horizontal scaling** with multiple instances — use the [Helm chart](https://github.com/lakekeeper/lakekeeper-charts/tree/main/charts/lakekeeper), which ships readiness/liveness probes and autoscaling. - **TLS termination** at a reverse proxy or ingress — Lakekeeper does not terminate connections itself. - **Monitoring & observability** wired to your stack. See the full [Production Checklist](./docs/nightly/production.md) for the complete, up-to-date list. ## Lakekeeper+: the production-grade path Lakekeeper+ builds on the open-source catalog with the capabilities regulated and large-scale platforms need, developed and supported by the team behind Lakekeeper.
- :material-file-lock:   **Permission-as-code with Cedar** --- Express access policy as versioned, reviewable [Cedar](./docs/nightly/authorization-cedar.md) policies — RBAC, ABAC, and property-based access in one model. Every decision is inspectable, with a per-decision policy trace and audit log, so you can prove *why* access was granted. Built for **regulated industries** where access must be governed, testable, and provable. - :material-robot:   **Autonomous maintenance** --- Keep query performance high and storage costs low without operating maintenance jobs yourself. [Table maintenance](./docs/nightly/table-maintenance.md) — expiring snapshots and removing orphan files — runs automatically and adaptively, scheduling itself per table based on how fast reclaimable data builds up. - :material-account-key:   **Enterprise identity & governance** --- Resolve roles through [role providers](./docs/nightly/configuration.md#role-provider) — Okta, Microsoft Entra ID, and LDAP; gate access with an external admission check; protect provider-synced roles from drift. Structured audit logs make the catalog auditable end to end. - :material-lifebuoy:   **Enterprise support & LTS** --- Commercial support from [Vakamo](https://vakamo.com){target="_blank" rel="noopener noreferrer"} for self-hosted and managed deployments, plus hardened Long-Term Support release lines — so you can run Lakekeeper at the heart of your platform with confidence.
## Get Lakekeeper+ Talk to the team at Vakamo about running Lakekeeper in production — self-hosted or managed. We help you understand which edition fits, and get you there.
See Plus release notes
Opens a secure contact form hosted by Zoho on behalf of Vakamo; submitting it shares your details with Vakamo. Prefer to chat with the community first? [Join us on Discord](https://discord.gg/jkAGG8p93B). --- # Community Source: https://docs.lakekeeper.io/support/ # Community
- :fontawesome-brands-discord:   __[Connect on Discord](https://discord.gg/jkAGG8p93B)__ --- Connect with us on Discord to ask all your questions, stay up-to-date with the latest announcements and learn how others are using Lakekeeper. - :fontawesome-brands-github:   __[Report Issues & Feature Request](https://github.com/lakekeeper/lakekeeper/issues/new)__ --- Open Feature Requests and report Issues on GitHub. - :material-webhook:   __[Enterprise Support](https://vakamo.com)__ --- Get enterprise support for Lakekeeper in self-hosted or managed environments.