Google Cloud Storage

The gcs storage profile backs Warehouses with a Google Cloud Storage bucket, with or without hierarchical namespaces (supported since Lakekeeper 0.8.2). Table locations use gs://. Lakekeeper vends downscoped STS tokens to clients; GCS has no remote signing.

Configuration Parameters

Parameter	Type	Required	Default	Description
bucket	String	Yes	-	Name of the GCS bucket.
key-prefix	String	No	None	Subpath in the bucket to use for this warehouse.
sts-enabled	Boolean	No	true	Whether to enable STS (Security Token Service) downscoped token generation for GCS. When disabled, clients cannot use vended credentials for this storage profile.
storage-layout	Object	No	{"type": "default"}	Controls how namespace and tabular directories are structured under the warehouse base location. See Storage Layout.

Credentials

The storage-credential of a GCS Warehouse has "type": "gcs" and one of these credential-type values:

credential-type	Use for
service-account-key	A service account key, see Service Account Key
gcp-system-identity	The service account the Lakekeeper process runs as, see GCP System Identity

Either way, the service account needs access to the objects of the bucket. The Storage Object User role (roles/storage.objectUser) on the bucket is enough, with or without hierarchical namespaces: Lakekeeper only reads, writes, lists and deletes objects, and the tokens it vends are downscoped to at most that role. Permissions to administer the bucket itself, such as those in Storage Admin, are not needed.

Service Account Key

Create the key in the Google Cloud Console and pass it when creating the Warehouse:

{
  "warehouse-name": "gcs_dev",
  "storage-profile": {
    "type": "gcs",
    "bucket": "...",
    "key-prefix": "..."
  },
  "storage-credential": {
    "type": "gcs",
    "credential-type": "service-account-key",
    "key": {
      "type": "service_account",
      "project_id": "example-project-1234",
      "private_key_id": "....",
      "private_key": "-----BEGIN PRIVATE KEY-----\n.....\n-----END PRIVATE KEY-----\n",
      "client_email": "abc@example-project-1234.iam.gserviceaccount.com",
      "client_id": "123456789012345678901",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abc%40example-project-1234.iam.gserviceaccount.com",
      "universe_domain": "googleapis.com"
    }
  }
}

Use the key as Google creates it. Lakekeeper supports only keys with the universe_domain googleapis.com and a private key in PKCS#8 format (BEGIN PRIVATE KEY). If your key starts with BEGIN RSA PRIVATE KEY, convert it with openssl pkcs8 -topk8 -nocrypt.

GCP System Identity

Warning

Enabling GCP system identities grants Lakekeeper access to any storage location the service account has permissions for. Carefully review and limit the permissions of the service account to avoid unintended access to sensitive resources. Additionally, limit Warehouse creation permissions in Lakekeeper to users who are authorized to access all locations that the system identity can access.

With a system identity, Lakekeeper authenticates with the service account the application or virtual machine runs as, either a Compute Engine default service account or a user-assigned one. Lakekeeper finds this identity with Application Default Credentials: the file named in GOOGLE_APPLICATION_CREDENTIALS, else the gcloud default credentials file, else the metadata server of Compute Engine and GKE. If GOOGLE_APPLICATION_CREDENTIALS names a file that does not exist, Lakekeeper reports an error and does not use the metadata server. Lakekeeper does not read GOOGLE_APPLICATION_CREDENTIALS_JSON and reports an error while it is set: write the key to a file and set GOOGLE_APPLICATION_CREDENTIALS to its path. The feature is disabled by default and must be enabled server-wide:

LAKEKEEPER__ENABLE_GCP_SYSTEM_CREDENTIALS=true

Then create the Warehouse with:

{
  "storage-credential": {
    "type": "gcs",
    "credential-type": "gcp-system-identity"
  }
}

Network

Lakekeeper needs a direct network connection to storage.googleapis.com on port 443. To vend credentials, it also connects to oauth2.googleapis.com and sts.googleapis.com. With a GCP system identity and no credentials file, it gets its tokens from the metadata server metadata.google.internal. The image needs a CA certificate bundle, see SSL Dependencies.

Lakekeeper reads and writes object data over HTTPS. It reads object metadata, lists objects and deletes objects over gRPC (HTTP/2). The gRPC requests do not go through HTTP proxies (HTTPS_PROXY).

CORS

LoQE needs a CORS policy on the bucket. Save this policy as cors.json, replacing https://lakekeeper.example.com with the origin where your Lakekeeper instance is hosted:

[
    {
        "origin": ["https://lakekeeper.example.com"],
        "method": ["GET", "HEAD", "PUT", "POST", "DELETE"],
        "responseHeader": ["Authorization", "Content-Type", "Range", "ETag", "Content-Range"],
        "maxAgeSeconds": 3600
    }
]

GCS returns responseHeader as the allowed request headers of a preflight, and browsers never let a * there cover Authorization, so the headers are listed explicitly. Apply the policy with gcloud storage buckets update gs://<bucket> --cors-file=cors.json.

Updating the Storage Profile

bucket and key-prefix cannot change on update-storage-profile. An update that omits storage-layout keeps the current layout.