Delta Lake / Iceberg / Hudi on object storage
Table formats on S3, GCS, or ADLS, queried via Spark, Trino, Athena, or Databricks.
Access mode: Read-only (typically)
Trino connector. Queryable via Flume’s Lakehouse. This system can be accessed both through its native protocol (for metadata introspection) and via Trino federation (for data profiling and cross-system analytical queries).
Required information
| Field | Details |
|---|---|
| Storage location | S3 bucket/prefix, GCS bucket, or ADLS container paths where table data resides. |
| Table format | Delta, Iceberg, or Hudi. Determines metadata catalog location. |
| Catalog | Where table metadata lives: AWS Glue Catalog, Hive Metastore, Nessie, Polaris, Unity Catalog, or Iceberg REST Catalog. |
| Query engine | How Flume should query: Athena, Trino/Starburst, Spark/Databricks, Dremio. Determines connection method. |
| Cloud credentials | IAM role (AWS), service account (GCP), or service principal (Azure) with read access to storage + catalog. |
Network considerations
Storage access: Depends on cloud. S3/GCS/ADLS accessed over HTTPS. If bucket is VPC-restricted, need VPC endpoint.
Catalog access: Glue is a regional AWS service (HTTPS). Hive Metastore (Thrift on port 9083) requires network access to the metastore host.
Query engine access: Network requirements depend on engine (see Databricks, Redshift, etc. connector pages).
Credential and auth management
Preferred: IAM role assumption (AWS). Cross-account role with S3 read + Glue read.
Preferred: GCP service account. With Storage Object Viewer + BigLake/Catalog permissions.
Preferred: Azure service principal. With Storage Blob Reader + relevant catalog access.
Storage credentials are separate from query engine credentials: Both must be configured.
Validation checks
| Check | Method | Expected result |
|---|---|---|
| Storage access | aws s3 ls s3://<bucket>/<prefix>/ or equivalent | Lists table partitions/files |
| Catalog access | Query the catalog API (e.g., aws glue get-tables --database-name <db>) | Lists tables in catalog |
| Query test | Execute via chosen engine: SELECT * FROM <table> LIMIT 1 | Returns a row |
| Metadata check | Read table metadata (e.g., DESCRIBE DETAIL <delta_table>) | Shows table format details |
Every connection starts from the pre-engagement checklist and goes through the universal validation protocol before production sign-off.