Databricks
Unity Catalog or Hive Metastore, on AWS, Azure, or GCP.
Access mode: Read-only or read-write
Trino connector. Queryable via Flume’s Lakehouse. This system can be accessed both through its native protocol (for metadata introspection) and via Trino federation (for data profiling and cross-system analytical queries).
Required information
| Field | Details |
|---|---|
| Workspace URL | https://<workspace>.cloud.databricks.com or regional equivalent. |
| SQL Warehouse / Cluster | Compute endpoint. Prefer SQL Warehouse (serverless or pro) for cost isolation. Provide the HTTP path. |
| Catalog(s) | Unity Catalog: list catalogs. Legacy Hive: typically hive_metastore. |
| Schema(s) | Within each catalog. |
| Authentication | OAuth M2M service principal (preferred), PAT, or Azure AD token. |
| Access level | Read-only: USE CATALOG, USE SCHEMA, SELECT on tables. Functions: EXECUTE FUNCTION. |
Network considerations
Default: SaaS, connections over HTTPS (443).
Private Link: If workspace uses AWS PrivateLink / Azure Private Link, need network peering or transit gateway.
IP access lists: If enabled, add Flume egress IPs. Check workspace Settings → Security → IP Access Lists.
Credential and auth management
Preferred: OAuth M2M (service principal). Create Databricks service principal, generate OAuth secret, grant to catalogs/schemas. Short-lived tokens, automatic rotation.
Acceptable: PAT. Simpler but has fixed expiry (max 1 year). No auto-rotation. OK for POC/dev.
Azure AD token: For Azure-hosted workspaces, use managed identity. Provide service principal app ID + secret.
Token refresh: Handled by Flume’s ODBC/SDK layer.
Stored procedure and logic access
Databricks UDFs (Python, SQL, Scala) accessible via SELECT. For Unity Catalog, EXECUTE permission required. Legacy Hive metastore: permissions managed at cluster level. UDF introspection via SHOW FUNCTIONS IN <catalog>.<schema>.
Validation checks
| Check | Method | Expected result |
|---|---|---|
| Network reachability | HTTPS GET to workspace URL | 200 OK |
| Authentication | SELECT current_user() | Returns service principal ID |
| Catalog access | SHOW SCHEMAS IN <catalog> | Lists schemas |
| Table access | SELECT * FROM <catalog>.<schema>.<table> LIMIT 1 | Returns a row |
| Function listing | SHOW FUNCTIONS IN <catalog>.<schema> | Lists UDFs |
| Permission audit | SHOW GRANTS ON SCHEMA <catalog>.<schema> | Grants match spec |
Every connection starts from the pre-engagement checklist and goes through the universal validation protocol before production sign-off.