Elasticsearch / OpenSearch
Self-managed, Elastic Cloud, Amazon OpenSearch Service.
Access mode: Read-only (typically)
Trino connector. Queryable via Flume’s Lakehouse. This system can be accessed both through its native protocol (for metadata introspection) and via Trino federation (for data profiling and cross-system analytical queries).
Required information
| Field | Details |
|---|---|
| Cluster endpoint | URL: https://<host>:9200 or Elastic Cloud endpoint. OpenSearch: AWS-provided endpoint. |
| Index pattern(s) | Specific indices or index patterns (e.g., logs-*, business-data-*). |
| Credentials | Username + password (basic auth), API key, or AWS IAM (OpenSearch). |
| Access level | Read-only: read privilege on target indices + monitor cluster privilege for health checks. |
Network considerations
Self-managed: VPN or SSH tunnel. Default port 9200 (HTTP) and 9300 (transport).
Elastic Cloud: HTTPS over 443. No VPN needed. IP filtering available in deployment settings.
Amazon OpenSearch: VPC-based requires VPC peering/PrivateLink. Public domain requires IP-based access policy.
TLS: Always use HTTPS. Self-signed certs require CA trust configuration.
Credential and auth management
Preferred: API key (Elastic). Scoped to specific indices and privileges. No password rotation needed. Set expiry.
Preferred: IAM auth (OpenSearch). Flume signs requests with SigV4. Requires IAM policy + OpenSearch access policy mapping.
Acceptable: Basic auth. Native realm username + password. Or LDAP/SAML integration.
Elastic Security roles: Create a custom role with read on target indices and assign to the Flume user.
Validation checks
| Check | Method | Expected result |
|---|---|---|
| Network reachability | curl -k https://<host>:9200 | Returns cluster info JSON |
| Authentication | GET /_security/_authenticate | Returns authenticated user |
| Index access | GET /_cat/indices/<pattern>?v | Lists matching indices |
| Read test | GET /<index>/_search?size=1 | Returns a document |
| Permission check | GET /_security/user/<user> | Roles match expected config |
Every connection starts from the pre-engagement checklist and goes through the universal validation protocol before production sign-off.