feat: add openlineage support

[razvan]

Description

Trino can emit OpenLineage events for queries. This PR makes essential properties configurable. The operator uses sensible defaults where possible.

Depends on:

• feat: add crd::openlineage module operator-rs#1250
• new op-rs release

Part of stackabletech/issues#856

Decision: https://github.com/stackabletech/decisions/issues/90

CRD change

See extra/crds.yaml for the CRD diff.

Examples

The example below shows how add inline spec.clusterConfig.lineage configuration to a TrinoCluster resource.

The following fields are new:

• connection : provided by the OpenLineageConfig struct from op-rs and common to all products supporting this in the future.
• namespace: same as connection above. The default is default. Identifies the namespace of Open Lineage Job events.
• jobNameFormat: specific to Trino. Template used to build names for Job events.
• datasetNamespaceUri: specific to Trino. Renamed from trinoUri in decision 90. Used to build the data set namespace. The default value is https://<trino cluster name>.<k8s namespace>.

Inline configuration

---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: simple-trino
spec:
  image:
    productVersion: "481"
  clusterConfig:
    catalogLabelSelector:
      matchLabels:
        trino: simple-trino
    lineage:
      connection:
        inline:
          http:
            # Must match a Subject Alternative Name in the backend's TLS certificate.
            host: marquez.lineage.svc.cluster.local
            port: 5000
            # Default: /api/v1/lineage
            path: /api/v1/lineage
            tls:
              verification:
                server:
                  caCert:
                    # SecretClass providing the CA that issued the backend's certificate.
                    secretClass: marquez-ca
            # Secret holding the bearer token under the key `apiKey`. Omit to disable authentication.
            credentialsSecretName: openlineage-token
      # OpenLineage job namespace. Default: default
      namespace: trino-prod
      # OpenLineage job name. Default: $QUERY_ID
      jobNameFormat: $USER-$QUERY_ID
      # Datasets are reported under trino://trino.example.com.
      # Default: https://<TrinoCluster name>.<Kubernetes namespace>
      datasetNamespaceUri: https://trino.example.com
  coordinators:
    roleGroups:
      default:
        replicas: 1
  workers:
    roleGroups:
      default:
        replicas: 1

Reference

---
# Reusable OpenLineage backend connection. Must be in the same namespace as the TrinoCluster.
apiVersion: lineage.stackable.tech/v1alpha1
kind: OpenLineageConnection
metadata:
  name: marquez
spec:
  http:
    # Must match a Subject Alternative Name in the backend's TLS certificate.
    host: marquez.lineage.svc.cluster.local
    port: 5000
    # Default: /api/v1/lineage
    path: /api/v1/lineage
    tls:
      verification:
        server:
          caCert:
            # SecretClass providing the CA that issued the backend's certificate.
            secretClass: marquez-ca
    # Secret holding the bearer token under the key `apiKey`. Omit to disable authentication.
    credentialsSecretName: openlineage-token
---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
  name: simple-trino
spec:
  image:
    productVersion: "481"
  clusterConfig:
    catalogLabelSelector:
      matchLabels:
        trino: simple-trino
    lineage:
      connection:
        # Name of the OpenLineageConnection above.
        reference: marquez
      # OpenLineage job namespace. Default: default
      namespace: trino-prod
      # OpenLineage job name. Default: $QUERY_ID
      jobNameFormat: $USER-$QUERY_ID
      # Datasets are reported under trino://trino.example.com.
      # Default: https://<TrinoCluster name>.<Kubernetes namespace>
      datasetNamespaceUri: https://trino.example.com
  coordinators:
    roleGroups:
      default:
        replicas: 1
  workers:
    roleGroups:
      default:
        replicas: 1

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[lfrancke]

Comparing this with our decision trinoUri is missing.

[razvan]

Comparing this with our decision trinoUri is missing.

I renamed it to datasetNamespaceUri. I believe it conveys it's purpose better

I updated the description to point out the renaming.