feat: Support configuring Iceberg REST catalog

[sbernauer]

Part of #849

Needs stackabletech/operator-rs#1293

Description

This PR allows users to connect Trino to a Iceberg REST catalog via the CRD.
As it's a breaking change (moving to a complex enum), we introduce v1alpha2 for this.

⚠️ Same as with stackabletech/operator-rs#1287, we need figure out how to correctly handle roundtrip conversions. Currently we loose data when converting a REST catalog config from v1alph2 to v1alpha1.

CRD change

See the extra/crds.yaml change. It's a bit hard to spot the diff because of the addition of v1alpha2, so I added it here for now, might be outdated.

Details

diff --git a/extra/crds.yaml b/extra/crds.yaml
index 901063b..037c2c4 100644
--- a/extra/crds.yaml
+++ b/extra/crds.yaml
@@ -3544,7 +3544,7 @@ spec:
     singular: trinocatalog
   scope: Namespaced
   versions:
-  - name: v1alpha1
+  - name: v1alpha2
     schema:
       openAPIV3Schema:
         description: The TrinoCatalog resource can be used to define catalogs in Kubernetes objects.
@@ -4053,6 +4053,100 @@ spec:
                   iceberg:
                     description: An [Apache Iceberg](https://docs.stackable.tech/home/nightly/trino/usage-guide/catalogs/iceberg) connector.
                     properties:
+                      catalog:
+                        description: |-
+                          Connection to a metadata catalog, which will be used as a storage for metadata.
+
+                          We support the following backends:
+
+                          * REST catalog
+                          * Hive metastore
+                          * User provided
+
+                          Details can be found on the corresponding documentation
+                        oneOf:
+                        - required:
+                          - rest
+                        - required:
+                          - hiveMetastore
+                        - required:
+                          - userProvided
+                        properties:
+                          hiveMetastore:
+                            description: Use a Hive metastore to store metadata.
+                            properties:
+                              configMap:
+                                description: Name of the [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery) providing information about the Hive metastore.
+                                maxLength: 253
+                                minLength: 1
+                                pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                                type: string
+                            required:
+                            - configMap
+                            type: object
+                          rest:
+                            description: (Recommended) use a REST catalog to store metadata.
+                            properties:
+                              security:
+                                default:
+                                  none: {}
+                                description: How to authenticate against the REST catalog.
+                                oneOf:
+                                - required:
+                                  - none
+                                - required:
+                                  - oAuth2
+                                properties:
+                                  none:
+                                    description: Don't authenticate against the REST catalog (chosen by default).
+                                    type: object
+                                  oAuth2:
+                                    description: |-
+                                      Use OAuth2 to authenticate against the REST catalog.
+
+                                      Note that we only support configuring a subset of Trino's properties, you might need to use
+                                      `configOverrides` to be able to set all [available properties](https://trino.io/docs/current/object-storage/metastores.html#iceberg-specific-metastores).
+                                    properties:
+                                      credential:
+                                        description: The credential to present to the REST catalog.
+                                        oneOf:
+                                        - required:
+                                          - tokenSecretName
+                                        - required:
+                                          - credentialSecretName
+                                        properties:
+                                          credentialSecretName:
+                                            description: The Secret needs to contain the `clientId` and `clientSecret` keys.
+                                            type: string
+                                          tokenSecretName:
+                                            description: |-
+                                              Authenticate using a bearer token.
+
+                                              The Secret needs to contain the `token` key.
+                                            type: string
+                                        type: object
+                                      serverUri:
+                                        description: The endpoint to retrieve access token from OAuth2 Server.
+                                        format: uri
+                                        type: string
+                                    required:
+                                    - credential
+                                    - serverUri
+                                    type: object
+                                type: object
+                              uri:
+                                description: URL of the rest catalog server.
+                                format: uri
+                                type: string
+                            required:
+                            - uri
+                            type: object
+                          userProvided:
+                            description: |-
+                              The operator doesn't configure any catalog, the user needs to do that,
+                              e.g. using `configOverrides`.
+                            type: object
+                        type: object
                       hdfs:
                         description: |-
                           Connection to an HDFS cluster.
@@ -4068,23 +4162,6 @@ spec:
                         required:
                         - configMap
                         type: object
-                      metastore:
-                        description: |-
-                          Optional connection to a Hive Metastore, which will be used as a storage for metadata.
-
-                          The connection is optional, as Iceberg also supports other catalogs, such as a REST catalog,
-                          which (currently) can only be added using configOverrides.
-                        nullable: true
-                        properties:
-                          configMap:
-                            description: Name of the [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery) providing information about the Hive metastore.
-                            maxLength: 253
-                            minLength: 1
-                            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
-                            type: string
-                        required:
-                        - configMap
-                        type: object
                       s3:
                         description: |-
                           Connection to an S3 store.
@@ -4237,6 +4314,8 @@ spec:
                           reference:
                             type: string
                         type: object
+                    required:
+                    - catalog
                     type: object
                   postgresql:
                     description: An [PostgreSQL](https://docs.stackable.tech/home/nightly/trino/usage-guide/catalogs/postgresql) connector.

Usage examples

Up to date examples are in the PR, but some examples:

- connector:
    iceberg:
      catalog:
        hiveMetastore:
          configMap: simple-hive
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.rest.com/iceberg
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.secure.rest
          security:
            oAuth2:
              serverUri: https://keycloak.default.svc.cluster.local:8443/realms/test/protocol/openid-connect/token
              credential:
                credentialSecretName: my-keycloak-credentials
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.secure.rest
          security:
            oAuth2:
              serverUri: https://keycloak.default.svc.cluster.local:8443/realms/test/protocol/openid-connect/token
              credential:
                tokenSecretName: my-keycloak-token
- connector:
    iceberg:
      catalog:
        userProvided: {}

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[lfrancke]

• credential.credentialSecretName doesn't read well. Can we move this up one level next to serverUri?
  clientCredentialsSecret & tokenSecret. I'm not entirely sure if that works with our enum stuff that said as I just reviewed the OpenLineageConnection I believe we do it exactly the same there so I hope it does work.
  Renaming without name because that's what we use elsewhere in the CRD and others (we also use with name though....)

[sbernauer]

All good points, thanks for your feedback!

1. Re security vs authentication

I think authentication is better, however this maps to the Trino property iceberg.rest-catalog.security (docs: The type of security to use (default: NONE). Possible values are NONE, SIGV4, GOOGLE or OAUTH2. OAUTH2 requires either a token or a credential.).
However, in such discussions in the past I was on the team of "We are a platform, we aim for consistency between products over product experts that know all Trino config properties already". Thus you convinced me, I changed it to security in b3e3e54.

2. Re credential.credentialSecretName

I was considering using an untagged enum like

security:
  oAuth2:
    serverUri: https://keycloak.default.svc.cluster.local:8443/realms/test/protocol/openid-connect/token
    credentialSecretName: my-keycloak-credentials
    # or
    tokenSecretName: my-keycloak-token

This has one level of indentation less, but allows users to configure both (credentials and token) and our operator defines what takes precedence (I'd say token?) and silently ignores the other. That's the tradeoff. I don't have a strong opinion, but I feel like in the past we tended towards less untagged enums and more "impossible to express invalid config". But as mentioned no strong opinion, would be interested in other opinions.

3. Re secret vs secretName

I personally prefer secret over secretName, however in past decisions (e.g. of mine) this was changed to include "Name". Something to do with Kubernetes conventions, and we did that for the last fields I can remember (e.g. https://github.com/stackabletech/decisions/issues/90).

[razvan]

Did you consider adding a new catalog type icebergREST (naming ...) instead if mutating the existing one?

No breaking changes, no conversion problems no new CRD version needed.

[sbernauer]

I have do admit I never though of such a change.
But to be honest it feels like a hack to me.
What catalog is used (REST, glue, dynamodb, hive, jdbc or nessie) is a detail of the iceberg connector, but all of them are the same iceberg connector.
If we follow this approach we would also need a icebergUserProvidedCatalog catalog.