Skip to content

Add KEDA Kafka consumer-lag scaling (scale.keda) - #4069

Open
aliok wants to merge 2 commits into
knative:mainfrom
aliok:2026-09-28-kafka-lag-scaling
Open

aliok wants to merge 2 commits into
knative:mainfrom
aliok:2026-09-28-kafka-lag-scaling

Conversation

@aliok

@aliok aliok commented Sep 28, 2026

Copy link
Copy Markdown
Member

What

Part 2 of splitting the original KEDA Kafka scaling work (#4051) into reviewable
pieces. This adds the scale.keda surface — KEDA scaler types and triggers —
and the KEDA deployer implementation for Kafka consumer-lag scaling.

Changes

  • Data model — ScaleOptions.KEDA (KEDAScaleOptions + KEDATrigger):
    pollingInterval, cooldownPeriod, and triggers (type http/kafka, with
    targetValue, lagThreshold, activationLagThreshold).
  • Validation — scale.keda/scale.kpa are mutually exclusive; scale.keda
    requires deployer: keda and scale.kpa requires deployer: knative. Kafka
    triggers require a run.kafka config. Bounds-checked numeric fields.
  • KEDA deployer (pkg/keda/kafka_scaling.go) — builds the ScaledObject and,
    when SASL/TLS is configured, a TriggerAuthentication, both owner-referenced to
    the Deployment so they are garbage-collected on delete.
  • describer / lister / remover — corroborate readiness/URL via the Deployment
    when no HTTPScaledObject is present (kafka scaler case).
  • Observed state + refuse gate — the provisioned scaler type is recorded as
    observed state (f.Deploy.ScalerType), mirroring the existing Deployer/Expose
    pattern (written back only on a successful deploy, cleared only on a successful
    remove).
  • Schema + docs regenerated.

Design decision: refuse in-place scaler/trigger-type switches

Unlike #4051, this PR refuses switching the deployer, scaler type, or trigger
type (http↔kafka) on an already-deployed function — the user must func delete
first. Switching in place could orphan a second scaler on the same Deployment while
Deploy reports success (owner-ref GC can't reclaim it because the Deployment
survives). Same-type parameter tweaks (lagThreshold, topic, consumerGroup,
pollingInterval, …) still update the existing ScaledObject in place — no delete
required.

Behavior tightening

run.kafka.sasl.mechanism is now required when SASL is configured (previously an
empty mechanism was silently accepted). Flagging explicitly for reviewers.

Related issue

Testing

  • Unit tests for the validation surface, the KEDA deployer preflights, the refuse
    gate (ValidateScalerSwitch truth table + Client.Deploy refuse/allow cases), and
    observed-state integrity on remove failure.
  • Integration test (//go:build integration) that deploys a Kafka-only function and
    asserts the ScaledObject + TriggerAuthentication are created.
  • go build, go vet, and unit tests for pkg/functions, pkg/keda,
    pkg/deployers pass.

@knative-prow

knative-prow Bot commented Sep 28, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aliok
Once this PR has been reviewed and has the lgtm label, please assign matejvasek for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow Bot added the size/XXL 🤖 PR changes 1000+ lines, ignoring generated files. label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL 🤖 PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant