Skip to content

Convert the legacy SPA endpoints to private operations #2227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dd-octo-sts wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Convert the legacy SPA endpoints to private operations #2227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
317 changes: 0 additions & 317 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21328,13 +21328,6 @@ components:
- description: "If this is a string, it must be a valid JavaScript expression that evaluates to a boolean."
example: "${true}"
type: string
ComponentRecommendation:
description: Resource recommendation for a single Spark component (driver or executor). Contains estimation data used to patch Spark job specs.
properties:
estimation:
$ref: "#/components/schemas/Estimation"
required: [estimation]
type: object
ComponentType:
description: The UI component type.
enum:
Expand Down Expand Up @@ -24057,23 +24050,6 @@ components:
type: string
x-enum-varnames:
- CI_APP_COVERAGE_SUMMARY
Cpu:
description: CPU usage statistics derived from historical Spark job metrics. Provides multiple estimates so users can choose between conservative and cost-saving risk profiles.
properties:
max:
description: Maximum CPU usage observed for the job, expressed in millicores. This represents the upper bound of usage.
format: int64
type: integer
p75:
description: 75th percentile of CPU usage (millicores). Represents a cost-saving configuration while covering most workloads.
format: int64
type: integer
p95:
description: 95th percentile of CPU usage (millicores). Balances performance and cost, providing a safer margin than p75.
format: int64
type: integer
type: object
x-model-simple-name: SpaCpu
CreateActionConnectionRequest:
description: Request used to create an action connection.
properties:
Expand Down Expand Up @@ -40501,28 +40477,6 @@ components:
type: string
x-enum-varnames:
- ESCALATION_POLICY_STEPS
Estimation:
description: Recommended resource values for a Spark driver or executor, derived from recent real usage metrics. Used by SPA to propose more efficient pod sizing.
properties:
cpu:
$ref: "#/components/schemas/Cpu"
ephemeral_storage:
description: Recommended ephemeral storage allocation (in MiB). Derived from job temporary storage patterns.
format: int64
type: integer
heap:
description: Recommended JVM heap size (in MiB).
format: int64
type: integer
memory:
description: Recommended total memory allocation (in MiB). Includes both heap and overhead.
format: int64
type: integer
overhead:
description: Recommended JVM overhead (in MiB). Computed as total memory - heap.
format: int64
type: integer
type: object
Event:
additionalProperties: {}
description: The metadata associated with a request.
Expand Down Expand Up @@ -101716,96 +101670,6 @@ components:
required:
- data
type: object
RecommendationAttributes:
description: Attributes of the SPA Recommendation resource. Contains recommendations for both driver and executor components.
properties:
confidence_level:
description: The confidence level of the recommendation, expressed as a value between 0.0 (low confidence) and 1.0 (high confidence).
format: double
type: number
driver:
$ref: "#/components/schemas/ComponentRecommendation"
executor:
$ref: "#/components/schemas/ComponentRecommendation"
matched_params:
description: |-
Only returned by the v2 endpoint. The job parameters whose values the recommendation was matched on, as `parameter=value` pairs joined by `|`.
An empty string means the service-wide (coarse) recommendation was used.
example: "detailed_metrics=false|Region=US1"
type: string
required: [driver, executor]
type: object
RecommendationData:
description: JSON:API resource object for SPA Recommendation. Includes type, optional ID, and resource attributes with structured recommendations.
properties:
attributes:
$ref: "#/components/schemas/RecommendationAttributes"
id:
description: Resource identifier for the recommendation. Optional in responses.
type: string
type:
$ref: "#/components/schemas/RecommendationType"
required: [type, attributes]
type: object
RecommendationDocument:
description: JSON:API document containing a single Recommendation resource. Returned by SPA when the Spark Gateway requests recommendations.
properties:
data:
$ref: "#/components/schemas/RecommendationData"
required: [data]
type: object
RecommendationType:
default: recommendation
description: JSON:API resource type for Spark Pod Autosizing recommendations. Identifies the Recommendation resource returned by SPA.
enum:
- recommendation
example: recommendation
type: string
x-enum-varnames:
- RECOMMENDATION
RecommendationV2RequestAttributes:
description: |-
Attributes for requesting SPA recommendations by forwarding a Spark job's raw arguments
instead of a pre-computed shard.
properties:
arguments:
description: |-
Raw, unfiltered Spark job arguments as submitted (for example, `--org_id=2`).
SPA determines which arguments are relevant for the given service.
example:
- ""
items:
type: string
type: array
required: [arguments]
type: object
RecommendationV2RequestBody:
description: |-
Request body for retrieving SPA recommendations by forwarding a Spark job's raw arguments
instead of a pre-computed shard.
properties:
data:
$ref: "#/components/schemas/RecommendationV2RequestData"
required: [data]
type: object
RecommendationV2RequestData:
description: JSON:API resource object for the SPA v2 recommendation request.
properties:
attributes:
$ref: "#/components/schemas/RecommendationV2RequestAttributes"
type:
$ref: "#/components/schemas/RecommendationV2RequestType"
required: [type, attributes]
type: object
RecommendationV2RequestType:
default: recommendation_v2_request
description: JSON:API resource type for the SPA v2 recommendation request.
enum:
- recommendation_v2_request
example: recommendation_v2_request
type: string
x-enum-varnames:
- RECOMMENDATION_V2_REQUEST
RecommendationsFilterRequest:
description: JSON:API request body for filtering cost recommendations.
example:
Expand Down Expand Up @@ -234730,185 +234594,6 @@ paths:
x-unstable: |-
**Note**: This endpoint is in preview and is subject to change.
If you have any feedback, contact [Datadog support](https://docs.datadoghq.com/help/).
/api/v2/spa/recommendations-v2/{service}:
post:
description: |-
This endpoint is experimental and restricted to Datadog internal use only.
Retrieve resource recommendations for a Spark job. The caller (Spark Gateway) provides
a service name and the job's raw arguments. SPA determines which arguments are relevant
for the service and returns structured recommendations for driver and executor resources.
operationId: GetSPARecommendationsV2
parameters:
- description: The service name for a Spark job
in: path
name: service
required: true
schema:
type: string
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/RecommendationV2RequestBody"
required: true
responses:
"200":
content:
application/json:
example:
data:
attributes:
driver:
estimation:
cpu: {max: 1500, p75: 1000, p95: 1200}
ephemeral_storage: 896
heap: 6144
memory: 7168
overhead: 1024
executor:
estimation:
cpu: {max: 2000, p75: 1200, p95: 1500}
ephemeral_storage: 512
heap: 3072
memory: 4096
overhead: 1024
matched_params: "detailed_metrics=false"
id: "dedupeactivecontexts:adp_dedupeactivecontexts_org2"
type: recommendation
schema:
$ref: "#/components/schemas/RecommendationDocument"
description: OK
"400":
$ref: "#/components/responses/BadRequestResponse"
"403":
$ref: "#/components/responses/NotAuthorizedResponse"
"404":
$ref: "#/components/responses/NotFoundResponse"
"429":
$ref: "#/components/responses/TooManyRequestsResponse"
security:
- AuthZ: []
summary: Get SPA recommendations v2
tags:
- Spa
x-unstable: |-
**Note**: This endpoint is in preview and may change in the future. It is not yet recommended for production use.
/api/v2/spa/recommendations/{service}:
get:
description: This endpoint is currently experimental and restricted to Datadog internal use only. Retrieve resource recommendations for a Spark job. The caller (Spark Gateway or DJM UI) provides a service name and SPA returns structured recommendations for driver and executor resources. The version with a shard should be preferred, where possible, as it gives more accurate results.
operationId: GetSPARecommendations
parameters:
- description: The recommendation service should not use its metrics cache.
in: query
name: bypass_cache
schema:
type: string
- description: The service name for a spark job.
in: path
name: service
required: true
schema:
type: string
responses:
"200":
content:
application/json:
example:
data:
attributes:
driver:
estimation:
cpu: {max: 1500, p75: 1000, p95: 1200}
ephemeral_storage: 896
heap: 6144
memory: 7168
overhead: 1024
executor:
estimation:
cpu: {max: 2000, p75: 1200, p95: 1500}
ephemeral_storage: 512
heap: 3072
memory: 4096
overhead: 1024
id: "dedupeactivecontexts:adp_dedupeactivecontexts_org2"
type: recommendation
schema:
$ref: "#/components/schemas/RecommendationDocument"
description: OK
"400":
$ref: "#/components/responses/BadRequestResponse"
"403":
$ref: "#/components/responses/NotAuthorizedResponse"
"429":
$ref: "#/components/responses/TooManyRequestsResponse"
security:
- AuthZ: []
summary: Get SPA Recommendations
tags:
- Spa
x-unstable: |-
**Note**: This endpoint is in preview and may change in the future. It is not yet recommended for production use.
/api/v2/spa/recommendations/{service}/{shard}:
get:
description: This endpoint is currently experimental and restricted to Datadog internal use only. Retrieve resource recommendations for a Spark job. The caller (Spark Gateway or DJM UI) provides a service name and shard identifier, and SPA returns structured recommendations for driver and executor resources.
operationId: GetSPARecommendationsWithShard
parameters:
- description: The shard tag for a spark job, which differentiates jobs within the same service that have different resource needs
in: path
name: shard
required: true
schema:
type: string
- description: The service name for a spark job
in: path
name: service
required: true
schema:
type: string
- description: The recommendation service should not use its metrics cache.
in: query
name: bypass_cache
schema:
type: string
responses:
"200":
content:
application/json:
example:
data:
attributes:
driver:
estimation:
cpu: {max: 1500, p75: 1000, p95: 1200}
ephemeral_storage: 896
heap: 6144
memory: 7168
overhead: 1024
executor:
estimation:
cpu: {max: 2000, p75: 1200, p95: 1500}
ephemeral_storage: 512
heap: 3072
memory: 4096
overhead: 1024
id: "dedupeactivecontexts:adp_dedupeactivecontexts_org2"
type: recommendation
schema:
$ref: "#/components/schemas/RecommendationDocument"
description: OK
"400":
$ref: "#/components/responses/BadRequestResponse"
"403":
$ref: "#/components/responses/NotAuthorizedResponse"
"429":
$ref: "#/components/responses/TooManyRequestsResponse"
security:
- AuthZ: []
summary: Get SPA Recommendations with a shard parameter
tags:
- Spa
x-unstable: |-
**Note**: This endpoint is in preview and may change in the future. It is not yet recommended for production use.
/api/v2/spans/analytics/aggregate:
post:
description: |-
Expand Down Expand Up @@ -249345,8 +249030,6 @@ tags:
externalDocs:
url: https://docs.datadoghq.com/service_catalog/service_definitions#metadata-schema-v30-beta
name: Software Catalog
- description: SPA (Spark Pod Autosizing) API. Provides resource recommendations and cost insights to help optimize Spark job configurations.
name: Spa
- description: |-
Search and aggregate your spans from your Datadog platform over HTTP.
name: Spans
Expand Down
22 changes: 0 additions & 22 deletions examples/v2_spa_GetSPARecommendations.rs
View file
Open in desktop

This file was deleted.

Loading
Loading