Thanks for maintaining this library. CustomContainerTrainingJob.run(max_wait_duration=0) drops the value because the high-level training-job implementation uses a truthiness check. This prevents callers from requesting the documented indefinite wait for DWS Flex Start.
Environment details
- OS type and version: macOS
- Python version: 3.12
google-cloud-aiplatform versions: reproduced in 1.148.1; source inspection confirms the same behavior in 1.150.0, 1.163.0, and current main
Steps to reproduce
- Create a
CustomContainerTrainingJob using any valid project, staging bucket, and container image.
- Run it with Flex Start and
max_wait_duration=0.
- Inspect the scheduling inputs produced by
_prepare_training_task_inputs_and_output_dir or the resulting TrainingPipeline request.
Code example
job.run(
scheduling_strategy=custom_job.Scheduling.Strategy.FLEX_START,
max_wait_duration=0,
)
Observed scheduling value:
Expected scheduling value:
For comparison, a positive input such as 14400 is preserved as 14400s.
Cause
The high-level training-job path converts the duration using a truthiness check, so zero follows the same branch as None:
https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform/training_jobs.py#L1663-L1667
The generated API contract says that explicit zero means indefinite waiting, while omission defaults to 24 hours:
https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform_v1/types/custom_job.py#L577-L582
max_wait_duration is a presence-aware google.protobuf.Duration, so explicit zero and an omitted field are distinct requests.
A previous fix correctly changed the lower-level CustomJob and hyperparameter-tuning paths to use is not None, explicitly noting that 0 is valid:
d9675fd
That commit did not update the CustomContainerTrainingJob path in training_jobs.py.
Suggested fix
Use an explicit is not None check in _prepare_training_task_inputs_and_output_dir, consistent with the lower-level implementation, and add tests covering omitted, zero, and positive durations.
Stack trace
No exception is raised. The problem is a silent request-semantics change: 0 is serialized as omission.
Thanks for maintaining this library.
CustomContainerTrainingJob.run(max_wait_duration=0)drops the value because the high-level training-job implementation uses a truthiness check. This prevents callers from requesting the documented indefinite wait for DWS Flex Start.Environment details
google-cloud-aiplatformversions: reproduced in 1.148.1; source inspection confirms the same behavior in 1.150.0, 1.163.0, and currentmainSteps to reproduce
CustomContainerTrainingJobusing any valid project, staging bucket, and container image.max_wait_duration=0._prepare_training_task_inputs_and_output_diror the resultingTrainingPipelinerequest.Code example
Observed scheduling value:
Expected scheduling value:
For comparison, a positive input such as
14400is preserved as14400s.Cause
The high-level training-job path converts the duration using a truthiness check, so zero follows the same branch as
None:https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform/training_jobs.py#L1663-L1667
The generated API contract says that explicit zero means indefinite waiting, while omission defaults to 24 hours:
https://github.com/googleapis/python-aiplatform/blob/v1.163.0/google/cloud/aiplatform_v1/types/custom_job.py#L577-L582
max_wait_durationis a presence-awaregoogle.protobuf.Duration, so explicit zero and an omitted field are distinct requests.A previous fix correctly changed the lower-level
CustomJoband hyperparameter-tuning paths to useis not None, explicitly noting that0is valid:d9675fd
That commit did not update the
CustomContainerTrainingJobpath intraining_jobs.py.Suggested fix
Use an explicit
is not Nonecheck in_prepare_training_task_inputs_and_output_dir, consistent with the lower-level implementation, and add tests covering omitted, zero, and positive durations.Stack trace
No exception is raised. The problem is a silent request-semantics change:
0is serialized as omission.