Job with many results: read timeout on metadata

[jdries]

So one LCFM job has this read timeout on result metadata, the job generated many results:

url = '/openeo/1.2/jobs/j-26013001053549dc9fbe27b7f0c25299/results

https://jenkins.vgt.vito.be/job/openEO/job/openeo-apex-benchmarks/418/testReport/junit/tests/test_b…

https://openeo-admin.vgt.vito.be/job/j-26013001053549dc9fbe27b7f0c25299

Not a blocker, but requires further investigation:

Does request time increase with size of the result? Do we need different timeout?
What about stac 1.1 codepath, is that better?
Anything we can fix? (But avoiding deep dive into 1.0 codepaths...)

[jdries]

@JeroenVerstraelen assigned for triage and prio planning.
It causes false positives in apex tests, and users can also experience this.

[JeroenVerstraelen]

Context
s3://openeo-data-staging-waw4-1/batch_jobs/j-26013001053549dc9fbe27b7f0c25299/job_metadata.json
Contains 312 items in the .assets[] property.

This causes a timeout in the following request:
GET https://openeo-staging.dataspace.copernicus.eu/openeo/1.2/jobs/j-26013001053549dc9fbe27b7f0c25299/results

1. Does request time increase with size of the result? Do we need different timeout?
   - Yes, due to file I/O stats and URL signing per asset (312 assets × ~100ms ≈ 31s); timeout likely needs increasing.

2. What about stac 1.1 codepath, is that better?
   - Yes, when result_metadata.items is populated. It returns a lightweight Collection with item links instead of embedding all 312 assets inline. Edit: stac 1.1 will codepath will provide no improvement.

3. Anything we can fix? (But avoiding deep dive into 1.0 codepaths...)
   - pre-compute file:size during job completion (not at request time) to avoid file I/O at request time, increase timeout for large result sets.

[JeroenVerstraelen]

Added to planning backlog.

[bossie]

2. It returns a lightweight Collection with item links instead of embedding all 312 assets inline.

The STAC 1.1 Collection document also embeds these assets I'm afraid, so this makes no difference. This is how the API mandates it and thus how the clients download these assets: they find the assets in a top-level assets property* rather than walk the STAC tree.

There could potentially be less item links in the STAC 1.1 Collection document because there is no longer a 1:1 mapping of asset to item but I think these are edge cases really (think: separate_assets_per_band).

*this was actually a very real issue that messed up the download of STAC 1.1 assets. Because the top-level assets property is a dictionary with unique keys, and the STAC 1.1 implementation has unified asset keys across items, you can't have multiple assets with the same asset key in this dictionary. The solution there was to use {item_id}_{asset_key} as the dictionary key instead.

[EmileSonneveld]

file:size can be logged as a scala issue