Problems with IngestV2 on existing indexes after Update Quickwit from 0.8 to 0.9

[chsuhr]

Describe the bug
A clear and concise description of what the bug is.

After Upgrade Quickwit to Version 0.9, ingestv2 has problems with existing indexes.

Before the update stopped ingest and created the backups of the metastore (RDS Postgres)
The Update itself was executed like described in the Upgrade Guide and didn't show any errors.

We use Vector.dev to do the log ingest and transformation.
After start ingest again we got errors like this from quickwit:

2026-08-13T14:06:01.781Z INFO index-doc-batches{index_id=aws-r53-vpc-dns-logs-v0_7 source_id=_ingest-source pipeline_uid=01KZXPHNWDJPX3V6HM3THZGWC8 workbench_id=01KZXQ1KAVA6XBGE6N4ADD0BYV}:publisher{split_update=SplitsUpdate { index_id: "aws-r53-vpc-dns-logs-v0_7", new_splits: "01KZXQ1KAXR1EZVZTFZK85MGKR", checkpoint_delta: Some(_ingest-source:∆(01KZXPHNW5G7S9VJZ6HPZDHSN
M:(00000000000000156859..00000000000000165480])) }}: quickwit_indexing::actors::log_publisher_impl: publish-new-splits num_splits=1 num_docs=8621 split_size_bytes=655075
2026-08-13T14:06:02.725Z INFO ingest.ingester.truncate_shards:ingester.truncate_shard: quickwit_ingest::ingest_v2::state: truncated shard aws-r53-vpc-dns-logs-v0_7:01JT0KZ21XHVHWT520FMCGS5NK/_ingest-source/01KZXPHNW5G7S9VJZ6HPZDHSNM at 00000000000000165480 initiated via indexer RPC
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required" num_suppressed=173405
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required"
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required"
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required"
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required"
2026-08-13T14:06:54.508Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-cloudtrail-logs-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=5350 num_bytes=10020192}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the document must contain field "timestamp field is required"

Or:
2026-08-13T14:42:57.228Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-vpcflowlogs-v5-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=11496 num_bytes=8012859}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the field version could not be parsed: expected string, got 5
2026-08-13T14:42:57.228Z ERROR http_request{otel.kind="Server" http.request.method=POST url.path=/api/v1/aws-vpcflowlogs-v5-v0_7/ingest}:ingest.router.ingest{commit_type=1}:ingest.ingester.persist{leader_id=prod-quickwit-indexer-0 commit_type=1}:ingester.validate_doc_batch{num_docs=11496 num_bytes=8012859}: quickwit_ingest::ingest_v2::doc_mapper: failed to validate JSON document: the field version could not be parsed: expected string, got 5

The log format hasnt changed and the fields mentioned in the errors are defined in the index config.

The cloudtrail index is type dynamic and the vpcflowlogs is type strict

We couldnt ingest any data until we forced quickwit to use ingestv1 by adding the "?use_legacy_ingest=true" to the vector sink.

qw_aws_cloudtrail:
  type: http
  method: post
  inputs:
    - split_cloudtrail
  encoding:
    codec: "json"
  framing:
    method: "newline_delimited"
  uri: "http://prod-quickwit-indexer.quickwit.svc.cluster.local:7280/api/v1/aws-cloudtrail-logs-v0_7/ingest?use_legacy_ingest=true"

Then those error messages were gone and we were able to ingest again.

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

1. Create Indexes with Quickwit 0.8 and ingest data.
2. Upgrade Quickwit to 0.9 and ingest data with now default IngestV2 API

Expected behavior
A clear and concise description of what you expected to happen.

existing Indexes work with Ingestv2 and quickwit 0.9 as well after Upgrade.

Configuration:
Please provide:

1. Output of quickwit --version --> Quickwit 0.9.0 (aarch64-unknown-linux-gnu 2026-07-25T17:48:09Z cc420c3

2. AWS EKS 1.35, Quickwit helmchart 0.8.14, AWS RDS Postgres 16.13 Metastore

3. one Indexer and Searcher Node

4. Ingest with vector.dev

5. The index_config.yaml

I got an error (Failed to upload ), if i try to upload the index config yaml to github

[chsuhr]

aws-cloudtrail-logs-v0_7.yaml:
version: "0.7"
index_id: aws-cloudtrail-logs-v0_7
doc_mapping:
mode: dynamic
field_mappings:
- name: eventTime
type: datetime
fast: true
- name: eventSource
type: text
tokenizer: raw
fast: true
- name: eventName
type: text
tokenizer: raw
fast: true
- name: awsRegion
type: text
tokenizer: raw
fast: true
- name: sourceIPAddress
type: text
tokenizer: raw
fast: true
- name: userAgent
type: text
- name: errorCode
type: text
tokenizer: raw
fast: true
- name: errorMessage
type: text
- name: requestParameters
type: json
- name: responseElements
type: json
- name: eventType
type: text
tokenizer: raw
fast: true
- name: readOnly
type: bool
fast: true
- name: recipientAccountId
type: text
tokenizer: raw
fast: true
- name: userIdentity.type
type: text
tokenizer: raw
fast: true
- name: userIdentity.principalId
type: text
tokenizer: raw
fast: true
- name: userIdentity.arn
type: text
tokenizer: raw
fast: true
- name: userIdentity.accountId
type: text
tokenizer: raw
fast: true
- name: userIdentity.accessKeyId
type: text
tokenizer: raw
fast: true
- name: userIdentity.userName
type: text
tokenizer: raw
fast: true
- name: userIdentity.sessionContext.sessionIssuer.type
type: text
tokenizer: raw
- name: userIdentity.sessionContext.sessionIssuer.userName
type: text
tokenizer: raw
- name: resources
type: array
- name: apiVersion
type: text
tokenizer: raw
timestamp_field: eventTime
indexing_settings:
commit_timeout_secs: 60
split_num_docs_target: 10000000
docstore_compression_level: 8
search_settings:
default_search_fields:
- eventName
- userIdentity.userName
- userIdentity.principalId
- sourceIPAddress
- errorMessage
- eventSource
retention:
period: 365 days
schedule: daily

aws-vpcflowlogs-v5-v0_7.yaml:
version: 0.7

index_id: aws-vpcflowlogs-v5-v0_7

doc_mapping:
mode: strict
field_mappings:
- name: version
type: u64
fast: true
- name: account_id
type: text
tokenizer: raw
fast: true
- name: interface_id
type: text
tokenizer: raw
fast: true
- name: srcaddr
type: ip
fast: true
- name: dstaddr
type: ip
fast: true
- name: srcport
type: u64
fast: true
- name: dstport
type: u64
fast: true
- name: protocol
type: u64
fast: true
- name: protocol_name
type: text
tokenizer: raw
fast: true
- name: packets
type: u64
fast: true
- name: bytes
type: u64
fast: true
- name: start
type: datetime
fast: true
- name: end
type: datetime
fast: true
- name: action
type: text
tokenizer: raw
- name: log_status
type: text
tokenizer: raw
- name: vpc_id
type: text
tokenizer: raw
fast: true
- name: subnet_id
type: text
tokenizer: raw
fast: true
- name: instance_id
type: text
tokenizer: raw
fast: true
- name: tcp_flags
type: u64
fast: true
- name: type
type: text
tokenizer: raw
fast: true
- name: pkt_srcaddr
type: ip
fast: true
- name: pkt_dstaddr
type: ip
fast: true
- name: region
type: text
tokenizer: raw
- name: az_id
type: text
tokenizer: raw
- name: sublocation_type
type: text
tokenizer: raw
- name: sublocation_id
type: text
tokenizer: raw
- name: pkt_src_aws_service
type: text
tokenizer: raw
- name: pkt_dst_aws_service
type: text
tokenizer: raw
- name: flow_direction
type: text
tokenizer: raw
fast: true
- name: traffic_path
type: u64
fast: true
timestamp_field: start

search_settings:
default_search_fields: [srcaddr, dstaddr, action, vpc_id, instance_id]

indexing_settings:
commit_timeout_secs: 60
split_num_docs_target: 10000000
docstore_compression_level: 8

retention:
period: 90 days
schedule: daily