From 11f7294fcc2f332ffbd4adf63696307c1d92fcdc Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Wed, 5 Aug 2026 00:18:44 +0200 Subject: [PATCH] packfile: fix perf regression with many packs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since 589127caa730 (packfile: move list of packs into the packfile store, 2025-10-30), there is a performance regression when many packfiles need to be loaded: `packfile_store_add_pack()` now calls `packfile_list_remove_internal()` to detect whether the packfile was _already_ in the list, if if so, move it to the end of the list. This function linearly scans the existing list before every insertion. Newly loading N packs therefore has complexity O(N²). In one reported use case (https://github.com/microsoft/git/issues/970), N equals 37,815 and caused a slow-down of a simple `git rev-parse --short HEAD` (which is regularly executed as part of `GIT_PS1`) from 0.4s to 4.5s. In another, heavily exercised CI scenario, clone times increased from under 2 minutes to over half an hour. Let's fix this by establishing a fast path for known-new packfiles. The keen reader will note that there is currently only a single, "known-new" caller of the `packfile_list_append()` function, and wonder why not simply remove this check whether the packfile already exists in the list? Originally, when above-mentioned commit introduced that logic, there was a second caller in `prepare_midx()`, which would have required that check, but that caller was removed in 6aff1f25a046 (packfile: always add packfiles to MRU when adding a pack, 2025-10-30). Still, the function is declared in a header file, and to avoid any problems with in-flight or downstream callers, it is safer to extend the signature to be explicit whether or not to skip that check. Signed-off-by: Johannes Schindelin --- packfile.c | 7 ++++--- packfile.h | 2 +- t/perf/p5303-many-packs.sh | 4 ++++ 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/packfile.c b/packfile.c index 84c36d5f0f2a22..83d48266a73317 100644 --- a/packfile.c +++ b/packfile.c @@ -103,11 +103,12 @@ void packfile_list_prepend(struct packfile_list *list, struct packed_git *pack) list->tail = entry; } -void packfile_list_append(struct packfile_list *list, struct packed_git *pack) +void packfile_list_append(struct packfile_list *list, struct packed_git *pack, + int is_new) { struct packfile_list_entry *entry; - entry = packfile_list_remove_internal(list, pack); + entry = is_new ? NULL : packfile_list_remove_internal(list, pack); if (!entry) { entry = xmalloc(sizeof(*entry)); entry->pack = pack; @@ -865,7 +866,7 @@ void packfile_store_add_pack(struct packfile_store *store, if (pack->pack_fd != -1) pack_open_fds++; - packfile_list_append(&store->packs, pack); + packfile_list_append(&store->packs, pack, 1); strmap_put(&store->packs_by_path, pack->pack_name, pack); } diff --git a/packfile.h b/packfile.h index e4bc0a53828fba..180f67646c9346 100644 --- a/packfile.h +++ b/packfile.h @@ -66,7 +66,7 @@ struct packfile_list_entry { void packfile_list_clear(struct packfile_list *list); void packfile_list_remove(struct packfile_list *list, struct packed_git *pack); void packfile_list_prepend(struct packfile_list *list, struct packed_git *pack); -void packfile_list_append(struct packfile_list *list, struct packed_git *pack); +void packfile_list_append(struct packfile_list *list, struct packed_git *pack, int is_new); /* * Find the pack within the "packs" list whose index contains the object diff --git a/t/perf/p5303-many-packs.sh b/t/perf/p5303-many-packs.sh index af173a7b73e398..4221f9dd706243 100755 --- a/t/perf/p5303-many-packs.sh +++ b/t/perf/p5303-many-packs.sh @@ -141,4 +141,8 @@ test_perf "load 10,000 packs" ' git rev-parse --verify "HEAD^{commit}" ' +test_perf "abbreviate with 10,000 packs" ' + git rev-parse --short HEAD +' + test_done