From c035be5d54573bc8dfcbef3ab66de88f2adce7db Mon Sep 17 00:00:00 2001 From: Kristoffer Haugsbakk Date: Sun, 23 Aug 2026 13:32:46 +0200 Subject: [PATCH 001/156] doc: git: list gitdatamodel(7) as a concept guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit dee80940 (doc: add an explanation of Git's data model, 2025-11-12) added gitdatamodel(7), documenting Git’s data model. But it is not mentioned anywhere. Let’s start by listing it under Guides in git(1) and with `git help --guides`. Signed-off-by: Kristoffer Haugsbakk Signed-off-by: Junio C Hamano --- command-list.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/command-list.txt b/command-list.txt index 21b802c42026b3..63ae2a67c94055 100644 --- a/command-list.txt +++ b/command-list.txt @@ -217,6 +217,7 @@ gitcli userinterfaces gitcore-tutorial guide gitcredentials guide gitcvs-migration guide +gitdatamodel guide gitdiffcore guide giteveryday guide gitfaq guide From 3e13dd44e81f344a74173d8e7726a27fb79b2058 Mon Sep 17 00:00:00 2001 From: Kristoffer Haugsbakk Date: Sun, 23 Aug 2026 13:32:47 +0200 Subject: [PATCH 002/156] doc: git: link to the gitdatamodel(7) tutorial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit added the first mention of gitdatamodel(7) here, under Guides. But there are also other sections where a mention is relevant. Let’s mention it: • under Description, since it is as useful as the other tutorials already mentioned there for those who are interested; • under Terminology, since it complements gitglossary(7) as a pedagogical rather than reference source for the core terms;[1] and • under See Also, since the other tutorials (plus the user manual) are mentioned there. We don’t need to mention it under Further Documentation since we now mention it under Description. † 1: See dee80940 (doc: add an explanation of Git's data model, 2025-11-12): `gitglossary`. This makes a good effort, but it's an alphabetically ordered dictionary and a dictionary is not a good way to learn concepts. You have to jump around too much and it's not possible to present the concepts in the order that they should be explained. Signed-off-by: Kristoffer Haugsbakk Signed-off-by: Junio C Hamano --- Documentation/git.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/Documentation/git.adoc b/Documentation/git.adoc index 8a5cdd3b3d22c5..6f0075f9188774 100644 --- a/Documentation/git.adoc +++ b/Documentation/git.adoc @@ -25,7 +25,8 @@ and full access to internals. See linkgit:gittutorial[7] to get started, then see linkgit:giteveryday[7] for a useful minimum set of commands. The link:user-manual.html[Git User's Manual] has a more -in-depth introduction. +in-depth introduction. See linkgit:gitdatamodel[7] if you want to +learn about the data model and important terminology. After you mastered the basic concepts, you can come back to this page to learn what commands Git offers. You can learn more about @@ -469,7 +470,9 @@ Higher level SCMs may provide and manage additional information in the Terminology ----------- -Please see linkgit:gitglossary[7]. +Please see linkgit:gitglossary[7]. See linkgit:gitdatamodel[7] for a +discussion of the core data model, which includes important terminology +used throughout the documentation. Environment Variables @@ -1199,8 +1202,9 @@ SEE ALSO -------- linkgit:gittutorial[7], linkgit:gittutorial-2[7], linkgit:giteveryday[7], linkgit:gitcvs-migration[7], -linkgit:gitglossary[7], linkgit:gitcore-tutorial[7], -linkgit:gitcli[7], link:user-manual.html[The Git User's Manual], +linkgit:gitglossary[7], linkgit:gitdatamodel[7], +linkgit:gitcore-tutorial[7], linkgit:gitcli[7], +link:user-manual.html[The Git User's Manual], linkgit:gitworkflows[7] GIT From b27722d03144ec723ab20dc337ff21be2ea6a0b7 Mon Sep 17 00:00:00 2001 From: Kristoffer Haugsbakk Date: Sun, 23 Aug 2026 13:32:48 +0200 Subject: [PATCH 003/156] doc: glossary: link four of the terms to gitdatamodel(7) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four of the terms in the glossary are discussed in gitdatamodel(7). Let’s link to the data model page from the glossary. The phrasing needs to be tweaked based on what gitdatamodel(7) offers for each term compared to the glossary, or even other pages (see the git-reflog(1) mention). For instance, the ref/reference discussion can be called a “see also” since the glossary here already goes into detail. On the other hand, gitdatamodel(7) offers more details on the subject of “the index”. Let’s also add gitdatamodel(7) to See Also. It is at least as relevant as the other tutorial pages that are already mentioned. Signed-off-by: Kristoffer Haugsbakk Signed-off-by: Junio C Hamano --- Documentation/gitglossary.adoc | 1 + Documentation/glossary-content.adoc | 13 ++++++++++--- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/Documentation/gitglossary.adoc b/Documentation/gitglossary.adoc index 0e85be48470ea0..b046d9cb293977 100644 --- a/Documentation/gitglossary.adoc +++ b/Documentation/gitglossary.adoc @@ -16,6 +16,7 @@ include::glossary-content.adoc[] SEE ALSO -------- +linkgit:gitdatamodel[7], linkgit:gittutorial[7], linkgit:gittutorial-2[7], linkgit:gitcvs-migration[7], diff --git a/Documentation/glossary-content.adoc b/Documentation/glossary-content.adoc index 8c4e9dd3beede1..52301a56ddaa60 100644 --- a/Documentation/glossary-content.adoc +++ b/Documentation/glossary-content.adoc @@ -251,7 +251,8 @@ for a more flexible and robust system to do the same thing. as objects. The index is a stored version of your <>. Truth be told, it can also contain a second, and even a third version of a working tree, which are used - when <>. + when <>. See "THE INDEX" in + linkgit:gitdatamodel[7] for details. [[def_index_entry]]index entry:: The information regarding a particular file, stored in the @@ -290,7 +291,8 @@ This commit is referred to as a "merge commit", or sometimes just a [[def_object]]object:: The unit of storage in Git. It is uniquely identified by the <> of its contents. Consequently, an - object cannot be changed. + object cannot be changed. See "OBJECTS" in + linkgit:gitdatamodel[7] for details. [[def_object_database]]object database:: Stores a set of "objects", and an individual <> is @@ -587,12 +589,17 @@ extended in the future: Different subhierarchies are used for different purposes. For example, the `refs/heads/` hierarchy is used to represent local branches whereas the `refs/tags/` hierarchy is used to represent local tags.. ++ +See also "REFERENCES" in linkgit:gitdatamodel[7]. [[def_reflog]]reflog:: A reflog shows the local "history" of a ref. In other words, it can tell you what the 3rd last revision in _this_ repository was, and what was the current state in _this_ repository, - yesterday 9:14pm. See linkgit:git-reflog[1] for details. + yesterday 9:14pm. ++ +See "REFLOGS" in linkgit:gitdatamodel[7] for a short explanation of the +format. See linkgit:git-reflog[1] for details. [[def_refspec]]refspec:: A "refspec" is used by <> and From 7b2ce85f4c55b4d8eb9c9decd964eb3d899af5ba Mon Sep 17 00:00:00 2001 From: Kristoffer Haugsbakk Date: Sun, 23 Aug 2026 13:32:49 +0200 Subject: [PATCH 004/156] doc: datamodel: link to the glossary We linked from the glossary to the data model page in the last commit. It can also be useful to link the other way for readers who might want to reference more terminology. Signed-off-by: Kristoffer Haugsbakk Signed-off-by: Junio C Hamano --- Documentation/gitdatamodel.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/gitdatamodel.adoc b/Documentation/gitdatamodel.adoc index dcfdff0346f669..d588630e63354f 100644 --- a/Documentation/gitdatamodel.adoc +++ b/Documentation/gitdatamodel.adoc @@ -24,6 +24,8 @@ Git's core operations use 4 kinds of data: 3. <>, also known as the staging area 4. <>: logs of changes to references ("ref log") +See linkgit:gitglossary[7] for a comprehensive terminology reference. + [[objects]] OBJECTS ------- @@ -300,6 +302,10 @@ $ git reflog main --date=iso --no-decorate 4ccb6d7 main@{2025-09-29 15:16:48 -0400}: commit (initial): Initial commit ---- +SEE ALSO +-------- +linkgit:gitglossary[7] + GIT --- Part of the linkgit:git[1] suite From ef8fb7fd96f548a1f2a05463d7102cab2798cc82 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:05 +0200 Subject: [PATCH 005/156] builtin/clone: defer setup of the object database When cloning a repository we defer initialization of the reference database. This is because we don't yet know all details required for us to initialize the refdb in the first place. Most importantly, what we are missing is information about the object hash. We don't do the same thing for the object database yet, but here we essentially have the same problem. While the "files" database does not need any information about the object format at creation time, alternate backends are likely to require that information so that they can properly set up their data structures. Besides this forward-looking future proofing though, we also have a second use case for deferring initialization of the object database, namely alternates. When initializing the object database we do not yet know whether we'll need alternates or not because this depends on the repository we're about to clone from. If it is a local repository and the user has passed "--refernce{,-if-able}", then we will end up writing alternates into the object database. The ugly part though is that we cannot determine where the repository is getting cloned from before it has been initialized. While we of course already have access to the user-provided URI, that URI can be very well rewritten via "url..insteadOf". We can of course read the global- and system-level configuration to resolve it. But we explicitly resolve the URI a second time after we have initialized the repository because it can happen that we copy a ".git/config" over from our templates, and that file may cause us to rewrite the path. In a subsequent commit though we'll start to write alternates as part of the repository initialization, so we'll need to have the URI properly resolved before we can initialize the object database. This is ugly, but as mentioned above it makes sense for us to defer its initialization anyway so that we also know about the object hash already. Introduce a new flag that makes `init_db()` skip initializing the object database. Expose `create_object_database()` and make use of it after we have resolved the URI. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 12 ++++++++---- setup.c | 8 +++++--- setup.h | 15 ++++++++++----- 3 files changed, 23 insertions(+), 12 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 5b25cca5102956..0a67492ebdb818 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1184,11 +1184,14 @@ int cmd_clone(int argc, * database. We do not yet know about the object format of the * repository, and reference backends may persist that information into * their on-disk data structures. + * + * Furthermore, we skip initializing the object database so that we can + * first resolve potential alternates before creating it. */ init_db(the_repository, git_dir, real_git_dir, work_tree, option_template, GIT_HASH_UNKNOWN, ref_storage_format, NULL, do_not_override_repo_unix_permissions, - INIT_DB_QUIET | INIT_DB_SKIP_REFDB); + INIT_DB_QUIET | INIT_DB_SKIP_REFDB | INIT_DB_SKIP_ODB); if (real_git_dir) { free((char *)git_dir); @@ -1311,9 +1314,6 @@ int cmd_clone(int argc, strbuf_reset(&key); } - if (option_required_reference.nr || option_optional_reference.nr) - setup_reference(); - remote = remote_get_early(remote_name); if (!option_rev) @@ -1342,6 +1342,10 @@ int cmd_clone(int argc, if (option_local > 0 && !is_local) warning(_("--local is ignored")); + create_object_database(the_repository); + if (option_required_reference.nr || option_optional_reference.nr) + setup_reference(); + transport = transport_get(remote, path ? path : remote->url.v[0]); transport_set_verbosity(transport, option_verbosity, option_progress); transport->family = family; diff --git a/setup.c b/setup.c index d90654f5842deb..e654e27d05114f 100644 --- a/setup.c +++ b/setup.c @@ -2647,7 +2647,7 @@ static int create_default_files(struct repository *repo, return reinit; } -static void create_object_database(struct repository *repo) +void create_object_database(struct repository *repo) { /* * Create the "objects" directory in the common directory. This is done @@ -2829,7 +2829,8 @@ int init_db(struct repository *repo, const char *template_dir, int hash, enum ref_storage_format ref_storage_format, const char *initial_branch, - int init_shared_repository, unsigned int flags) + int init_shared_repository, + enum init_db_flags flags) { int reinit; int exist_ok = flags & INIT_DB_EXIST_OK; @@ -2903,7 +2904,8 @@ int init_db(struct repository *repo, if (!(flags & INIT_DB_SKIP_REFDB)) create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET); - create_object_database(repo); + if (!(flags & INIT_DB_SKIP_ODB)) + create_object_database(repo); startup_info->have_repository = 1; diff --git a/setup.h b/setup.h index 763fd384e86c28..570ebcd1506967 100644 --- a/setup.h +++ b/setup.h @@ -256,9 +256,12 @@ int apply_repository_format(struct repository *repo, const char *get_template_dir(const char *option_template); -#define INIT_DB_QUIET (1 << 0) -#define INIT_DB_EXIST_OK (1 << 1) -#define INIT_DB_SKIP_REFDB (1 << 2) +enum init_db_flags { + INIT_DB_QUIET = (1 << 0), + INIT_DB_EXIST_OK = (1 << 1), + INIT_DB_SKIP_REFDB = (1 << 2), + INIT_DB_SKIP_ODB = (1 << 3), +}; int init_db(struct repository *repo, const char *git_dir, @@ -266,13 +269,15 @@ int init_db(struct repository *repo, const char *worktree, const char *template_dir, int hash_algo, enum ref_storage_format ref_storage_format, - const char *initial_branch, int init_shared_repository, - unsigned int flags); + const char *initial_branch, + int init_shared_repository, + enum init_db_flags flags); void initialize_repository_version(struct repository *repo, int hash_algo, enum ref_storage_format ref_storage_format, int reinit); void create_reference_database(struct repository *repo, const char *initial_branch, int quiet); +void create_object_database(struct repository *repo); /* * NOTE NOTE NOTE!! From 1cd2aaa24bd3ce714a8706cdb102c91a07f1683c Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:06 +0200 Subject: [PATCH 006/156] builtin/clone: move around `setup_reference()` In a subsequent commit, `setup_reference()` will start to call `copy_alternates()`. Prepare for this by moving the function further down so that we can avoid adding a declaration. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 0a67492ebdb818..8c990ce0ccf741 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -179,16 +179,6 @@ static int add_one_reference(struct string_list_item *item, void *cb_data) return 0; } -static void setup_reference(void) -{ - int required = 1; - for_each_string_list(&option_required_reference, - add_one_reference, &required); - required = 0; - for_each_string_list(&option_optional_reference, - add_one_reference, &required); -} - static void copy_alternates(struct strbuf *src, const char *src_repo) { /* @@ -228,6 +218,16 @@ static void copy_alternates(struct strbuf *src, const char *src_repo) fclose(in); } +static void setup_reference(void) +{ + int required = 1; + for_each_string_list(&option_required_reference, + add_one_reference, &required); + required = 0; + for_each_string_list(&option_optional_reference, + add_one_reference, &required); +} + static void mkdir_if_missing(const char *pathname, mode_t mode) { struct stat st; From 2ab05bcd00679a88985937163800d4e89c334f11 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:07 +0200 Subject: [PATCH 007/156] builtin/clone: refactor handling of "--reference{,-if-able}" Users can pass "--reference{,-if-able}" to git-clone(1) to instruct it to set up alternates for the newly created repository. This allows it to reuse objects from the source repository so that in the best case we don't have to clone all objects over. Those options are handled by the confusingly named `setup_reference()` function -- without the above context, one might rightfully believe that it was about refs, not about alternates. The function itself is rather simple: we loop through all provided alternate paths and then, if such an alternate is valid, we write it to the object database. In subsequent commits we're about to consolidate the complete setup of alternates into this function, and furthermore we'll refactor the setup of the object database to handle doing this for us instead of writing the alterantes into it one by one. Prepare for this refactoring by collecting the alternates into a strvec. Rename the function to `collect_alternates()` to clarify its scope. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 8c990ce0ccf741..8eae3ac7d9bc6b 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -154,24 +154,26 @@ static char *get_repo_path(const char *repo, int *is_bundle) return canon; } -static int add_one_reference(struct string_list_item *item, void *cb_data) +struct add_one_alternate_data { + struct strvec *alternates; + int required; +}; + +static int add_one_alternate(struct string_list_item *item, void *cb_data) { + struct add_one_alternate_data *data = cb_data; struct strbuf err = STRBUF_INIT; - int *required = cb_data; char *ref_git = compute_alternate_path(item->string, &err); if (!ref_git) { - if (*required) + if (data->required) die("%s", err.buf); else fprintf(stderr, _("info: Could not add alternate for '%s': %s\n"), item->string, err.buf); } else { - struct strbuf sb = STRBUF_INIT; - strbuf_addf(&sb, "%s/objects", ref_git); - odb_add_to_alternates_file(the_repository->objects, sb.buf); - strbuf_release(&sb); + strvec_pushf(data->alternates, "%s/objects", ref_git); } strbuf_release(&err); @@ -218,14 +220,20 @@ static void copy_alternates(struct strbuf *src, const char *src_repo) fclose(in); } -static void setup_reference(void) +static void collect_alternates(struct strvec *alternates) { - int required = 1; - for_each_string_list(&option_required_reference, - add_one_reference, &required); - required = 0; - for_each_string_list(&option_optional_reference, - add_one_reference, &required); + if (option_required_reference.nr || option_optional_reference.nr) { + struct add_one_alternate_data data = { + .alternates = alternates, + .required = 1, + }; + + for_each_string_list(&option_required_reference, + add_one_alternate, &data); + data.required = 0; + for_each_string_list(&option_optional_reference, + add_one_alternate, &data); + } } static void mkdir_if_missing(const char *pathname, mode_t mode) @@ -999,6 +1007,7 @@ int cmd_clone(int argc, N_("uri"), N_("a URI for downloading bundles before fetching from origin remote")), OPT_END() }; + struct strvec alternates = STRVEC_INIT; const char * const builtin_clone_usage[] = { N_("git clone [] [--] []"), @@ -1343,8 +1352,10 @@ int cmd_clone(int argc, warning(_("--local is ignored")); create_object_database(the_repository); - if (option_required_reference.nr || option_optional_reference.nr) - setup_reference(); + collect_alternates(&alternates); + + for (size_t i = 0; i < alternates.nr; i++) + odb_add_to_alternates_file(the_repository->objects, alternates.v[i]); transport = transport_get(remote, path ? path : remote->url.v[0]); transport_set_verbosity(transport, option_verbosity, option_progress); @@ -1641,6 +1652,7 @@ int cmd_clone(int argc, string_list_clear(&option_not, 0); string_list_clear(&option_config, 0); string_list_clear(&server_options, 0); + strvec_clear(&alternates); free(remote_name); strbuf_release(&reflog_msg); From ec2df7848f526c183f9e6678dd21e9610f1c10ce Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:08 +0200 Subject: [PATCH 008/156] builtin/clone: move setup of alternates for shared local clones When cloning a local repository with "--shared" we add that repository to the new repository's alternates. This is done in `clone_local()`, which is responsible for performing local clones. Move the logic into `collect_alternates()` to unify our setup of alternates. Furthermore, this will allow us to set up alternates right at creation time of the object database. Note that the logic for cloning a local repository with "--no-shared" is not yet part of `collect_alternates()`. This will be handled in the next commit. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 8eae3ac7d9bc6b..08c8f5a94ffa8b 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -220,7 +220,8 @@ static void copy_alternates(struct strbuf *src, const char *src_repo) fclose(in); } -static void collect_alternates(struct strvec *alternates) +static void collect_alternates(struct strvec *alternates, + const char *src_repo, bool is_local) { if (option_required_reference.nr || option_optional_reference.nr) { struct add_one_alternate_data data = { @@ -234,6 +235,16 @@ static void collect_alternates(struct strvec *alternates) for_each_string_list(&option_optional_reference, add_one_alternate, &data); } + + if (is_local) { + struct strbuf commondir = STRBUF_INIT; + + get_common_dir(&commondir, src_repo); + if (option_shared) + strvec_pushf(alternates, "%s/objects", commondir.buf); + + strbuf_release(&commondir); + } } static void mkdir_if_missing(const char *pathname, mode_t mode) @@ -357,13 +368,7 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, static void clone_local(const char *src_repo, const char *dest_repo) { - if (option_shared) { - struct strbuf alt = STRBUF_INIT; - get_common_dir(&alt, src_repo); - strbuf_addstr(&alt, "/objects"); - odb_add_to_alternates_file(the_repository->objects, alt.buf); - strbuf_release(&alt); - } else { + if (!option_shared) { struct strbuf src = STRBUF_INIT; struct strbuf dest = STRBUF_INIT; get_common_dir(&src, src_repo); @@ -1352,7 +1357,7 @@ int cmd_clone(int argc, warning(_("--local is ignored")); create_object_database(the_repository); - collect_alternates(&alternates); + collect_alternates(&alternates, path, is_local); for (size_t i = 0; i < alternates.nr; i++) odb_add_to_alternates_file(the_repository->objects, alternates.v[i]); From 2e26876b48a9048ce5658168cf39aac56a27a405 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:09 +0200 Subject: [PATCH 009/156] builtin/clone: move setup of alternates for non-shared local clones Similar as in the preceding commit, move the setup of alternates for local clones with "--no-shared" into `collect_alternates()`. With this step, the complete setup of alternates is now handled by that function. Note that besides moving stuff around, it also fixes a bug: previously, we did not know to resolve the referenced repository's common directory. Consequently, when referencing a worktree we failed to resolve alternates. But as `collect_alternates()` already knows to resolve the commondir for "--local" we can simply reuse this resolved path for our purpose. Add two tests, the first one of which exercises this bug to avoid future regressions. The second patch ensures that we properly handle relative alternates for a referenced worktree. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 34 +++++++++++++++++++++++----------- t/t5604-clone-reference.sh | 25 +++++++++++++++++++++++++ 2 files changed, 48 insertions(+), 11 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 08c8f5a94ffa8b..2e3473fddfa388 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -181,7 +181,7 @@ static int add_one_alternate(struct string_list_item *item, void *cb_data) return 0; } -static void copy_alternates(struct strbuf *src, const char *src_repo) +static void read_alternates(struct strvec *alternates, const char *src_repo) { /* * Read from the source objects/info/alternates file @@ -195,29 +195,41 @@ static void copy_alternates(struct strbuf *src, const char *src_repo) * to turn entries with paths relative to the original * absolute, so that they can be used in the new repository. */ - FILE *in = xfopen(src->buf, "r"); + FILE *in; + struct strbuf path = STRBUF_INIT; struct strbuf line = STRBUF_INIT; + strbuf_addf(&path, "%s/objects/info/alternates", src_repo); + + in = fopen(path.buf, "r"); + if (!in) { + if (errno == ENOENT) + goto out; + die_errno("could not read alternates file '%s'", path.buf); + } + while (strbuf_getline(&line, in) != EOF) { char *abs_path; if (!line.len || line.buf[0] == '#') continue; if (is_absolute_path(line.buf)) { - odb_add_to_alternates_file(the_repository->objects, - line.buf); + strvec_push(alternates, line.buf); continue; } abs_path = mkpathdup("%s/objects/%s", src_repo, line.buf); if (!normalize_path_copy(abs_path, abs_path)) - odb_add_to_alternates_file(the_repository->objects, - abs_path); + strvec_push(alternates, abs_path); else warning("skipping invalid relative alternate: %s/%s", src_repo, line.buf); free(abs_path); } + +out: + strbuf_release(&path); strbuf_release(&line); - fclose(in); + if (in) + fclose(in); } static void collect_alternates(struct strvec *alternates, @@ -242,6 +254,8 @@ static void collect_alternates(struct strvec *alternates, get_common_dir(&commondir, src_repo); if (option_shared) strvec_pushf(alternates, "%s/objects", commondir.buf); + else + read_alternates(alternates, commondir.buf); strbuf_release(&commondir); } @@ -320,11 +334,9 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest, continue; } - /* Files that cannot be copied bit-for-bit... */ - if (!fspathcmp(iter->relative_path, "info/alternates")) { - copy_alternates(src, src_repo); + /* Alternates were already handled earlier. */ + if (!fspathcmp(iter->relative_path, "info/alternates")) continue; - } if (unlink(dest->buf) && errno != ENOENT) die_errno(_("failed to unlink '%s'"), dest->buf); diff --git a/t/t5604-clone-reference.sh b/t/t5604-clone-reference.sh index 39a0c318dfb2d4..9e4b98fdb82597 100755 --- a/t/t5604-clone-reference.sh +++ b/t/t5604-clone-reference.sh @@ -383,4 +383,29 @@ test_expect_success 'dissociate from repo with commit graph' ' git clone --no-local --reference graph.git --dissociate orig clone ' +test_expect_success 'local clone from linked worktree carries over alternates' ' + rm -fr base derived derived-wt dst expect && + git init base && + test_commit -C base one && + git clone --shared base derived && + git -C derived worktree add ../derived-wt && + git clone derived-wt dst && + echo "$(pwd)/base/.git/objects" >expect && + test_cmp expect dst/.git/objects/info/alternates && + git -C dst fsck +' + +test_expect_success 'local clone from linked worktree resolves relative alternates' ' + rm -fr base derived derived-wt dst expect && + git init base && + test_commit -C base one && + git clone --shared base derived && + echo "../../../base/.git/objects" >derived/.git/objects/info/alternates && + git -C derived worktree add ../derived-wt && + git clone derived-wt dst && + echo "$(pwd)/base/.git/objects" >expect && + test_cmp expect dst/.git/objects/info/alternates && + git -C dst fsck +' + test_done From d540449b157e1a78ef5d1af664646ff6a90e3758 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:10 +0200 Subject: [PATCH 010/156] odb/source: support writing alternates when creating the database Add the ability to write alternates when creating the object database. This change allows us to remove the `write_alternates()` callback in a subsequent patch. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- odb/source-files.c | 76 ++++++++++++++++++++++++++++++++++++++++++++-- odb/source.h | 17 +++++++++-- setup.c | 4 ++- 3 files changed, 91 insertions(+), 6 deletions(-) diff --git a/odb/source-files.c b/odb/source-files.c index b7b3a297bb79d7..8fe65d91f8eb6b 100644 --- a/odb/source-files.c +++ b/odb/source-files.c @@ -18,6 +18,7 @@ #include "run-command.h" #include "strbuf.h" #include "string-list.h" +#include "strmap.h" #include "strvec.h" #include "tree.h" #include "write-or-die.h" @@ -51,9 +52,14 @@ static void odb_source_files_close(struct odb_source *source) odb_source_close(&files->packed->base); } -static int odb_source_files_create_on_disk(struct odb_source *source) +static int odb_source_files_create_on_disk(struct odb_source *source, + const struct odb_create_on_disk_options *opts) { + struct lock_file alternates_lock = LOCK_INIT; struct strbuf path = STRBUF_INIT; + struct strset seen = STRSET_INIT; + struct strbuf line = STRBUF_INIT; + int ret; safe_create_dir(source->odb->repo, source->path, 1); @@ -64,8 +70,74 @@ static int odb_source_files_create_on_disk(struct odb_source *source) strbuf_addf(&path, "%s/info", source->path); safe_create_dir(source->odb->repo, path.buf, 1); + if (opts->alternates && opts->alternates->nr) { + FILE *alternates, *orig; + + strbuf_reset(&path); + strbuf_addf(&path, "%s/info/alternates", source->path); + + repo_hold_lock_file_for_update(source->odb->repo, &alternates_lock, + path.buf, LOCK_DIE_ON_ERROR); + + alternates = fdopen_lock_file(&alternates_lock, "w"); + if (!alternates) { + ret = error_errno(_("unable to fdopen alternates lockfile")); + goto out; + } + + /* + * The alternates file may already exist, e.g. when it has been + * seeded from a template directory. Read any preexisting + * entries so that we don't end up writing duplicates. + */ + orig = fopen(path.buf, "r"); + if (orig) { + while (strbuf_getline(&line, orig) != EOF) { + strset_add(&seen, line.buf); + fprintf(alternates, "%s\n", line.buf); + } + + if (ferror(orig)) { + ret = error_errno(_("unable to read alternates file")); + fclose(orig); + goto out; + } + + fclose(orig); + } else if (errno != ENOENT) { + ret = error_errno(_("unable to read alternates file")); + goto out; + } + + for (size_t i = 0; i < opts->alternates->nr; i++) { + const char *alternate = opts->alternates->v[i]; + if (!strset_add(&seen, alternate)) + continue; + fprintf(alternates, "%s\n", alternate); + } + + if (ferror(alternates)) { + ret = error_errno(_("unable to write alternates file")); + goto out; + } + + if (commit_lock_file(&alternates_lock)) { + ret = error_errno(_("unable to commit alternates file")); + goto out; + } + } + + /* Reprepare the object database to activate alternates. */ + odb_reprepare(source->odb); + + ret = 0; + +out: + rollback_lock_file(&alternates_lock); + strbuf_release(&line); strbuf_release(&path); - return 0; + strset_clear(&seen); + return ret; } static void odb_source_files_prepare(struct odb_source *source, diff --git a/odb/source.h b/odb/source.h index ea8675247e45f7..63f1c0c531f9a2 100644 --- a/odb/source.h +++ b/odb/source.h @@ -36,6 +36,15 @@ struct object_id; struct odb_stream; struct strvec; +struct odb_create_on_disk_options { + /* + * Alternates that shall be written into the newly created object + * database. Whether or not this option can be handled is specific to + * the backend. + */ + const struct strvec *alternates; +}; + /* * The source is the part of the object database that stores the actual * objects. It thus encapsulates the logic to read and write the specific @@ -106,7 +115,8 @@ struct odb_source { * This callback may be NULL in case the source does not need any * on-disk setup. */ - int (*create_on_disk)(struct odb_source *source); + int (*create_on_disk)(struct odb_source *source, + const struct odb_create_on_disk_options *opts); /* * This callback is expected to prepare the source so that it becomes @@ -356,11 +366,12 @@ static inline void odb_source_close(struct odb_source *source) * Create on-disk data structures that are required for this source to operate * correctly. Returns 0 on success, a negative error code otherwise. */ -static inline int odb_source_create_on_disk(struct odb_source *source) +static inline int odb_source_create_on_disk(struct odb_source *source, + const struct odb_create_on_disk_options *opts) { if (!source->create_on_disk) return 0; - return source->create_on_disk(source); + return source->create_on_disk(source, opts); } /* diff --git a/setup.c b/setup.c index e654e27d05114f..426cc7dff8194b 100644 --- a/setup.c +++ b/setup.c @@ -2649,6 +2649,8 @@ static int create_default_files(struct repository *repo, void create_object_database(struct repository *repo) { + struct odb_create_on_disk_options opts = { 0 }; + /* * Create the "objects" directory in the common directory. This is done * so that the repository can be discovered regardless of the backend @@ -2668,7 +2670,7 @@ void create_object_database(struct repository *repo) repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV); - if (odb_source_create_on_disk(repo->objects->sources) < 0) + if (odb_source_create_on_disk(repo->objects->sources, &opts) < 0) die(_("failed creating object database")); } From 1b65c850a5a898d15bd52e2d9d5b53d14fb56fda Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:11 +0200 Subject: [PATCH 011/156] builtin/clone: write alternates via `odb_create_on_disk()` When creating a repository with alternates we first initialize the object database and then write alternates to it in a separate step. This is unfortunate due to a couple of reasons: - It requires us to have a `write_alternates()` callback, which is unfortunate as we never even write alternates to an object database after it has been created. - We're about to make alternates an implementation detail of the object database's backend in a future patch series, so having this callback is suboptimal there. - The backend has more flexibility with how exactly alternates are configured when it itself is in full control over their setup at the time where it creates the object database itself. We have thus introduced the ability to write alternates right at creation time in the preceding commits, and we have unified setup of alternates into a single location. All that's left to do for us now is to wire up alternates as an option for the database creation. Do so. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- builtin/clone.c | 5 +---- setup.c | 9 ++++++--- setup.h | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 2e3473fddfa388..48ac379b1d2f5c 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1368,11 +1368,8 @@ int cmd_clone(int argc, if (option_local > 0 && !is_local) warning(_("--local is ignored")); - create_object_database(the_repository); collect_alternates(&alternates, path, is_local); - - for (size_t i = 0; i < alternates.nr; i++) - odb_add_to_alternates_file(the_repository->objects, alternates.v[i]); + create_object_database(the_repository, &alternates); transport = transport_get(remote, path ? path : remote->url.v[0]); transport_set_verbosity(transport, option_verbosity, option_progress); diff --git a/setup.c b/setup.c index 426cc7dff8194b..cfa286ff5913ff 100644 --- a/setup.c +++ b/setup.c @@ -2647,9 +2647,12 @@ static int create_default_files(struct repository *repo, return reinit; } -void create_object_database(struct repository *repo) +void create_object_database(struct repository *repo, + const struct strvec *alternates) { - struct odb_create_on_disk_options opts = { 0 }; + struct odb_create_on_disk_options opts = { + .alternates = alternates, + }; /* * Create the "objects" directory in the common directory. This is done @@ -2907,7 +2910,7 @@ int init_db(struct repository *repo, if (!(flags & INIT_DB_SKIP_REFDB)) create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET); if (!(flags & INIT_DB_SKIP_ODB)) - create_object_database(repo); + create_object_database(repo, NULL); startup_info->have_repository = 1; diff --git a/setup.h b/setup.h index 570ebcd1506967..34e86dad37278e 100644 --- a/setup.h +++ b/setup.h @@ -277,7 +277,7 @@ void initialize_repository_version(struct repository *repo, enum ref_storage_format ref_storage_format, int reinit); void create_reference_database(struct repository *repo, const char *initial_branch, int quiet); -void create_object_database(struct repository *repo); +void create_object_database(struct repository *repo, const struct strvec *alternates); /* * NOTE NOTE NOTE!! From 9e665c13b7f8a99048982cd9317548109609c991 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Mon, 31 Aug 2026 12:02:12 +0200 Subject: [PATCH 012/156] odb/source: remove the ability to write alternates There are no users of `odb_source_write_alternates()` in our tree anymore. Remove that function and its supporting infrastructure. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- odb.c | 9 -------- odb.h | 7 ------ odb/source-files.c | 54 ------------------------------------------- odb/source-inmemory.c | 7 ------ odb/source-loose.c | 7 ------ odb/source-packed.c | 7 ------ odb/source.h | 26 --------------------- 7 files changed, 117 deletions(-) diff --git a/odb.c b/odb.c index 67d98d64fcfdd9..b531cf8fb31902 100644 --- a/odb.c +++ b/odb.c @@ -239,15 +239,6 @@ static struct odb_source *odb_add_alternate_recursively(struct object_database * return alternate; } -void odb_add_to_alternates_file(struct object_database *odb, - const char *dir) -{ - int ret = odb_source_write_alternate(odb->sources, dir); - if (ret < 0) - die(NULL); - odb_add_alternate_recursively(odb, dir, 0); -} - struct odb_source *odb_add_to_alternates_memory(struct object_database *odb, const char *dir) { diff --git a/odb.h b/odb.h index b9e0db56ec4f5a..2d002461f89928 100644 --- a/odb.h +++ b/odb.h @@ -270,13 +270,6 @@ int odb_mkstemp(struct object_database *odb, */ int odb_has_alternates(struct object_database *odb); -/* - * Add the directory to the on-disk alternates file; the new entry will also - * take effect in the current process. - */ -void odb_add_to_alternates_file(struct object_database *odb, - const char *dir); - /* * Add the directory to the in-memory list of alternate sources (along with any * recursive alternates it points to), but do not modify the on-disk alternates diff --git a/odb/source-files.c b/odb/source-files.c index 8fe65d91f8eb6b..b3f340dff8ae3c 100644 --- a/odb/source-files.c +++ b/odb/source-files.c @@ -306,59 +306,6 @@ static int odb_source_files_read_alternates(struct odb_source *source, return 0; } -static int odb_source_files_write_alternate(struct odb_source *source, - const char *alternate) -{ - struct lock_file lock = LOCK_INIT; - char *path = xstrfmt("%s/%s", source->path, "info/alternates"); - FILE *in, *out; - int found = 0; - int ret; - - repo_hold_lock_file_for_update(source->odb->repo, &lock, path, - LOCK_DIE_ON_ERROR); - out = fdopen_lock_file(&lock, "w"); - if (!out) { - ret = error_errno(_("unable to fdopen alternates lockfile")); - goto out; - } - - in = fopen(path, "r"); - if (in) { - struct strbuf line = STRBUF_INIT; - - while (strbuf_getline(&line, in) != EOF) { - if (!strcmp(alternate, line.buf)) { - found = 1; - break; - } - fprintf_or_die(out, "%s\n", line.buf); - } - - strbuf_release(&line); - fclose(in); - } else if (errno != ENOENT) { - ret = error_errno(_("unable to read alternates file")); - goto out; - } - - if (found) { - rollback_lock_file(&lock); - } else { - fprintf_or_die(out, "%s\n", alternate); - if (commit_lock_file(&lock)) { - ret = error_errno(_("unable to move new alternates file into place")); - goto out; - } - } - - ret = 0; - -out: - free(path); - return ret; -} - static int too_many_loose_objects(struct odb_source_files *files, int limit) { unsigned long loose_count; @@ -842,7 +789,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb, files->base.write_object_stream = odb_source_files_write_object_stream; files->base.begin_transaction = odb_source_files_begin_transaction; files->base.read_alternates = odb_source_files_read_alternates; - files->base.write_alternate = odb_source_files_write_alternate; files->base.optimize = odb_source_files_optimize; files->base.optimize_required = odb_source_files_optimize_required; diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c index 795672adf255c6..b00248dfb2210d 100644 --- a/odb/source-inmemory.c +++ b/odb/source-inmemory.c @@ -326,12 +326,6 @@ static int odb_source_inmemory_read_alternates(struct odb_source *source UNUSED, return 0; } -static int odb_source_inmemory_write_alternate(struct odb_source *source UNUSED, - const char *alternate UNUSED) -{ - return error("in-memory source does not support alternates"); -} - static void odb_source_inmemory_close(struct odb_source *source UNUSED) { } @@ -388,7 +382,6 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb) source->base.freshen_object = odb_source_inmemory_freshen_object; source->base.begin_transaction = odb_source_inmemory_begin_transaction; source->base.read_alternates = odb_source_inmemory_read_alternates; - source->base.write_alternate = odb_source_inmemory_write_alternate; return source; } diff --git a/odb/source-loose.c b/odb/source-loose.c index bb3455dfbd3334..0f9b30bac176f1 100644 --- a/odb/source-loose.c +++ b/odb/source-loose.c @@ -982,12 +982,6 @@ static int odb_source_loose_read_alternates(struct odb_source *source UNUSED, return 0; } -static int odb_source_loose_write_alternate(struct odb_source *source UNUSED, - const char *alternate UNUSED) -{ - return error("loose source does not support alternates"); -} - static void odb_source_loose_clear_cache(struct odb_source_loose *loose) { oidtree_clear(loose->cache); @@ -1053,7 +1047,6 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb, loose->base.write_object_stream = odb_source_loose_write_object_stream; loose->base.begin_transaction = odb_source_loose_begin_transaction; loose->base.read_alternates = odb_source_loose_read_alternates; - loose->base.write_alternate = odb_source_loose_write_alternate; if (!is_absolute_path(loose->base.path)) chdir_notify_register(NULL, odb_source_loose_reparent, loose); diff --git a/odb/source-packed.c b/odb/source-packed.c index 630d9555856d7c..c2d253759c97d1 100644 --- a/odb/source-packed.c +++ b/odb/source-packed.c @@ -628,12 +628,6 @@ static int odb_source_packed_read_alternates(struct odb_source *source UNUSED, return 0; } -static int odb_source_packed_write_alternate(struct odb_source *source UNUSED, - const char *alternate UNUSED) -{ - return error("packed backend cannot write alternates"); -} - void (*report_garbage)(unsigned seen_bits, const char *path); static void report_helper(const struct string_list *list, @@ -849,7 +843,6 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb, packed->base.write_object_stream = odb_source_packed_write_object_stream; packed->base.begin_transaction = odb_source_packed_begin_transaction; packed->base.read_alternates = odb_source_packed_read_alternates; - packed->base.write_alternate = odb_source_packed_write_alternate; if (!is_absolute_path(path)) chdir_notify_register(NULL, odb_source_packed_reparent, packed); diff --git a/odb/source.h b/odb/source.h index 63f1c0c531f9a2..693a9fc6049acf 100644 --- a/odb/source.h +++ b/odb/source.h @@ -286,19 +286,6 @@ struct odb_source { int (*read_alternates)(struct odb_source *source, struct strvec *out); - /* - * This callback is expected to persist the singular alternate passed - * to it into its list of alternates. Any pre-existing alternates are - * expected to remain active. Subsequent calls to `read_alternates` are - * thus expected to yield the pre-existing list of alternates plus the - * newly added alternate appended to its end. - * - * The callback is expected to return 0 on success, a negative error - * code otherwise. - */ - int (*write_alternate)(struct odb_source *source, - const char *alternate); - /* * This callback is expected to optimize the object database source. * Returns 0 on success, a negative error code otherwise. @@ -518,19 +505,6 @@ static inline int odb_source_read_alternates(struct odb_source *source, return source->read_alternates(source, out); } -/* - * Write and persist a new alternate object database source for the given - * source. Any preexisting alternates are expected to stay valid, and the new - * alternate shall be appended to the end of the list. - * - * Returns 0 on success, a negative error code otherwise. - */ -static inline int odb_source_write_alternate(struct odb_source *source, - const char *alternate) -{ - return source->write_alternate(source, alternate); -} - /* * Create a new transaction that can be used to write objects into a temporary * staging area. The objects will only be persisted when the transaction is From 189149575d5d87708e92edc0cb626f1e0cecb269 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 19 Dec 2019 00:52:53 -0500 Subject: [PATCH 013/156] t5300: use test_config We currently leave pack.packSizeLimit set after these tests, making it more difficult to add new tests. Signed-off-by: Jeff King --- t/t5300-pack-object.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index aac139e6a096eb..a03e007c684b2b 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -509,7 +509,7 @@ test_expect_success 'cleanup for --strict and --fsck-objects downgrading fsck ms ' test_expect_success 'honor pack.packSizeLimit' ' - git config pack.packSizeLimit 3m && + test_config pack.packSizeLimit 3m && packname_10=$(git pack-objects test-10 Date: Mon, 2 Dec 2024 00:57:18 -0500 Subject: [PATCH 014/156] t5300: refactor check_deltas This avoids having to say "--progress" each time. --- t/t5300-pack-object.sh | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index a03e007c684b2b..c6836fe1568799 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -161,27 +161,27 @@ test_expect_success 'pack-object -# e.g.: check_deltas stderr -gt 0 +# usage: check_deltas +# e.g.: check_deltas -gt 0 --window=10 check_deltas() { - deltas=$(sed -n 's/Total [0-9][0-9]* (delta \([0-9][0-9]*\)).*/\1/p' "$1") && - shift && - if ! test "$deltas" "$@" + local op="$1"; shift + local nr="$1"; shift + git pack-objects --progress "$@" 2>stderr && + deltas=$(perl -lne '/delta (\d+)/ and print $1' stderr) && + deltas=$(sed -n 's/Total [0-9][0-9]* (delta \([0-9][0-9]*\)).*/\1/p' stderr) && + if ! test "$deltas" "$op" "$nr" then - echo >&2 "unexpected number of deltas (compared $delta $*)" + echo >&2 "unexpected number of deltas (compared $delta $op $nr)" return 1 fi } test_expect_success 'pack without delta' ' - packname_1=$(git pack-objects --progress --window=0 test-1 \ - stderr) && - check_deltas stderr = 0 + packname_1=$(check_deltas = 0 --window=0 test-1 stderr && - check_deltas stderr = 0 + check_deltas = 0 --window=-1 neg-window stderr) && - check_deltas stderr -gt 0 + packname_2=$(check_deltas -gt 0 test-2 stderr) && - check_deltas stderr -gt 0 + packname_3=$(check_deltas -gt 0 --delta-base-offset test-3 Date: Mon, 2 Dec 2024 00:57:45 -0500 Subject: [PATCH 015/156] t5300: pass --no-reuse-delta for check_deltas --- t/t5300-pack-object.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh index c6836fe1568799..19a2f55df7c2e0 100755 --- a/t/t5300-pack-object.sh +++ b/t/t5300-pack-object.sh @@ -166,7 +166,7 @@ test_expect_success 'pack-object stderr && + git pack-objects --progress --no-reuse-delta "$@" 2>stderr && deltas=$(perl -lne '/delta (\d+)/ and print $1' stderr) && deltas=$(sed -n 's/Total [0-9][0-9]* (delta \([0-9][0-9]*\)).*/\1/p' stderr) && if ! test "$deltas" "$op" "$nr" From b5856ffc05d5357f3070907bd7f7b001d39540f9 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 2 Dec 2019 07:11:23 -0800 Subject: [PATCH 016/156] pack-objects: make try_delta() result an enum We already have a tri-state return; in addition to 0 for "yes, we found a better delta" and 1 for "no we did not", we can return -1 to mean "no, and do not bother looking further in the window". Let's turn that into an enum. That makes the existing code a bit more obvious, and will also let us add further return cases without sacrificing readability. Signed-off-by: Jeff King --- builtin/pack-objects.c | 45 ++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 708b719f403b96..e44bb3e006f46f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -2805,8 +2805,19 @@ size_t oe_get_size_slow(struct packing_data *pack, return size; } -static int try_delta(struct unpacked *trg, struct unpacked *src, - unsigned max_depth, size_t *mem_usage) +enum try_delta_result { + /* do not bother looking further in the window */ + TRY_DELTA_STOP, + /* we found a better delta */ + TRY_DELTA_FOUND, + /* no better delta found */ + TRY_DELTA_NONE, +}; + +static enum try_delta_result try_delta(struct unpacked *trg, + struct unpacked *src, + unsigned max_depth, + size_t *mem_usage) { struct object_entry *trg_entry = trg->entry; struct object_entry *src_entry = src->entry; @@ -2818,7 +2829,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, /* Don't bother doing diffs between different types */ if (oe_type(trg_entry) != oe_type(src_entry)) - return -1; + return TRY_DELTA_STOP; /* * We do not bother to try a delta that we discarded on an @@ -2833,11 +2844,11 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, !src_entry->preferred_base && trg_entry->in_pack_type != OBJ_REF_DELTA && trg_entry->in_pack_type != OBJ_OFS_DELTA) - return 0; + return TRY_DELTA_NONE; /* Let's not bust the allowed depth. */ if (src->depth >= max_depth) - return 0; + return TRY_DELTA_NONE; /* Now some size filtering heuristics. */ trg_size = SIZE(trg_entry); @@ -2851,16 +2862,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, max_size = (uint64_t)max_size * (max_depth - src->depth) / (max_depth - ref_depth + 1); if (max_size == 0) - return 0; + return TRY_DELTA_NONE; src_size = SIZE(src_entry); sizediff = src_size < trg_size ? trg_size - src_size : 0; if (sizediff >= max_size) - return 0; + return TRY_DELTA_NONE; if (trg_size < src_size / 32) - return 0; + return TRY_DELTA_NONE; if (!in_same_island(&trg->entry->idx.oid, &src->entry->idx.oid)) - return 0; + return TRY_DELTA_NONE; /* Load data if not already done */ if (!trg->data) { @@ -2900,7 +2911,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, * them if they can't be read, in case the * pack could be created nevertheless. */ - return 0; + return TRY_DELTA_NONE; } die(_("object %s cannot be read"), oid_to_hex(&src_entry->idx.oid)); @@ -2917,21 +2928,21 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, static int warned = 0; if (!warned++) warning(_("suboptimal pack - out of memory")); - return 0; + return TRY_DELTA_NONE; } *mem_usage += sizeof_delta_index(src->index); } delta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size); if (!delta_buf) - return 0; + return TRY_DELTA_NONE; if (DELTA(trg_entry)) { /* Prefer only shallower same-sized deltas. */ if (delta_size == DELTA_SIZE(trg_entry) && src->depth + 1 >= trg->depth) { free(delta_buf); - return 0; + return TRY_DELTA_NONE; } } @@ -2959,7 +2970,7 @@ static int try_delta(struct unpacked *trg, struct unpacked *src, SET_DELTA_SIZE(trg_entry, delta_size); trg->depth = src->depth + 1; - return 1; + return TRY_DELTA_FOUND; } static unsigned int check_delta_limit(struct object_entry *me, unsigned int n) @@ -3047,7 +3058,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, j = window; while (--j > 0) { - int ret; + enum try_delta_result ret; uint32_t other_idx = idx + j; struct unpacked *m; if (other_idx >= window) @@ -3056,9 +3067,9 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, if (!m->entry) break; ret = try_delta(n, m, max_depth, &mem_usage); - if (ret < 0) + if (ret == TRY_DELTA_STOP) break; - else if (ret > 0) + else if (ret == TRY_DELTA_FOUND) best_base = other_idx; } From ce4967f4e3768d2c21a6f9943bc46f479af5b6a2 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 5 Jun 2013 16:54:36 -0400 Subject: [PATCH 017/156] upload-archive: restrict remote objects with reachability check When serving a remote request, git-upload-archive tries to restrict access to unreachable objects, which matches the behavior of upload-pack. However, we did so by restricting the requested tree to "[:]", because it is fast. That covers the common cases, but does not allow requesting items by a specific sha1 (either a tree or a commit sha1). Instead, let's do the correct-but-slower method of actually walking back from the tips to see if the requested object is reachable. The performance impact of this is roughly: 1. For a recent commit, the speed is about the same (we traverse in reverse chronological order, so we see it almost immediately). 2. For an older commit, even one pointed at directly by a ref (e.g., an old tag), we are slower, because we traverse from the more recent tips. We are bounded in this case by the time to look at all commits (i.e., "time git rev-list --all"). 3. When we see "$ref:$path", we typically perform much worse, because our traversal looks at all commits first, followed by all trees. 4. The worst case (which we hit for an unreachable object) is equivalent to "time rev-list --objects --all", which is about the same amount of time pack-objects spends preparing a full clone (which can be in the tens of seconds for a large repository). The implementation is a fairly straightforward application of the traverse_commit_list function. Using the mark_objects_reachable function would seem more appropriate, but it has no mechanism for looking for a specific object, which lets us end the traversal early in common cases. Signed-off-by: Jeff King --- archive.c | 71 +++++++++++++++++++++----- t/meson.build | 1 + t/t5005-archive-resolution.sh | 93 +++++++++++++++++++++++++++++++++++ 3 files changed, 154 insertions(+), 11 deletions(-) create mode 100755 t/t5005-archive-resolution.sh diff --git a/archive.c b/archive.c index 59790be98697c6..a6711e5e43b43f 100644 --- a/archive.c +++ b/archive.c @@ -23,6 +23,9 @@ #include "parse-options.h" #include "unpack-trees.h" #include "quote.h" +#include "diff.h" +#include "revision.h" +#include "list-objects.h" static char const * const archive_usage[] = { N_("git archive [] [...]"), @@ -482,6 +485,59 @@ static void parse_pathspec_arg(const char **pathspec, } } +struct reachable_object_data { + struct rev_info revs; + struct object *obj; +}; + +static void check_object(struct object *obj, const char *name UNUSED, void *vdata) +{ + struct reachable_object_data *data = vdata; + /* + * We found it; the caller will take care of marking it SEEN, + * but we can end the traversal early. + */ + if (obj == data->obj) { + commit_list_free(data->revs.commits); + data->revs.commits = NULL; + + object_array_clear(&data->revs.pending); + } +} + +static void check_commit(struct commit *commit, void *vdata) +{ + check_object(&commit->object, NULL, vdata); +} + +static int object_is_reachable(struct object_id *oid) +{ + static const char *argv[] = { + "rev-list", + "--objects", + "--all", + NULL + }; + struct reachable_object_data data; + + data.obj = parse_object(the_repository, oid); + if (!data.obj) + return 0; + + save_commit_buffer = 0; + repo_init_revisions(the_repository, &data.revs, NULL); + setup_revisions(ARRAY_SIZE(argv) - 1, argv, &data.revs, NULL); + if (prepare_revision_walk(&data.revs)) { + release_revisions(&data.revs); + return 0; + } + + traverse_commit_list(&data.revs, check_commit, check_object, &data); + release_revisions(&data.revs); + + return data.obj->flags & SEEN; +} + static void parse_treeish_arg(const char **argv, struct archiver_args *ar_args, int remote) { @@ -493,21 +549,14 @@ static void parse_treeish_arg(const char **argv, struct object_id oid; char *ref = NULL; - /* Remotes are only allowed to fetch actual refs */ - if (remote && !remote_allow_unreachable) { - const char *colon = strchrnul(name, ':'); - int refnamelen = colon - name; - - if (!repo_dwim_ref(the_repository, name, refnamelen, &oid, &ref, 0)) - die(_("no such ref: %.*s"), refnamelen, name); - } else { - repo_dwim_ref(the_repository, name, strlen(name), &oid, &ref, - 0); - } + repo_dwim_ref(the_repository, name, strlen(name), &oid, &ref, 0); if (repo_get_oid(the_repository, name, &oid)) die(_("not a valid object name: %s"), name); + if (remote && !object_is_reachable(&oid)) + die("Not a valid object name"); + commit = lookup_commit_reference_gently(ar_args->repo, &oid, 1); if (commit) { commit_oid = &commit->object.oid; diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..2a6a7e3d90db33 100644 --- a/t/meson.build +++ b/t/meson.build @@ -600,6 +600,7 @@ integration_tests = [ 't5002-archive-attr-pattern.sh', 't5003-archive-zip.sh', 't5004-archive-corner-cases.sh', + 't5005-archive-resolution.sh', 't5100-mailinfo.sh', 't5150-request-pull.sh', 't5200-update-server-info.sh', diff --git a/t/t5005-archive-resolution.sh b/t/t5005-archive-resolution.sh new file mode 100755 index 00000000000000..c603b9d9b833a0 --- /dev/null +++ b/t/t5005-archive-resolution.sh @@ -0,0 +1,93 @@ +#!/bin/sh + +test_description='test object resolution methods for local and remote archive' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +. ./test-lib.sh + +test_expect_success 'setup' ' + echo one >one && git add one && git commit -m one && + sha1_referenced=`git rev-parse HEAD` && + git tag tagged && + echo two >two && git add two && git commit -m two && + sha1_unreferenced=`git rev-parse HEAD` && + git reset --hard HEAD^ && + echo three >three && git add three && git commit -m three && + git tag tagged-tree HEAD^{tree} && + git reset --hard HEAD^ && + mkdir subdir && + echo four >subdir/four && git add subdir && git commit -m four && + sha1_subtree=`git rev-parse HEAD:subdir` +' + +# check that archiving $what from $where produces expected paths +check() { + desc=$1; shift; # human-readable description + where=$1; shift; # local|remote + what=$1; shift; # the commit/tree id + expect="$*"; # expected paths or "deny" + + cmd="git archive --format=tar -o result.tar" + test "$where" = "remote" && cmd="$cmd --remote=." + cmd="$cmd $what" + + if test "$expect" = "deny"; then + test_expect_success "archive $desc ($where, should deny)" " + test_must_fail $cmd + " + else + test_expect_success "archive $desc ($where, should work)" ' + '"$cmd"' && + for i in '"$expect"'; do + echo "$i:`basename $i`" || return 1 + done >expect && + rm -rf result && + mkdir result && + (cd result && + tar xf ../result.tar && + for i in `find * -type f -print`; do + echo "$i:`cat $i`" || exit 1 + done >../actual + ) && + test_cmp expect actual + ' + fi +} + +check 'ref' local main one subdir/four +check 'ref' remote main one subdir/four + +check 'relative ref' local main^ one +check 'relative ref' remote main^ one + +check 'reachable sha1' local $sha1_referenced one +check 'reachable sha1' remote $sha1_referenced one + +check 'unreachable sha1' local $sha1_unreferenced one two +check 'unreachable sha1' remote $sha1_unreferenced deny + +check 'reachable reflog' local main@{0} one subdir/four +check 'reachable reflog' remote main@{0} one subdir/four + +check 'unreachable reflog' local main@{4} one two +check 'unreachable reflog' remote main@{4} deny + +check 'tree via ref^{tree}' local main^{tree} one subdir/four +check 'tree via ref^{tree}' remote main^{tree} one subdir/four + +check 'tree via ref:' local main: one subdir/four +check 'tree via ref:' remote main: one subdir/four + +check 'subtree via ref:sub' local main:subdir four +check 'subtree via ref:sub' remote main:subdir four + +check 'subtree via sha1' local $sha1_subtree four +check 'subtree via sha1' remote $sha1_subtree four + +check 'tagged commit' local tagged one +check 'tagged commit' remote tagged one + +check 'tagged tree' local tagged-tree one three +check 'tagged tree' remote tagged-tree one three + +test_done From 3cf7c2f2a8e7da783f57fe9f9ef9411e12890054 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 2 Dec 2019 07:11:23 -0800 Subject: [PATCH 018/156] pack-objects: annotate fast/slow returns from try_delta() Our try_delta() function may return TRY_DELTA_NONE, indicating that we did not find a better delta. Let's subdivide that into the case where we found that out quickly (via heuristics) versus slowly (by actually computing the delta). That will open up options for dynamically adjusting the window size based on how much work we're doing. Signed-off-by: Jeff King --- builtin/pack-objects.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index e44bb3e006f46f..31069830e48581 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -2810,8 +2810,10 @@ enum try_delta_result { TRY_DELTA_STOP, /* we found a better delta */ TRY_DELTA_FOUND, - /* no better delta found */ - TRY_DELTA_NONE, + /* heuristics quickly told us we would not find a better delta */ + TRY_DELTA_NONE_FAST, + /* we computed a delta, but it was not an improvement */ + TRY_DELTA_NONE_SLOW, }; static enum try_delta_result try_delta(struct unpacked *trg, @@ -2844,11 +2846,11 @@ static enum try_delta_result try_delta(struct unpacked *trg, !src_entry->preferred_base && trg_entry->in_pack_type != OBJ_REF_DELTA && trg_entry->in_pack_type != OBJ_OFS_DELTA) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; /* Let's not bust the allowed depth. */ if (src->depth >= max_depth) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; /* Now some size filtering heuristics. */ trg_size = SIZE(trg_entry); @@ -2862,16 +2864,16 @@ static enum try_delta_result try_delta(struct unpacked *trg, max_size = (uint64_t)max_size * (max_depth - src->depth) / (max_depth - ref_depth + 1); if (max_size == 0) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; src_size = SIZE(src_entry); sizediff = src_size < trg_size ? trg_size - src_size : 0; if (sizediff >= max_size) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; if (trg_size < src_size / 32) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; if (!in_same_island(&trg->entry->idx.oid, &src->entry->idx.oid)) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; /* Load data if not already done */ if (!trg->data) { @@ -2911,7 +2913,7 @@ static enum try_delta_result try_delta(struct unpacked *trg, * them if they can't be read, in case the * pack could be created nevertheless. */ - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; } die(_("object %s cannot be read"), oid_to_hex(&src_entry->idx.oid)); @@ -2928,21 +2930,21 @@ static enum try_delta_result try_delta(struct unpacked *trg, static int warned = 0; if (!warned++) warning(_("suboptimal pack - out of memory")); - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_FAST; } *mem_usage += sizeof_delta_index(src->index); } delta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size); if (!delta_buf) - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_SLOW; if (DELTA(trg_entry)) { /* Prefer only shallower same-sized deltas. */ if (delta_size == DELTA_SIZE(trg_entry) && src->depth + 1 >= trg->depth) { free(delta_buf); - return TRY_DELTA_NONE; + return TRY_DELTA_NONE_SLOW; } } From 58464fb590211bc137ae535a34713e788f767fa3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 14 May 2021 02:15:56 -0400 Subject: [PATCH 019/156] pack-bitmap: drop type parameter from find_tip_objects() There's no reason we need to reduce the set of objects, and this makes it more generic. Signed-off-by: Jeff King --- pack-bitmap.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index e0fb57d3321889..14decd634cb3b4 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1791,8 +1791,7 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git, } static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, - enum object_type type) + struct object_list *tip_objects) { struct bitmap *result = bitmap_new(); struct object_list *p; @@ -1800,9 +1799,6 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, for (p = tip_objects; p; p = p->next) { int pos; - if (p->item->type != type) - continue; - pos = bitmap_position(bitmap_git, &p->item->oid); if (pos < 0) continue; @@ -1829,7 +1825,7 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, * objects which the other side specifically asked for, * so we must match that behavior. */ - tips = find_tip_objects(bitmap_git, tip_objects, type); + tips = find_tip_objects(bitmap_git, tip_objects); /* * We can use the type-level bitmap for 'type' to work in whole @@ -1921,7 +1917,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, eword_t mask; uint32_t i; - tips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB); + tips = find_tip_objects(bitmap_git, tip_objects); for (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB); i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it); From d1109ec357270d4b3dc78658748a91c68d5643a8 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 14 May 2021 02:26:53 -0400 Subject: [PATCH 020/156] pull out dispatch function Signed-off-by: Jeff King --- pack-bitmap.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 14decd634cb3b4..38dcc2918e115b 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1984,10 +1984,10 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git, filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB); } -static int filter_bitmap(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, - struct bitmap *to_filter, - const struct list_objects_filter_options *filter) +static int filter_bitmap_dispatch(struct bitmap_index *bitmap_git, + struct object_list *tip_objects, + struct bitmap *to_filter, + const struct list_objects_filter_options *filter) { if (!filter || filter->choice == LOFC_DISABLED) return 0; @@ -2027,8 +2027,8 @@ static int filter_bitmap(struct bitmap_index *bitmap_git, if (filter->choice == LOFC_COMBINE) { int i; for (i = 0; i < filter->sub_nr; i++) { - if (filter_bitmap(bitmap_git, tip_objects, to_filter, - &filter->sub[i]) < 0) + if (filter_bitmap_dispatch(bitmap_git, tip_objects, to_filter, + &filter->sub[i]) < 0) return -1; } return 0; @@ -2038,6 +2038,21 @@ static int filter_bitmap(struct bitmap_index *bitmap_git, return -1; } +static int filter_bitmap(struct bitmap_index *bitmap_git, + struct object_list *tip_objects, + struct bitmap *to_filter, + const struct list_objects_filter_options *filter) +{ + int r; + + if (!filter || filter->choice == LOFC_DISABLED) + return 0; + + r = filter_bitmap_dispatch(bitmap_git, tip_objects, to_filter, filter); + + return r; +} + bool can_filter_bitmap(const struct list_objects_filter_options *filter) { return !filter_bitmap(NULL, NULL, NULL, filter); From 2ec2b27207bf70495d0543bd9e144269e9114964 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 5 Dec 2025 20:01:27 -0500 Subject: [PATCH 021/156] complain about argv[1] because we have KEEP_ARGV0 --- t/unit-tests/unit-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/unit-tests/unit-test.c b/t/unit-tests/unit-test.c index 752fb38fb324a1..8eda87856ca012 100644 --- a/t/unit-tests/unit-test.c +++ b/t/unit-tests/unit-test.c @@ -48,7 +48,7 @@ int cmd_main(int argc, const char **argv) argc = parse_options(argc, argv, NULL, options, unit_test_usage, PARSE_OPT_KEEP_ARGV0); if (argc > 1) - usagef(_("extra command line parameter '%s'"), argv[0]); + usagef(_("extra command line parameter '%s'"), argv[1]); strvec_push(&args, argv[0]); strvec_push(&args, "-t"); From 5ce7e4a44ab7d25154191fe7a72b9b068b093c68 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 14 May 2021 02:27:11 -0400 Subject: [PATCH 022/156] actual fix Signed-off-by: Jeff King --- pack-bitmap.c | 13 +++++++++++++ t/lib-bitmap.sh | 10 ++++++++++ 2 files changed, 23 insertions(+) diff --git a/pack-bitmap.c b/pack-bitmap.c index 38dcc2918e115b..3f2646de6580d2 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -2049,6 +2049,19 @@ static int filter_bitmap(struct bitmap_index *bitmap_git, return 0; r = filter_bitmap_dispatch(bitmap_git, tip_objects, to_filter, filter); + if (!r && to_filter) { + /* + * Add back in any bits for objects that were explicitly asked + * for, which matches the behavior of non-bitmap filters. Note + * that it isn't sufficient to just avoid filtering them in the + * first place; we must also add back in any removed by the + * AND-NOT of the "have" side while preparing the bitmap, as + * that is not aware of the filtering process. + */ + struct bitmap *tips = find_tip_objects(bitmap_git, tip_objects); + bitmap_or(to_filter, tips); + bitmap_free(tips); + } return r; } diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index 2de6646128d87b..4b1f6d2d8307e9 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -217,6 +217,16 @@ basic_bitmap_tests () { ) ' + test_expect_success 'follow-up fetch from bitmapped repository' ' + test_config uploadpack.allowfilter true && + ( + cd partial-clone.git && + blob=$(git rev-parse HEAD:file-1.t) && + git fetch origin "$blob" && + git cat-file -t "$blob" + ) + ' + test_expect_success 'setup further non-bitmapped commits' ' test_commit_bulk --id=further 10 ' From 8a1c40ea61e5d5bc05cd72676c7b7945f2443a21 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Tue, 1 Sep 2026 10:16:11 +0200 Subject: [PATCH 023/156] ci: fix missing Ruby dependency in "documentation" job Our "documentation" job has recently stopped working with the following error: + sudo gem install --version 1.5.8 asciidoctor + gem install --version 1.5.8 asciidoctor ./ci/install-dependencies.sh: 23: gem: not found The root cause of this is that we never explicitly install Ruby, and consequently gem(1) isn't explicitly pulled in, either. This used to work alright because we transitively pulled in Ruby via asciidoc. But due to an update it seems that we stopped pulling in the transitive dependency, and consequently we don't have gem(1) available anymore. Fix this by explicitly installing Ruby. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 10c3530d1aacdd..502e518077326f 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -164,7 +164,7 @@ sparse) libexpat-dev gettext zlib1g-dev sparse ;; Documentation) - sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make + sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make ruby test -n "$ALREADY_HAVE_ASCIIDOCTOR" || sudo gem install --version 1.5.8 asciidoctor From 0a58416343ab03c4c07b7b252c3c5d441362c517 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 3 Jun 2026 23:09:33 -0400 Subject: [PATCH 024/156] test-lib: drop test_external_has_tap assignment This hasn't been used since test_external was dropped. --- t/test-lib.sh | 2 -- 1 file changed, 2 deletions(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 1f0505e412d8ff..4c38af81777dab 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -770,8 +770,6 @@ test_success=0 test_missing_prereq= -test_external_has_tap=0 - die () { code=$? # This is responsible for running the atexit commands even when a From 9b38ba6bcafc94bd757afe5cae1e5280df61d7f0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 25 Mar 2025 01:17:41 -0400 Subject: [PATCH 025/156] ALLOC_GROW: always grow on NULL This should always grow anyway, assuming we want at least one item, because a NULL field will always have an allocation of 0. But making it explicit seems to unconfuse Coverity that the result of functions like string_list_append() will always be non-NULL, dropping about 100 false positives. --- git-compat-util.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/git-compat-util.h b/git-compat-util.h index a0f901ce793ed6..05c25a81439427 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -885,7 +885,7 @@ static inline void move_array(void *dst, const void *src, size_t n, size_t size) */ #define ALLOC_GROW(x, nr, alloc) \ do { \ - if ((nr) > alloc) { \ + if (!(x) || (nr) > alloc) { \ if (alloc_nr(alloc) < (nr)) \ alloc = (nr); \ else \ From 8caad7c541c788060fe409805a3566138708959a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 5 Jun 2013 17:57:02 -0400 Subject: [PATCH 026/156] list-objects: optimize "revs->blob_objects = 0" case If we are traversing trees during a "--objects" traversal, we may skip blobs if the "blob_objects" field of rev_info is not set. But we do so as the first thing in process_blob(), only after we have actually created the "struct blob" object, incurring a hash lookup. We can optimize out this no-op call completely. This does not actually affect any current code, as all of the current traversals always set blob_objects when looking at objects, anyway. Signed-off-by: Jeff King --- archive.c | 1 + list-objects.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/archive.c b/archive.c index a6711e5e43b43f..ca0626caf1b6d6 100644 --- a/archive.c +++ b/archive.c @@ -527,6 +527,7 @@ static int object_is_reachable(struct object_id *oid) save_commit_buffer = 0; repo_init_revisions(the_repository, &data.revs, NULL); setup_revisions(ARRAY_SIZE(argv) - 1, argv, &data.revs, NULL); + data.revs.blob_objects = 0; if (prepare_revision_walk(&data.revs)) { release_revisions(&data.revs); return 0; diff --git a/list-objects.c b/list-objects.c index 724d723c484de4..f33516a30f452c 100644 --- a/list-objects.c +++ b/list-objects.c @@ -133,7 +133,7 @@ static void process_tree_contents(struct traversal_context *ctx, } else if (S_ISGITLINK(entry.mode)) ; /* ignore gitlink */ - else { + else if (ctx->revs->blob_objects) { struct blob *b = lookup_blob(ctx->revs->repo, &entry.oid); if (!b) { die(_("entry '%s' in tree %s has blob mode, " From 807e12e26fc746652af1e0c2214d0dbe9792eb18 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 14 May 2021 02:28:37 -0400 Subject: [PATCH 027/156] drop redundant Signed-off-by: Jeff King --- pack-bitmap.c | 64 ++++++++++++--------------------------------------- 1 file changed, 15 insertions(+), 49 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 3f2646de6580d2..f0e5e90915714b 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1810,23 +1810,14 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, } static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter, enum object_type type) { struct eindex *eindex = &bitmap_git->ext_index; - struct bitmap *tips; struct ewah_or_iterator it; eword_t mask; uint32_t i; - /* - * The non-bitmap version of this filter never removes - * objects which the other side specifically asked for, - * so we must match that behavior. - */ - tips = find_tip_objects(bitmap_git, tip_objects); - /* * We can use the type-level bitmap for 'type' to work in whole * words for the objects that are actually in the bitmapped @@ -1835,8 +1826,6 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, for (i = 0, init_type_iterator(&it, bitmap_git, type); i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it); i++) { - if (i < tips->word_alloc) - mask &= ~tips->words[i]; to_filter->words[i] &= ~mask; } @@ -1848,21 +1837,17 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; i++) { size_t pos = st_add(i, bitmap_num_objects_total(bitmap_git)); if (eindex->objects[i]->type == type && - bitmap_get(to_filter, pos) && - !bitmap_get(tips, pos)) + bitmap_get(to_filter, pos)) bitmap_unset(to_filter, pos); } ewah_or_iterator_release(&it); - bitmap_free(tips); } static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter) { - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, - OBJ_BLOB); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_BLOB); } static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, @@ -1907,18 +1892,14 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, } static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter, unsigned long limit) { struct eindex *eindex = &bitmap_git->ext_index; - struct bitmap *tips; struct ewah_or_iterator it; eword_t mask; uint32_t i; - tips = find_tip_objects(bitmap_git, tip_objects); - for (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB); i < to_filter->word_alloc && ewah_or_iterator_next(&mask, &it); i++) { @@ -1933,8 +1914,7 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, offset += ewah_bit_ctz64(word >> offset); pos = i * BITS_IN_EWORD + offset; - if (!bitmap_get(tips, pos) && - get_size_by_pos(bitmap_git, pos) >= limit) + if (get_size_by_pos(bitmap_git, pos) >= limit) bitmap_unset(to_filter, pos); } } @@ -1943,31 +1923,25 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, size_t pos = st_add(i, bitmap_num_objects(bitmap_git)); if (eindex->objects[i]->type == OBJ_BLOB && bitmap_get(to_filter, pos) && - !bitmap_get(tips, pos) && get_size_by_pos(bitmap_git, pos) >= limit) bitmap_unset(to_filter, pos); } ewah_or_iterator_release(&it); - bitmap_free(tips); } static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter, unsigned long limit) { if (limit) BUG("filter_bitmap_tree_depth given non-zero limit"); - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, - OBJ_TREE); - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, - OBJ_BLOB); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_TREE); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_BLOB); } static void filter_bitmap_object_type(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter, enum object_type object_type) { @@ -1975,34 +1949,28 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git, BUG("filter_bitmap_object_type given invalid object"); if (object_type != OBJ_TAG) - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TAG); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_TAG); if (object_type != OBJ_COMMIT) - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_COMMIT); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_COMMIT); if (object_type != OBJ_TREE) - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_TREE); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_TREE); if (object_type != OBJ_BLOB) - filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB); + filter_bitmap_exclude_type(bitmap_git, to_filter, OBJ_BLOB); } static int filter_bitmap_dispatch(struct bitmap_index *bitmap_git, - struct object_list *tip_objects, struct bitmap *to_filter, const struct list_objects_filter_options *filter) { - if (!filter || filter->choice == LOFC_DISABLED) - return 0; - if (filter->choice == LOFC_BLOB_NONE) { if (bitmap_git) - filter_bitmap_blob_none(bitmap_git, tip_objects, - to_filter); + filter_bitmap_blob_none(bitmap_git, to_filter); return 0; } if (filter->choice == LOFC_BLOB_LIMIT) { if (bitmap_git) - filter_bitmap_blob_limit(bitmap_git, tip_objects, - to_filter, + filter_bitmap_blob_limit(bitmap_git, to_filter, filter->blob_limit_value); return 0; } @@ -2010,16 +1978,14 @@ static int filter_bitmap_dispatch(struct bitmap_index *bitmap_git, if (filter->choice == LOFC_TREE_DEPTH && filter->tree_exclude_depth == 0) { if (bitmap_git) - filter_bitmap_tree_depth(bitmap_git, tip_objects, - to_filter, + filter_bitmap_tree_depth(bitmap_git, to_filter, filter->tree_exclude_depth); return 0; } if (filter->choice == LOFC_OBJECT_TYPE) { if (bitmap_git) - filter_bitmap_object_type(bitmap_git, tip_objects, - to_filter, + filter_bitmap_object_type(bitmap_git, to_filter, filter->object_type); return 0; } @@ -2027,7 +1993,7 @@ static int filter_bitmap_dispatch(struct bitmap_index *bitmap_git, if (filter->choice == LOFC_COMBINE) { int i; for (i = 0; i < filter->sub_nr; i++) { - if (filter_bitmap_dispatch(bitmap_git, tip_objects, to_filter, + if (filter_bitmap_dispatch(bitmap_git, to_filter, &filter->sub[i]) < 0) return -1; } @@ -2048,7 +2014,7 @@ static int filter_bitmap(struct bitmap_index *bitmap_git, if (!filter || filter->choice == LOFC_DISABLED) return 0; - r = filter_bitmap_dispatch(bitmap_git, tip_objects, to_filter, filter); + r = filter_bitmap_dispatch(bitmap_git, to_filter, filter); if (!r && to_filter) { /* * Add back in any bits for objects that were explicitly asked From 2ad844fa311b7d2d2fc538396cbfe83a20667811 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 2 Sep 2026 02:55:45 -0400 Subject: [PATCH 028/156] ci: drop ALREADY_HAVE_ASCIIDOCTOR variable Our install-dependencies script avoids installing asciidoctor if the CI environment claims to have it already. But the only environment which made use of this was Azure Pipelines, which went away in 6081d3898f (ci: retire the Azure Pipelines definition, 2020-04-11). So this conditional was effectively doing nothing. Let's clean it up. --- ci/install-dependencies.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 502e518077326f..2f61fbb07c577d 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -166,7 +166,6 @@ sparse) Documentation) sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make ruby - test -n "$ALREADY_HAVE_ASCIIDOCTOR" || sudo gem install --version 1.5.8 asciidoctor sudo gem install concurrent-ruby ;; From 9e6cb23b3159b918d3a7fe72561a4fa0bd2e7a66 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 5 Dec 2025 20:31:37 -0500 Subject: [PATCH 029/156] hacky unit-tests in test suite This handles --verbose-log. But also means we do things like set up our LSAN_OPTIONS properly. E.g., this leak: diff --git a/t/unit-tests/u-strbuf.c b/t/unit-tests/u-strbuf.c index caa5d78aa3..4d252902ed 100644 --- a/t/unit-tests/u-strbuf.c +++ b/t/unit-tests/u-strbuf.c @@ -84,6 +84,8 @@ static void t_addstr(struct strbuf *buf, const void *data) size_t orig_alloc = buf->alloc; size_t orig_len = buf->len; + write(2, strdup("foo"), 3); + assert_sane_strbuf(buf); strbuf_addstr(buf, text); assert_sane_strbuf(buf); gets a crappy backtrace with the default options. --- t/meson.build | 1 + t/t0011-unit-tests.sh | 18 ++++++++++++++++++ t/test-lib.sh | 9 ++++++--- 3 files changed, 25 insertions(+), 3 deletions(-) create mode 100755 t/t0011-unit-tests.sh diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..05d55b9e7ae01f 100644 --- a/t/meson.build +++ b/t/meson.build @@ -84,6 +84,7 @@ integration_tests = [ 't0008-ignores.sh', 't0009-git-dir-validation.sh', 't0010-racy-git.sh', + 't0011-unit-tests.sh', 't0012-help.sh', 't0013-sha1dc.sh', 't0014-alias.sh', diff --git a/t/t0011-unit-tests.sh b/t/t0011-unit-tests.sh new file mode 100755 index 00000000000000..c292447739e376 --- /dev/null +++ b/t/t0011-unit-tests.sh @@ -0,0 +1,18 @@ +#!/bin/sh + +test_description='run clar unit tests' +test_external=t +. ./test-lib.sh + +# ugh, gross meson vs make difference +if test -d "$GIT_BUILD_DIR/t/unit-tests" +then + UNIT_TESTS="$GIT_BUILD_DIR/t/unit-tests/bin/unit-tests" +else + UNIT_TESTS="$GIT_BUILD_DIR/t/unit-tests" +fi + +"$UNIT_TESTS" ${immediate:+-i} || + error "unit-test binary failed" + +test_done diff --git a/t/test-lib.sh b/t/test-lib.sh index 4c38af81777dab..e5f531545d7d4d 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1273,8 +1273,11 @@ test_done () { # Maybe print SKIP message test -z "$skip_all" || skip_all="# SKIP $skip_all" - case "$test_count" in - 0) + case "$test_external,$test_count" in + t,*) + : nothing + ;; + *,0) say "1..$test_count${skip_all:+ $skip_all}" ;; *) @@ -1530,7 +1533,7 @@ then BAIL_OUT 'You need to build test-tool; Run "make t/helper/test-tool" in the source (toplevel) directory' fi -if test -n "$HARNESS_ACTIVE" +if test -n "$HARNESS_ACTIVE" && test -z "$test_external" then say "TAP version 13" say "pragma +strict" From 70814a346f4baa76e90246634c0e3c9e2f626e35 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 29 Mar 2025 21:35:12 -0400 Subject: [PATCH 030/156] strbuf: add extra explicit NUL termination This drops 162 defects. --- strbuf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/strbuf.c b/strbuf.c index 44955669e8c504..0de5c2db0b0472 100644 --- a/strbuf.c +++ b/strbuf.c @@ -311,6 +311,7 @@ void strbuf_add(struct strbuf *sb, const void *data, size_t len) strbuf_grow(sb, len); memcpy(sb->buf + sb->len, data, len); strbuf_setlen(sb, sb->len + len); + sb->buf[sb->len] = '\0'; } void strbuf_addstrings(struct strbuf *sb, const char *s, size_t n) From 95b9a79eadccef027e3dc0d2a41952a17773a409 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 15 Oct 2013 15:44:24 -0400 Subject: [PATCH 031/156] t5000: drop tag to supposedly unreachable object We try to create an unreachable commit, but test_commit will retain a tag pointing to it. Oops. Signed-off-by: Jeff King --- t/t5000-tar-tree.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/t5000-tar-tree.sh b/t/t5000-tar-tree.sh index 3ad600c7932b68..7c5419fd028754 100755 --- a/t/t5000-tar-tree.sh +++ b/t/t5000-tar-tree.sh @@ -299,6 +299,7 @@ test_expect_success 'git archive --remote outside of a git repo' ' test_expect_success 'clients cannot access unreachable commits' ' test_commit unreachable && + git tag -d unreachable && sha1=$(git rev-parse HEAD) && git reset --hard HEAD^ && git archive $sha1 >remote.tar && From 36ab36dba9e754c44e73f516b64c05a6ec1f3612 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 21 Jun 2016 15:06:14 -0400 Subject: [PATCH 032/156] diff: ignore mode in binary size optimization This makes add/delete work. Signed-off-by: Jeff King --- diff.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/diff.c b/diff.c index 414532d09f5ba3..857b0195b08e66 100644 --- a/diff.c +++ b/diff.c @@ -3944,9 +3944,7 @@ static void builtin_diff(const char *name_a, ( (!textconv_one && diff_filespec_is_binary(o->repo, one)) || (!textconv_two && diff_filespec_is_binary(o->repo, two)) )) { struct strbuf sb = STRBUF_INIT; - if (!one->data && !two->data && - S_ISREG(one->mode) && S_ISREG(two->mode) && - !o->flags.binary) { + if (!one->data && !two->data && !o->flags.binary) { if (oideq(&one->oid, &two->oid)) { if (must_show_header) emit_diff_symbol(o, DIFF_SYMBOL_HEADER, From ec42fed2efa1aea0d5c1053a2edbd239a89da09e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 31 Jan 2014 18:19:33 -0500 Subject: [PATCH 033/156] wip -- better protection against corruption Signed-off-by: Jeff King --- ewah/ewah_bitmap.c | 14 +++++++++++++ ewah/ewok.h | 1 + pack-bitmap.c | 50 +++++++++++++++++++++++++++++++++++++--------- 3 files changed, 56 insertions(+), 9 deletions(-) diff --git a/ewah/ewah_bitmap.c b/ewah/ewah_bitmap.c index 056c410efb7aed..6460614fd34fbc 100644 --- a/ewah/ewah_bitmap.c +++ b/ewah/ewah_bitmap.c @@ -333,6 +333,18 @@ int ewah_iterator_next(eword_t *next, struct ewah_iterator *it) if (it->pointer >= it->buffer_size) return 0; + /* + * If we return more bits than the ewah advertised, then either + * our data bits or the bit_size field was corrupted, and we + * risk a caller overwriting their own buffer (if they used + * bit_size to size their buffer in the first place). + * + * We don't have a good way of returning an error here, so let's + * just die. + */ + if (!it->words_remaining--) + die("ewah bitmap contains more bits than it claims"); + if (it->compressed < it->rl) { it->compressed++; *next = it->b ? (eword_t)(~0) : 0; @@ -361,6 +373,8 @@ void ewah_iterator_init(struct ewah_iterator *it, struct ewah_bitmap *parent) it->buffer_size = parent->buffer_size; it->pointer = 0; + it->words_remaining = (parent->bit_size + 63) / 64; + it->lw = 0; it->rl = 0; it->compressed = 0; diff --git a/ewah/ewok.h b/ewah/ewok.h index c29d3542361c84..7e35b256dc4b3a 100644 --- a/ewah/ewok.h +++ b/ewah/ewok.h @@ -119,6 +119,7 @@ struct ewah_iterator { size_t buffer_size; size_t pointer; + size_t words_remaining; eword_t compressed, literals; eword_t rl, lw; int b; diff --git a/pack-bitmap.c b/pack-bitmap.c index e0fb57d3321889..3f8c4c9523e5c9 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -188,15 +188,6 @@ struct ewah_bitmap *read_bitmap(const unsigned char *map, return b; } -/* - * Read a bitmap from the current read position on the mmaped - * index, and increase the read position accordingly - */ -static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) -{ - return read_bitmap(index->map, index->map_size, &index->map_pos); -} - static uint32_t bitmap_num_objects_total(struct bitmap_index *index) { if (index->midx) { @@ -235,6 +226,47 @@ static uint32_t bitmap_name_hash(struct bitmap_index *index, uint32_t pos) return get_be32(index->hashes + pos); } +/* + * Read a bitmap from the current read position on the mmaped + * index, and increase the read position accordingly + */ +static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) +{ + struct ewah_bitmap *b; + size_t expected_bits; + + b = read_bitmap(index->map, index->map_size, &index->map_pos); + if (!b) + return NULL; + + /* + * It's OK for us to have too fewer bits than objects, as the EWAH + * writer may have simply left off an ending that is all-zeroes. + * + * However it's not OK for us to have too many bits, as that would + * entail touching objects that we don't have. We are careful + * enough to avoid doing so in later code, but in the case of + * nonsensical values, we would want to avoid even allocating + * memory to hold the expanded bitmap. + * + * There is one exception: we may "go over" to round up to the next + * 64-bit ewah word, since the storage comes in chunks of that size. + */ + expected_bits = bitmap_num_objects(index); + if (expected_bits & 63) { + expected_bits &= ~63; + expected_bits += 64; + } + if (b->bit_size > expected_bits) { + error("unexpected number of bits in bitmap: %"PRIuMAX" > %"PRIuMAX, + (uintmax_t)b->bit_size, (uintmax_t)expected_bits); + ewah_pool_free(b); + return NULL; + } + + return b; +} + static struct repository *bitmap_repo(struct bitmap_index *bitmap_git) { if (bitmap_is_midx(bitmap_git)) From 25273731ab116980d290646633ab439d64e5c178 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 14 May 2021 03:34:54 -0400 Subject: [PATCH 034/156] more tests These show that the problem is _not_ actually in the bitmap code. It's a more general issue with partial clones. See: https://lore.kernel.org/git/YJ4mUJ+EEAnudI3G@coredump.intra.peff.net/ Signed-off-by: Jeff King --- t/t6113-rev-list-bitmap-filters.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/t/t6113-rev-list-bitmap-filters.sh b/t/t6113-rev-list-bitmap-filters.sh index 902854cbfaacce..ef510975c7fa9a 100755 --- a/t/t6113-rev-list-bitmap-filters.sh +++ b/t/t6113-rev-list-bitmap-filters.sh @@ -156,4 +156,18 @@ test_expect_success 'bitmap traversal with --unpacked' ' test_cmp expect actual ' +test_expect_success 'filter with negative tips' ' + git rev-list --objects --filter=blob:none one..two >expect && + git rev-list --use-bitmap-index \ + --objects --filter=blob:none one..two >actual && + test_bitmap_traversal expect actual +' + +test_expect_failure 'filter with negative tips and specified blob' ' + git rev-list --objects --filter=blob:none one..two one:one.t >expect && + git rev-list --use-bitmap-index \ + --objects --filter=blob:none one..two one:one.t >actual && + test_bitmap_traversal expect actual +' + test_done From f8b1b368b375757cc4be63fd8d2f727989f98c05 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 21 Mar 2016 23:58:48 -0400 Subject: [PATCH 035/156] checkout: pass resolved default HEAD to create_branch [TODO...] Signed-off-by: Jeff King --- builtin/checkout.c | 14 +++++++++++--- t/t0610-reftable-basics.sh | 2 +- t/t1412-reflog-loop.sh | 4 +++- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/builtin/checkout.c b/builtin/checkout.c index 55e3a89a852712..e27382e38cf613 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -977,16 +977,24 @@ static void update_refs_for_switch(const struct checkout_opts *opts, strbuf_release(&err); } free(refname); - } - else + } else { + const char *start_point; + if (!strcmp(new_branch_info->name, "HEAD") && + !new_branch_info->path && + old_branch_info->path) + start_point = old_branch_info->path; + else + start_point = new_branch_info->name; + create_branch(the_repository, - opts->new_branch, new_branch_info->name, + opts->new_branch, start_point, opts->new_branch_force ? 1 : 0, opts->new_branch_force ? 1 : 0, opts->new_branch_log, opts->quiet, opts->track, 0); + } free(new_branch_info->name); free(new_branch_info->refname); new_branch_info->name = xstrdup(opts->new_branch); diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 35e98b43dba035..208b1b3ef866a8 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -865,7 +865,7 @@ test_expect_success 'reflog: expiry empties reflog' ' cat >expect <<-EOF && commit: fileB commit: fileA - branch: Created from HEAD + branch: Created from refs/heads/main EOF git reflog show --format="%gs" refs/heads/branch >actual && test_cmp expect actual && diff --git a/t/t1412-reflog-loop.sh b/t/t1412-reflog-loop.sh index f7d69b66ff33e8..d1be5b23f83632 100755 --- a/t/t1412-reflog-loop.sh +++ b/t/t1412-reflog-loop.sh @@ -2,6 +2,8 @@ test_description='reflog walk shows repeated commits again' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh test_expect_success 'setup commits' ' @@ -23,7 +25,7 @@ test_expect_success 'reflog shows all entries' ' topic@{1} reset: moving to one topic@{2} reset: moving to two topic@{3} reset: moving to one - topic@{4} branch: Created from HEAD + topic@{4} branch: Created from refs/heads/main EOF git log -g --format="%gd %gs" topic >actual && test_cmp expect actual From ea529fce5be64714466d4de6db24265b93b7bc8d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 2 Sep 2026 02:58:07 -0400 Subject: [PATCH 036/156] ci: use system asciidoctor Our CI Documentation builds have pinned asciidoctor since 615a6c37e1 (ci: stick with Asciidoctor v1.5.8 for now, 2019-03-29). Back then a few changes were needed to build with the then-new v2.0.0. We've since made those changes, in f6461b82b9 (Documentation: fix build with Asciidoctor 2, 2019-09-15) and other commits. So it is not only safe to use newer versions of asciidoctor, but preferable: it's what people are likely to use for actually building the documentation in practice! I don't think there's any need to pin to a specific version. We can just use what ships in our Ubuntu image, which should give us an arbitrary representative version (and if it changes and something breaks, we'd want to know). --- ci/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 2f61fbb07c577d..d57dce56630173 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -164,9 +164,9 @@ sparse) libexpat-dev gettext zlib1g-dev sparse ;; Documentation) - sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make ruby + sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns make ruby \ + asciidoctor - sudo gem install --version 1.5.8 asciidoctor sudo gem install concurrent-ruby ;; esac From 3becdeb0cc497c162d1b96867bd7642e4433d971 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 11 Sep 2023 04:47:43 -0400 Subject: [PATCH 037/156] json checks --- .github/workflows/main.yml | 10 +++++++++- ci/config/README | 4 +++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 205325eb33b064..74838ae933959d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -21,7 +21,15 @@ concurrency: jobs: ci-config: name: config - if: vars.CI_BRANCHES == '' || contains(vars.CI_BRANCHES, github.ref_name) + if: | + (fromJSON(vars.CI_CONFIG).allow == '' || + contains(fromJSON(vars.CI_CONFIG).allow, github.ref_name)) && + (fromJSON(vars.CI_CONFIG).reject == '' || + !contains(fromJSON(vars.CI_CONFIG).reject, github.ref_name)) && + (fromJSON(vars.CI_CONFIG).allow-prefix == '' || + startsWith(github.ref_name, fromJSON(vars.CI_CONFIG).allow-prefix)) && + (fromJSON(vars.CI_CONFIG).reject-prefix == '' || + !startsWith(github.ref_name, fromJSON(vars.CI_CONFIG).reject-prefix)) runs-on: ubuntu-latest outputs: enabled: ${{ steps.check-ref.outputs.enabled }}${{ steps.skip-if-redundant.outputs.enabled }} diff --git a/ci/config/README b/ci/config/README index 8de3a04e328eed..ebdfb849d436a1 100644 --- a/ci/config/README +++ b/ci/config/README @@ -6,7 +6,9 @@ GitHub web interface. These can be found at: The following variables can be used: - - CI_BRANCHES + - CI_CONFIG + + ... By default, CI is run when any branch is pushed. If this variable is non-empty, then only the branches it lists will run CI. Branch names From b7b848132318a71d478fbc88e81c7abaeadaa935 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 5 Dec 2025 20:34:24 -0500 Subject: [PATCH 038/156] drop extra unit-test invocations This happens in the normal test suite now. --- ci/run-test-slice.sh | 5 ----- t/Makefile | 11 ++--------- t/meson.build | 1 - 3 files changed, 2 insertions(+), 15 deletions(-) diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh index ff948e397fcb70..fd66443626fb05 100755 --- a/ci/run-test-slice.sh +++ b/ci/run-test-slice.sh @@ -10,9 +10,4 @@ TESTS=$(cd t && ./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh) group "Run tests" make --quiet -C t T="$(echo "$TESTS" | tr '\n' ' ')" || handle_failed_tests -# We only have one unit test at the moment, so run it in the first slice -if [ "$1" == "0" ] ; then - group "Run unit tests" make --quiet -C t unit-tests-test-tool -fi - check_unignored_build_artifacts diff --git a/t/Makefile b/t/Makefile index e3249879c47575..2bbaa605959d8d 100644 --- a/t/Makefile +++ b/t/Makefile @@ -79,7 +79,7 @@ failed: prove: pre-clean $(TEST_LINT) @echo "*** prove (shell & unit tests) ***" - @$(CHAINLINTSUPPRESS) TEST_OPTIONS='$(GIT_TEST_OPTS)' TEST_SHELL_PATH='$(TEST_SHELL_PATH_SQ)' $(PROVE) --exec ./run-test.sh $(GIT_PROVE_OPTS) $(T) $(UNIT_TESTS) + @$(CHAINLINTSUPPRESS) $(PROVE) --exec '$(TEST_SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS) $(MAKE) clean-except-prove-cache $(T): @@ -88,7 +88,7 @@ $(T): $(UNIT_TESTS): @echo "*** $@ ***"; $@ -.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests-test-tool +.PHONY: unit-tests unit-tests-raw unit-tests-prove unit-tests: $(DEFAULT_UNIT_TEST_TARGET) unit-tests-raw: $(UNIT_TESTS) @@ -96,13 +96,6 @@ unit-tests-raw: $(UNIT_TESTS) unit-tests-prove: @echo "*** prove - unit tests ***"; $(PROVE) $(GIT_PROVE_OPTS) $(UNIT_TESTS) -unit-tests-test-tool: - @echo "*** test-tool - unit tests **" - ( \ - cd unit-tests/bin && \ - ../../helper/test-tool$X run-command testsuite $(UNIT_TESTS_NO_DIR)\ - ) - pre-clean: $(RM) -r '$(TEST_RESULTS_DIRECTORY_SQ)' diff --git a/t/meson.build b/t/meson.build index 05d55b9e7ae01f..b8fadc2384f9d3 100644 --- a/t/meson.build +++ b/t/meson.build @@ -68,7 +68,6 @@ clar_unit_tests = executable('unit-tests', ], dependencies: [libgit_commonmain], ) -test('unit-tests', clar_unit_tests, kwargs: test_kwargs) subdir('helper') From 24816f769806976eb4a552028cc311af4cf1e63d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 29 Nov 2025 05:21:01 -0500 Subject: [PATCH 039/156] parse: prefer bool to int for boolean returns All of the integer parsing functions in parse.[ch] return an int that is "0" for failure or "1" for success. Since most of the other functions in Git use "0" for success and "-1" for failure, this can be confusing. Let's switch the return types to bool to make it clear that we are using this other convention. Callers should not need to update at all. --- parse.c | 70 ++++++++++++++++++++++++++++----------------------------- parse.h | 18 +++++++-------- 2 files changed, 44 insertions(+), 44 deletions(-) diff --git a/parse.c b/parse.c index 266bbd539be8dc..207822a983894a 100644 --- a/parse.c +++ b/parse.c @@ -15,7 +15,7 @@ static uintmax_t get_unit_factor(const char *end) return 0; } -int git_parse_signed(const char *value, intmax_t *ret, intmax_t max) +bool git_parse_signed(const char *value, intmax_t *ret, intmax_t max) { if (value && *value) { char *end; @@ -28,30 +28,30 @@ int git_parse_signed(const char *value, intmax_t *ret, intmax_t max) errno = 0; val = strtoimax(value, &end, 0); if (errno == ERANGE) - return 0; + return false; if (end == value) { errno = EINVAL; - return 0; + return false; } factor = get_unit_factor(end); if (!factor) { errno = EINVAL; - return 0; + return false; } if ((val < 0 && (-max - 1) / factor > val) || (val > 0 && max / factor < val)) { errno = ERANGE; - return 0; + return false; } val *= factor; *ret = val; - return 1; + return true; } errno = EINVAL; - return 0; + return false; } -int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max) +bool git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max) { if (value && *value) { char *end; @@ -61,53 +61,53 @@ int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max) /* negative values would be accepted by strtoumax */ if (strchr(value, '-')) { errno = EINVAL; - return 0; + return false; } errno = 0; val = strtoumax(value, &end, 0); if (errno == ERANGE) - return 0; + return false; if (end == value) { errno = EINVAL; - return 0; + return false; } factor = get_unit_factor(end); if (!factor) { errno = EINVAL; - return 0; + return false; } if (unsigned_mult_overflows(factor, val) || factor * val > max) { errno = ERANGE; - return 0; + return false; } val *= factor; *ret = val; - return 1; + return true; } errno = EINVAL; - return 0; + return false; } -int git_parse_int(const char *value, int *ret) +bool git_parse_int(const char *value, int *ret) { intmax_t tmp; if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int))) - return 0; + return false; *ret = tmp; - return 1; + return true; } -int git_parse_int64(const char *value, int64_t *ret) +bool git_parse_int64(const char *value, int64_t *ret) { intmax_t tmp; if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(int64_t))) - return 0; + return false; *ret = tmp; - return 1; + return true; } -int git_parse_uint(const char *value, unsigned int *ret) +bool git_parse_uint(const char *value, unsigned int *ret) { uintmax_t tmp; if (!git_parse_unsigned(value, &tmp, maximum_unsigned_value_of_type(unsigned int))) @@ -116,25 +116,25 @@ int git_parse_uint(const char *value, unsigned int *ret) return 1; } -int git_parse_ulong(const char *value, unsigned long *ret) +bool git_parse_ulong(const char *value, unsigned long *ret) { uintmax_t tmp; if (!git_parse_unsigned(value, &tmp, maximum_unsigned_value_of_type(long))) - return 0; + return false; *ret = tmp; - return 1; + return true; } -int git_parse_ssize_t(const char *value, ssize_t *ret) +bool git_parse_ssize_t(const char *value, ssize_t *ret) { intmax_t tmp; if (!git_parse_signed(value, &tmp, maximum_signed_value_of_type(ssize_t))) - return 0; + return false; *ret = tmp; - return 1; + return true; } -int git_parse_size_t(const char *value, size_t *ret) +bool git_parse_size_t(const char *value, size_t *ret) { uintmax_t tmp; if (!git_parse_unsigned(value, &tmp, maximum_signed_value_of_type(size_t))) @@ -143,7 +143,7 @@ int git_parse_size_t(const char *value, size_t *ret) return 1; } -int git_parse_double(const char *value, double *ret) +bool git_parse_double(const char *value, double *ret) { char *end; double val; @@ -151,25 +151,25 @@ int git_parse_double(const char *value, double *ret) if (!value || !*value) { errno = EINVAL; - return 0; + return false; } errno = 0; val = strtod(value, &end); if (errno == ERANGE) - return 0; + return false; if (end == value) { errno = EINVAL; - return 0; + return false; } factor = get_unit_factor(end); if (!factor) { errno = EINVAL; - return 0; + return false; } val *= factor; *ret = val; - return 1; + return true; } int git_parse_maybe_bool_text(const char *value) diff --git a/parse.h b/parse.h index db742f35fb0e8e..adf8ed55c28c6e 100644 --- a/parse.h +++ b/parse.h @@ -1,15 +1,15 @@ #ifndef PARSE_H #define PARSE_H -int git_parse_signed(const char *value, intmax_t *ret, intmax_t max); -int git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max); -int git_parse_ssize_t(const char *, ssize_t *); -int git_parse_size_t(const char *, size_t *); -int git_parse_ulong(const char *, unsigned long *); -int git_parse_uint(const char *value, unsigned int *ret); -int git_parse_int(const char *value, int *ret); -int git_parse_int64(const char *value, int64_t *ret); -int git_parse_double(const char *value, double *ret); +bool git_parse_signed(const char *value, intmax_t *ret, intmax_t max); +bool git_parse_unsigned(const char *value, uintmax_t *ret, uintmax_t max); +bool git_parse_ssize_t(const char *, ssize_t *); +bool git_parse_size_t(const char *, size_t *); +bool git_parse_ulong(const char *, unsigned long *); +bool git_parse_uint(const char *value, unsigned int *ret); +bool git_parse_int(const char *value, int *ret); +bool git_parse_int64(const char *value, int64_t *ret); +bool git_parse_double(const char *value, double *ret); /** * Same as `git_config_bool`, except that it returns -1 on error rather From fac14d9ee44893ede0d2a1ed6f8683911a866e42 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 29 Nov 2025 05:37:29 -0500 Subject: [PATCH 040/156] parse: add functions for parsing from non-string buffers If you have a buffer that is not NUL-terminated but want to parse an integer, there aren't many good options. If you use strtol() and friends, you risk running off the end of the buffer if there is no non-digit terminating character. And even if you carefully make sure that there is such a character, ASan's strict-string-check mode will still complain. You can copy bytes into a temporary buffer, terminate it, and then call strtol(), but doing so adds some pitfalls (like making sure you soak up whitespace and leading +/- signs, and reporting overflow for overly long input). Or you can hand-parse the digits, but then you need to take some care to handle overflow (and again, whitespace and +/- signs). These things aren't impossible to do right, but it's error-prone to have to do them in every spot that wants to do such parsing. So let's add some functions which can be used across the code base. There are a few choices regarding the interface and the implementation. First, the implementation: - I went with with parsing the digits (rather than buffering and passing to libc functions). It ends up being a similar amount of code because we have to do some parsing either way. And likewise overflow detection depends on the exact type the caller wants, so we either have to do it by hand or write a separate wrapper for strtol(), strtoumax(), and so on. - Unsigned overflow detection is done using the same techniques as in unsigned_add_overflows(), etc. We can't use those macros directly because our core function is type-agnostic (so the caller passes in the max value, rather than us deriving it on the fly). This is similar to how git_parse_int(), etc, work. - Signed overflow detection assumes that we can express a negative value with magnitude one larger than our maximum positive value (e.g., -128..127 for a signed 8-bit value). I doubt this is guaranteed by the standard, but it should hold in practice, and we make the same assumption in git_parse_int(), etc. The nice thing about this is that we can derive the range from the number of bits in the type. For ints, you obviously could use INT_MIN..INT_MAX, but for an arbitrary type, we can use maximum_signed_value_of_type(). - I didn't bother with handling bases other than 10. It would complicate the code, and I suspect it won't be needed. We could probably retro-fit it later without too much work, if need be. For the interface: - What do we call it? We have git_parse_int() and friends, which aim to make parsing less error-prone. And in some ways, these are just buffer (rather than string) versions of those functions. But not entirely. Those functions are aimed at parsing a single user-facing value. So they accept a unit prefix (e.g., "10k"), which we won't always want. And they insist that the whole string is consumed (rather than passing back an "end" pointer). We also have strtol_i() and strtoul_ui() wrappers, which try to make error handling simpler (especially around overflow), but mostly behave like their libc counterparts. These also don't pass out an end pointer, though. So I started a new namespace, "parse__from_buf". - Like those other functions above, we use an out-parameter to store the result, which lets us return an error code directly. This avoids the complicated errno dance for detecting overflow that you get with strtol(). What should the error code look like? git_parse_int() uses a bool for success/failure. But strtol_ui() uses the syscall-like "0 is success, -1 is error" convention. I went with the bool approach here. Since the names are closest to those functions, I thought it would cause the least confusion. - Unlike git_parse_signed() and friends, we do not insist that the entire buffer be consumed. For parsing a specific standalone string that makes sense, but within an unterminated buffer you are much more likely to be parsing multiple fields from a larger data set. We pass out an "end" pointer the same way strtol() does. Another option is to accept the input as an in-out parameter and advance the pointer ourselves (and likewise shrink the length pointer). That would let you do something like: if (!parse_int_from_buf(&p, &len, &out)) return error(...); /* "p" and "len" were adjusted automatically */ if (!len || *p++ != ' ') return error(...); That saves a few lines of code in some spots, but requires a few more in others (depending on whether the caller has a length in the first place or is using an end pointer). Of the two callers I intend to immediately convert, we have one of each type! I went with the strtol() approach as flexible and time-tested. - We could likewise take the input buffer as two pointers (start and end) rather than a pointer and a length. That again makes life easier for some callers and harder for others. I stuck with pointer and length as the more usual interface. - What happens when a caller passes in a NULL end pointer? This is allowed by strtol(). But I think it's often a sign of a lurking bug, because there's no way to know how much was consumed (and even if a caller wants to assume everything is consumed, you have no way to verify it). So it is simply an error in this interface (you'd get a segfault). I am tempted to say that if the end pointer is NULL the functions could confirm that the entire buffer was consumed, as a convenience. But that felt a bit magical and surprising. Like git_parse_*(), there is a generic signed/unsigned helper, and then we can add type-specific helpers on top. I've added an int helper here to start, and we'll add more as we convert callers. --- Makefile | 1 + parse.c | 96 +++++++++++++++++++++++++++++++++++++ parse.h | 17 +++++++ t/meson.build | 1 + t/unit-tests/u-parse-int.c | 98 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 213 insertions(+) create mode 100644 t/unit-tests/u-parse-int.c diff --git a/Makefile b/Makefile index d4b775953d3842..288474635f8c2f 100644 --- a/Makefile +++ b/Makefile @@ -1546,6 +1546,7 @@ CLAR_TEST_SUITES += u-odb-inmemory CLAR_TEST_SUITES += u-oid-array CLAR_TEST_SUITES += u-oidmap CLAR_TEST_SUITES += u-oidtree +CLAR_TEST_SUITES += u-parse-int CLAR_TEST_SUITES += u-prio-queue CLAR_TEST_SUITES += u-reftable-basics CLAR_TEST_SUITES += u-reftable-block diff --git a/parse.c b/parse.c index 207822a983894a..5fa3d129e47805 100644 --- a/parse.c +++ b/parse.c @@ -227,3 +227,99 @@ unsigned long git_env_ulong(const char *k, unsigned long val) die(_("failed to parse %s"), k); return val; } + +/* + * Helper that handles both signed/unsigned cases. If "negate" is NULL, + * negative values are disallowed. If not NULL and the input is negative, + * the value is range-checked but the caller is responsible for actually doing + * the negatiion. You probably don't want to use this! Use one of + * parse_signed_from_buf() or parse_unsigned_from_buf() below. + */ +static bool parse_from_buf_internal(const char *buf, size_t len, + const char **ep, bool *negate, + uintmax_t *ret, uintmax_t max) +{ + const char *end = buf + len; + uintmax_t val = 0; + + while (buf < end && isspace(*buf)) + buf++; + + if (negate) + *negate = false; + if (buf < end && *buf == '-') { + if (!negate) { + errno = EINVAL; + return false; + } + buf++; + *negate = true; + /* Assume negative range is always one larger than positive. */ + max = max + 1; + } else if (buf < end && *buf == '+') { + buf++; + } + + if (buf == end || !isdigit(*buf)) { + errno = EINVAL; + return false; + } + + while (buf < end && isdigit(*buf)) { + int digit = *buf - '0'; + + if (val > max / 10) { + errno = ERANGE; + return false; + } + val *= 10; + if (val > max - digit) { + errno = ERANGE; + return false; + } + val += digit; + + buf++; + } + + *ep = buf; + *ret = val; + return true; +} + +bool parse_unsigned_from_buf(const char *buf, size_t len, const char **ep, + uintmax_t *ret, uintmax_t max) +{ + return parse_from_buf_internal(buf, len, ep, NULL, ret, max); +} + +bool parse_signed_from_buf(const char *buf, size_t len, const char **ep, + intmax_t *ret, intmax_t max) +{ + uintmax_t u_ret; + bool negate; + + if (!parse_from_buf_internal(buf, len, ep, &negate, &u_ret, max)) + return false; + /* + * Range already checked internally, but we must apply negation + * ourselves since only we have the signed integer type. + */ + if (negate) { + *ret = u_ret; + *ret = -*ret; + } else { + *ret = u_ret; + } + return true; +} + +bool parse_int_from_buf(const char *buf, size_t len, const char **ep, int *ret) +{ + intmax_t tmp; + if (!parse_signed_from_buf(buf, len, ep, &tmp, + maximum_signed_value_of_type(int))) + return false; + *ret = tmp; + return true; +} diff --git a/parse.h b/parse.h index adf8ed55c28c6e..ebb22f58c1fdf7 100644 --- a/parse.h +++ b/parse.h @@ -21,4 +21,21 @@ int git_parse_maybe_bool_text(const char *value); int git_env_bool(const char *, int); unsigned long git_env_ulong(const char *, unsigned long); +/* + * These functions parse an integer from a buffer that does not need to be + * NUL-terminated. They return true on success, or false if no integer is found + * (in which case errno is set to EINVAL) or if the integer is out of the + * allowable range (in which case errno is ERANGE). + * + * You must pass in a non-NULL value for "ep", which returns a pointer to the + * next character in the buf (similar to strtol(), etc). + * + * These functions always parse in base 10 (and do not allow input like "0xff" + * to switch to base 16). They do not allow unit suffixes like git_parse_int(), + * above. + */ +bool parse_unsigned_from_buf(const char *buf, size_t len, const char **ep, uintmax_t *ret, uintmax_t max); +bool parse_signed_from_buf(const char *buf, size_t len, const char **ep, intmax_t *ret, intmax_t max); +bool parse_int_from_buf(const char *buf, size_t len, const char **ep, int *ret); + #endif /* PARSE_H */ diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..c8627b1cf47e0d 100644 --- a/t/meson.build +++ b/t/meson.build @@ -10,6 +10,7 @@ clar_test_suites = [ 'unit-tests/u-oid-array.c', 'unit-tests/u-oidmap.c', 'unit-tests/u-oidtree.c', + 'unit-tests/u-parse-int.c', 'unit-tests/u-prio-queue.c', 'unit-tests/u-reftable-basics.c', 'unit-tests/u-reftable-block.c', diff --git a/t/unit-tests/u-parse-int.c b/t/unit-tests/u-parse-int.c new file mode 100644 index 00000000000000..a1601bb16bdcbf --- /dev/null +++ b/t/unit-tests/u-parse-int.c @@ -0,0 +1,98 @@ +#include "unit-test.h" +#include "parse.h" + +static void check_int(const char *buf, size_t len, + size_t expect_ep_ofs, int expect_errno, + int expect_result) +{ + const char *ep; + int result; + bool ok = parse_int_from_buf(buf, len, &ep, &result); + + if (expect_errno) { + cl_assert(!ok); + cl_assert_equal_i(expect_errno, errno); + return; + } + + cl_assert(ok); + cl_assert_equal_i(expect_result, result); + cl_assert_equal_i(expect_ep_ofs, ep - buf); +} + +static void check_int_str(const char *buf, size_t ofs, int err, int res) +{ + check_int(buf, strlen(buf), ofs, err, res); +} + +static void check_int_full(const char *buf, int res) +{ + check_int_str(buf, strlen(buf), 0, res); +} + +static void check_int_err(const char *buf, int err) +{ + check_int(buf, strlen(buf), 0, err, 0); +} + +void test_parse_int__basic(void) +{ + cl_invoke(check_int_full("0", 0)); + cl_invoke(check_int_full("11", 11)); + cl_invoke(check_int_full("-23", -23)); + cl_invoke(check_int_full("+23", 23)); + + cl_invoke(check_int_str(" 31337 ", 7, 0, 31337)); + + cl_invoke(check_int_err(" garbage", EINVAL)); + cl_invoke(check_int_err("", EINVAL)); + cl_invoke(check_int_err("-", EINVAL)); + + cl_invoke(check_int("123", 2, 2, 0, 12)); +} + +void test_parse_int__range(void) +{ + /* + * These assume a 32-bit int. We could avoid that with some + * conditionals, but it's probably better for the test to + * fail noisily and we can decide how to handle it then. + */ + cl_invoke(check_int_full("2147483647", 2147483647)); + cl_invoke(check_int_err("2147483648", ERANGE)); + cl_invoke(check_int_full("-2147483647", -2147483647)); + cl_invoke(check_int_full("-2147483648", -2147483648)); + cl_invoke(check_int_err("-2147483649", ERANGE)); +} + +static void check_unsigned(const char *buf, uintmax_t max, + int expect_errno, uintmax_t expect_result) +{ + const char *ep; + uintmax_t result; + bool ok = parse_unsigned_from_buf(buf, strlen(buf), &ep, &result, max); + + if (expect_errno) { + cl_assert(!ok); + cl_assert_equal_i(expect_errno, errno); + return; + } + + cl_assert(ok); + cl_assert_equal_s(ep, ""); + /* + * Do not use cl_assert_equal_i_fmt(..., PRIuMAX) here. The macro + * casts to int under the hood, corrupting the values. + */ + clar__assert_equal(CLAR_CURRENT_FILE, CLAR_CURRENT_FUNC, + CLAR_CURRENT_LINE, + "expect_result != result", 1, + "%"PRIuMAX, expect_result, result); +} + +void test_parse_int__unsigned(void) +{ + cl_invoke(check_unsigned("4294967295", UINT_MAX, 0, 4294967295U)); + cl_invoke(check_unsigned("1053", 1000, ERANGE, 0)); + cl_invoke(check_unsigned("-17", UINT_MAX, EINVAL, 0)); +} From 546279b287b5ca5442d370b73d8e5eca30dde210 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Sep 2016 03:21:54 -0400 Subject: [PATCH 041/156] pretty: allow formatting names as initials Initials are shorter and often unique enough in a per-project setting, so they can be used to give a more informative version of --oneline. The 'S' in the placeholder is for "short" (and 's' is already taken by DATE_SHORT), but obviously that's pretty arcane. Possibly there should be more customization of initials, asking for only 2-letter initials, etc. Signed-off-by: Jeff King --- pretty.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/pretty.c b/pretty.c index d8a9f370f6c2f5..f85eb7adba724b 100644 --- a/pretty.c +++ b/pretty.c @@ -785,6 +785,23 @@ static int mailmap_name(const char **email, size_t *email_len, return mail_map->nr && map_user(mail_map, email, email_len, name, name_len); } +static void format_initials(struct strbuf *out, const char *name, size_t len) +{ + int initial = 1; + size_t i; + + for (i = 0; i < len; i++) { + char c = name[i]; + if (isspace(c)) { + initial = 1; + continue; + } + if (initial && isalpha(c)) + strbuf_addch(out, tolower(c)); + initial = 0; + } +} + static size_t format_person_part(struct strbuf *sb, char part, const char *msg, int len, struct date_mode dmode) @@ -820,6 +837,10 @@ static size_t format_person_part(struct strbuf *sb, char part, strbuf_add(sb, mail, maillen); return placeholder_len; } + if (part == 'S') { + format_initials(sb, name, namelen); + return placeholder_len; + } if (!s.date_begin) goto skip; From 1cdfe3857a1d31afd418e4c349ccbdd095548c2e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 2 Aug 2025 04:07:38 -0400 Subject: [PATCH 042/156] strbuf: more explicit NUL termination I suspect coverity thinks strings are not NUL-terminated anywhere that we call strbuf_setlen(), because of its code-path to avoid writing into the slopbuf. Even though there's an assert there, that doesn't seem to be enough (though maybe a BUG() would do it?). Anyway, it should be safe to manually terminate after each call as long as we are in a function that definitely called strbuf_grow(). I hit a new case here based on strbuf_splice(), but I'm hoping this will cleanup a lot of extra false positives, too. --- strbuf.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0de5c2db0b0472..2b822d59e2c192 100644 --- a/strbuf.c +++ b/strbuf.c @@ -258,6 +258,7 @@ void strbuf_splice(struct strbuf *sb, size_t pos, size_t len, sb->len - pos - len); memcpy(sb->buf + pos, data, dlen); strbuf_setlen(sb, sb->len + dlen - len); + sb->buf[sb->len] = '\0'; } void strbuf_insert(struct strbuf *sb, size_t pos, const void *data, size_t len) @@ -291,6 +292,7 @@ void strbuf_vinsertf(struct strbuf *sb, size_t pos, const char *fmt, va_list ap) if (len2 != len) BUG("your vsnprintf is broken (returns inconsistent lengths)"); strbuf_setlen(sb, sb->len + len); + sb->buf[sb->len] = '\0'; } void strbuf_insertf(struct strbuf *sb, size_t pos, const char *fmt, ...) @@ -328,6 +330,7 @@ void strbuf_addbuf(struct strbuf *sb, const struct strbuf *sb2) strbuf_grow(sb, sb2->len); memcpy(sb->buf + sb->len, sb2->buf, sb2->len); strbuf_setlen(sb, sb->len + sb2->len); + sb->buf[sb->len] = '\0'; } const char *strbuf_join_argv(struct strbuf *buf, @@ -350,6 +353,7 @@ void strbuf_addchars(struct strbuf *sb, int c, size_t n) strbuf_grow(sb, n); memset(sb->buf + sb->len, c, n); strbuf_setlen(sb, sb->len + n); + sb->buf[sb->len] = '\0'; } void strbuf_addf(struct strbuf *sb, const char *fmt, ...) @@ -434,6 +438,7 @@ void strbuf_vaddf(struct strbuf *sb, const char *fmt, va_list ap) BUG("your vsnprintf is broken (insatiable)"); } strbuf_setlen(sb, sb->len + len); + sb->buf[sb->len] = '\0'; } int strbuf_expand_step(struct strbuf *sb, const char **formatp) @@ -1057,6 +1062,7 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm, } strbuf_release(&munged_fmt); strbuf_setlen(sb, sb->len + len); + sb->buf[sb->len] = '\0'; } /* @@ -1124,6 +1130,7 @@ void strbuf_stripspace(struct strbuf *sb, const char *comment_prefix) } strbuf_setlen(sb, j); + sb->buf[sb->len] = '\0'; } void strbuf_strip_file_from_path(struct strbuf *sb) From 7d74b5c6e9efe485900744c434120888bb6c9228 Mon Sep 17 00:00:00 2001 From: Anders Kaseorg Date: Wed, 19 Aug 2015 15:46:27 -0400 Subject: [PATCH 043/156] t9350-fast-export: Add failing test for symlink-to-directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit git fast-export | git fast-import fails to preserve a commit that replaces a symlink with a directory. Add a failing test case demonstrating this bug. The fast-export output for the commit in question looks like commit refs/heads/master mark :4 author … committer … data 4 two M 100644 :1 foo/world D foo fast-import deletes the symlink foo and ignores foo/world. Swapping the M line with the D line would give the correct result. Signed-off-by: Anders Kaseorg Signed-off-by: Jeff King --- t/t9350-fast-export.sh | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh index a487476ad45beb..d657573852f566 100755 --- a/t/t9350-fast-export.sh +++ b/t/t9350-fast-export.sh @@ -781,6 +781,30 @@ test_expect_success 'directory becomes symlink' ' (cd result && git show main:foo) ' +test_expect_failure 'symlink becomes directory' ' + git init symlinktodir && + git init symlinktodirresult && + ( + cd symlinktodir && + mkdir bar && + echo hello > bar/world && + test_ln_s_add bar foo && + git add foo bar/world && + git commit -q -mone && + git rm foo && + mkdir foo && + echo hello > foo/world && + git add foo/world && + git commit -q -mtwo + ) && + ( + cd symlinktodir && + git fast-export master -- foo | + (cd ../symlinktodirresult && git fast-import --quiet) + ) && + (cd symlinktodirresult && git show master:foo) +' + test_expect_success PERL_TEST_HELPERS 'fast-export quotes pathnames' ' git init crazy-paths && test_config -C crazy-paths core.protectNTFS false && From a7f05f6cc6e50a10306eac70649cfd2963d5ec40 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 29 Nov 2025 05:37:39 -0500 Subject: [PATCH 044/156] cache-tree: use parse_int_from_buf() In c4c9089584 (cache-tree: avoid strtol() on non-string buffer, 2025-11-18) we wrote an ad-hoc integer parser which did not detect overflow. This wasn't too big a problem, since the original use of strtol() did not do so either. But now that we have a more robust parsing function, let's use that. It reduces the amount of code and should catch more cases of malformed entries. I kept our local parse_int() wrapper here, since it handles management of our ptr/len pair (rather than doing it inline in the entry parser of read_one()). --- cache-tree.c | 28 +++++----------------------- 1 file changed, 5 insertions(+), 23 deletions(-) diff --git a/cache-tree.c b/cache-tree.c index a220372a420197..8980f31fdc633c 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -17,6 +17,7 @@ #include "promisor-remote.h" #include "trace.h" #include "trace2.h" +#include "parse.h" #ifndef DEBUG_CACHE_TREE #define DEBUG_CACHE_TREE 0 @@ -596,32 +597,13 @@ void cache_tree_write(struct strbuf *sb, struct cache_tree *root) static int parse_int(const char **ptr, unsigned long *len_p, int *out) { - const char *s = *ptr; - unsigned long len = *len_p; - int ret = 0; - int sign = 1; - - while (len && *s == '-') { - sign *= -1; - s++; - len--; - } - - while (len) { - if (!isdigit(*s)) - break; - ret *= 10; - ret += *s - '0'; - s++; - len--; - } + const char *ep; - if (s == *ptr) + if (!parse_int_from_buf(*ptr, *len_p, &ep, out)) return -1; - *ptr = s; - *len_p = len; - *out = sign * ret; + *len_p -= ep - *ptr; + *ptr = ep; return 0; } From ae3b2047981a3d5c332dcbc6172174478c67c9a7 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Sep 2016 03:11:07 -0400 Subject: [PATCH 045/156] graph: helper functions for printing commit header The idea here is to make it possible to print something right _before_ a commit in the graph. It's a bit ugly, but it seems to mostly work (see the next commit for some oddness). Signed-off-by: Jeff King --- graph.c | 14 +++++++++++++- graph.h | 6 ++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/graph.c b/graph.c index 28bef1b88f5a7a..faf26c6dcd7ffa 100644 --- a/graph.c +++ b/graph.c @@ -1851,7 +1851,7 @@ int graph_is_commit_finished(struct git_graph const *graph) return (graph->state == GRAPH_PADDING); } -void graph_show_commit(struct git_graph *graph) +static void graph_show_commit_1(struct git_graph *graph, int only_pre) { struct strbuf msgbuf = STRBUF_INIT; int shown_commit_line = 0; @@ -1872,6 +1872,8 @@ void graph_show_commit(struct git_graph *graph) } while (!shown_commit_line && !graph_is_commit_finished(graph)) { + if (only_pre && graph->state == GRAPH_COMMIT) + break; shown_commit_line = graph_next_line(graph, &msgbuf); fwrite(msgbuf.buf, sizeof(char), msgbuf.len, graph->revs->diffopt.file); @@ -1885,6 +1887,16 @@ void graph_show_commit(struct git_graph *graph) strbuf_release(&msgbuf); } +void graph_show_precommit(struct git_graph *graph) +{ + graph_show_commit_1(graph, 1); +} + +void graph_show_commit(struct git_graph *graph) +{ + graph_show_commit_1(graph, 0); +} + void graph_show_oneline(struct git_graph *graph) { struct strbuf msgbuf = STRBUF_INIT; diff --git a/graph.h b/graph.h index 1193711fb8892c..8c4b938cd3b0f3 100644 --- a/graph.h +++ b/graph.h @@ -218,6 +218,12 @@ int graph_width(struct git_graph *graph); */ void graph_show_commit(struct git_graph *graph); +/* + * Same as graph_show_commit, but stop just _before_ printing + * the actual commit line. + */ +void graph_show_precommit(struct git_graph *graph); + /* * If the graph is non-NULL, print one line of the history graph to stdout. * Does not print a terminating newline on the last line. From 31df923f726e34697e88f6c72250bcd87abf71d4 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 3 Apr 2024 19:08:01 -0400 Subject: [PATCH 046/156] pull: drop sketchy mkpath() We return it from a function, giving it an unclear lifetime (though probably OK in practice?). --- builtin/pull.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/builtin/pull.c b/builtin/pull.c index db3ee0aab3ed91..468cd1fb4e2cc2 100644 --- a/builtin/pull.c +++ b/builtin/pull.c @@ -606,11 +606,11 @@ static const char *get_upstream_branch(const char *remote) * FIXME: The current implementation assumes the default mapping of * refs/heads/ to refs/remotes//. */ -static const char *get_tracking_branch(const char *remote, const char *refspec) +static char *get_tracking_branch(const char *remote, const char *refspec) { struct refspec_item spec; const char *spec_src; - const char *merge_branch; + char *merge_branch; if (!refspec_item_init_fetch(&spec, refspec, the_hash_algo)) die(_("invalid refspec '%s'"), refspec); @@ -628,9 +628,9 @@ static const char *get_tracking_branch(const char *remote, const char *refspec) if (*spec_src) { if (!strcmp(remote, ".")) - merge_branch = mkpath("refs/heads/%s", spec_src); + merge_branch = mkpathdup("refs/heads/%s", spec_src); else - merge_branch = mkpath("refs/remotes/%s/%s", remote, spec_src); + merge_branch = mkpathdup("refs/remotes/%s/%s", remote, spec_src); } else merge_branch = NULL; @@ -649,6 +649,7 @@ static int get_rebase_fork_point(struct object_id *fork_point, const char *repo, int ret; struct branch *curr_branch; const char *remote_branch; + char *to_free = NULL; struct child_process cp = CHILD_PROCESS_INIT; struct strbuf sb = STRBUF_INIT; @@ -657,7 +658,7 @@ static int get_rebase_fork_point(struct object_id *fork_point, const char *repo, return -1; if (refspec) - remote_branch = get_tracking_branch(repo, refspec); + remote_branch = to_free = get_tracking_branch(repo, refspec); else remote_branch = get_upstream_branch(repo); @@ -680,6 +681,7 @@ static int get_rebase_fork_point(struct object_id *fork_point, const char *repo, cleanup: strbuf_release(&sb); + free(to_free); return ret ? -1 : 0; } From b6dff52519202103759345cff50b1f1514a8b85b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 27 Aug 2025 19:44:30 -0400 Subject: [PATCH 047/156] khash null keys hackery --- khash.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/khash.h b/khash.h index ff881631778586..acc8e0ae08bd88 100644 --- a/khash.h +++ b/khash.h @@ -175,7 +175,7 @@ static const double __ac_HASH_UPPER = 0.77; SCOPE khint_t kh_put_##name(kh_##name##_t *h, khkey_t key, int *ret) \ { \ khint_t x; \ - if (h->n_occupied >= h->upper_bound) { /* update the hash table */ \ + if (!h->keys || !h->flags || h->n_occupied >= h->upper_bound) { /* update the hash table */ \ if (h->n_buckets > (h->size<<1)) { \ kh_resize_##name(h, h->n_buckets - 1); /* clear "deleted" elements */ \ } else { \ From da69d2f297aa5b6700c05f0d9da91d4fd8f007ea Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 9 Jun 2022 14:43:25 -0400 Subject: [PATCH 048/156] show: skip diff when possible Running: git show -s $commit will still compute a diff for $commit, even though we aren't going to show it. This is wasted computation, since it cannot affect the output or exit code of the program. In the more general case: - if the requested diff format is NO_OUTPUT, then we won't change the output of the diff itself - if rev_info.always_show_header is set, then we will show the commit regardless of whether the diff is empty (which is true for git-show, for example, but not git-log) - we don't use --exit-code here (should check?) --- log-tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/log-tree.c b/log-tree.c index 83a3c4bf9b16b9..5997230a21d8d8 100644 --- a/log-tree.c +++ b/log-tree.c @@ -1196,6 +1196,10 @@ static int log_tree_diff(struct rev_info *opt, struct commit *commit, struct log if (!all_need_diff && !opt->merges_need_diff) return 0; + if (opt->diffopt.output_format == DIFF_FORMAT_NO_OUTPUT && + opt->always_show_header) + return 0; + if (opt->line_level_traverse) { line_log_queue_pairs(opt, commit); log_tree_diff_flush(opt); From 6f2811ec93c1bbf6a4e92b33e47b97010cabf463 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 21 Aug 2015 12:19:00 -0400 Subject: [PATCH 049/156] fix? Signed-off-by: Jeff King --- builtin/fast-export.c | 8 ++++++++ t/t9350-fast-export.sh | 6 +++--- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/builtin/fast-export.c b/builtin/fast-export.c index 50278bef7a6a4d..ca78970ce289fc 100644 --- a/builtin/fast-export.c +++ b/builtin/fast-export.c @@ -357,6 +357,14 @@ static int depth_first(const void *a_, const void *b_) const char *name_a, *name_b; int len_a, len_b, len; int cmp; + int deletion; + + /* + * Move all deletions first, to make room for any later modifications. + */ + deletion = (b->status == 'D') - (a->status == 'D'); + if (deletion) + return deletion; name_a = a->one ? a->one->path : a->two->path; name_b = b->one ? b->one->path : b->two->path; diff --git a/t/t9350-fast-export.sh b/t/t9350-fast-export.sh index d657573852f566..b57712647fea38 100755 --- a/t/t9350-fast-export.sh +++ b/t/t9350-fast-export.sh @@ -781,7 +781,7 @@ test_expect_success 'directory becomes symlink' ' (cd result && git show main:foo) ' -test_expect_failure 'symlink becomes directory' ' +test_expect_success 'symlink becomes directory' ' git init symlinktodir && git init symlinktodirresult && ( @@ -799,10 +799,10 @@ test_expect_failure 'symlink becomes directory' ' ) && ( cd symlinktodir && - git fast-export master -- foo | + git fast-export main -- foo | (cd ../symlinktodirresult && git fast-import --quiet) ) && - (cd symlinktodirresult && git show master:foo) + (cd symlinktodirresult && git show main:foo) ' test_expect_success PERL_TEST_HELPERS 'fast-export quotes pathnames' ' From ad2fa636f63cf71cf8b219ff97f16bfcc0483099 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 2 Apr 2026 03:59:19 -0400 Subject: [PATCH 050/156] fetch: support fetch.defaultRemote This overrides fetching from the remote of the current branch, which doesn't always make sense (e.g., when @{upstream} points to a local branch which is in turn based on "origin"). --- builtin/fetch.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index ab7db2be06d14c..600596dd26a896 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -104,6 +104,7 @@ static struct string_list negotiation_include = STRING_LIST_INIT_NODUP; struct fetch_config { enum display_format display_format; enum follow_remote_head_settings follow_remote_head; + char *default_remote; int all; int prune; int prune_tags; @@ -123,6 +124,11 @@ static int git_fetch_config(const char *k, const char *v, return 0; } + if (!strcmp(k, "fetch.defaultremote")) { + FREE_AND_NULL(fetch_config->default_remote); + return git_config_string(&fetch_config->default_remote, k, v); + } + if (!strcmp(k, "fetch.prune")) { fetch_config->prune = git_config_bool(k, v); return 0; @@ -2758,7 +2764,7 @@ int cmd_fetch(int argc, remote = remote_get(list.items[0].string); } else if (argc == 0) { /* No arguments -- use default remote */ - remote = remote_get(NULL); + remote = remote_get(config.default_remote); } else if (multiple) { /* All arguments are assumed to be remotes or groups */ for (i = 0; i < argc; i++) From 57c61b85371d7ab3fbc2d9d095494c194b5780bf Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 18 Sep 2023 19:11:10 -0400 Subject: [PATCH 051/156] rev-parse: clean up funky git-dir cwd handling --- builtin/rev-parse.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c index 43693454d53f0c..62a96a05a14081 100644 --- a/builtin/rev-parse.c +++ b/builtin/rev-parse.c @@ -1031,8 +1031,7 @@ int cmd_rev_parse(int argc, if (!strcmp(arg, "--git-dir") || !strcmp(arg, "--absolute-git-dir")) { const char *gitdir = getenv(GIT_DIR_ENVIRONMENT); - char *cwd; - int len; + struct strbuf path = STRBUF_INIT; enum format_type wanted = format; if (arg[2] == 'g') { /* --git-dir */ if (gitdir) { @@ -1055,12 +1054,12 @@ int cmd_rev_parse(int argc, continue; } } - cwd = xgetcwd(); - len = strlen(cwd); - strbuf_reset(&buf); - strbuf_addf(&buf, "%s%s.git", cwd, len && cwd[len-1] != '/' ? "/" : ""); - free(cwd); - print_path(buf.buf, prefix, wanted, DEFAULT_CANONICAL); + if (strbuf_getcwd(&path)) + die_errno(_("unable to get current working directory")); + strbuf_complete(&path, '/'); + strbuf_addstr(&path, ".git"); + print_path(path.buf, prefix, wanted, DEFAULT_CANONICAL); + strbuf_release(&path); continue; } if (!strcmp(arg, "--git-common-dir")) { From fe1aad59ddf03034e1c1f5b367d48acf31422002 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 30 Jun 2023 03:22:56 -0400 Subject: [PATCH 052/156] index_mem(): check return value of fsck_finish() This would catch problems in blobs that were newly linked to git metafiles (like .gitmodules) by a tree we are checking. We should add a test. Note that fsck_finish() both checks the blobs and frees the memory. So in a short-circuit "||", we'd fail to do that cleanup if fsck_buffer() reports a problem. That's OK in this case because we die(). Maybe this should stay as two separate conditionals. --- object-file.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/object-file.c b/object-file.c index a4cbf8b081df1e..7f588e0273cc4d 100644 --- a/object-file.c +++ b/object-file.c @@ -590,9 +590,9 @@ static int index_mem(struct index_state *istate, fsck_options_init(&opts, the_repository, FSCK_OPTIONS_DEFAULT); opts.strict = 1; opts.error_func = hash_format_check_report; - if (fsck_buffer(null_oid(istate->repo->hash_algo), type, buf, size, &opts)) + if (fsck_buffer(null_oid(istate->repo->hash_algo), type, buf, size, &opts) || + fsck_finish(&opts)) die(_("refusing to create malformed object")); - fsck_finish(&opts); } if (write_object) From 4248bd8f831ca7e52a2acb00ddb630e514671c8e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 23 Jul 2012 14:52:18 -0400 Subject: [PATCH 053/156] commit: give a hint when a commit message has been abandoned If we launch an editor for the user to create a commit message, they may put significant work into doing so. Typically we try to check common mistakes that could cause the commit to fail early, so that we die before the user goes to the trouble. We may still experience some errors afterwards, though; in this case, the user is given no hint that their commit message has been saved. Let's tell them where it is. Signed-off-by: Jeff King --- builtin/commit.c | 15 +++++++++++++++ t/t7500-commit-template-squash-signoff.sh | 3 +-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 28f61745034506..e6f2828b215093 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -166,6 +166,16 @@ static int opt_parse_porcelain(const struct option *opt, const char *arg, int un return 0; } +static int mention_abandoned_message; +static void maybe_mention_abandoned_message(void) +{ + if (!mention_abandoned_message) + return; + advise(_("Your commit message has been saved in '%s' and will be\n" + "overwritten by the next invocation of \"git commit\"."), + git_path_commit_editmsg()); +} + static int opt_parse_m(const struct option *opt, const char *arg, int unset) { struct strbuf *buf = opt->value; @@ -1127,6 +1137,8 @@ static int prepare_to_commit(const char *index_file, const char *prefix, exit(1); } strvec_clear(&env); + atexit(maybe_mention_abandoned_message); + mention_abandoned_message = 1; } if (!no_verify && @@ -1906,11 +1918,13 @@ int cmd_commit(int argc, if (message_is_empty(&sb, cleanup_mode) && !allow_empty_message) { rollback_index_files(); fprintf(stderr, _("Aborting commit due to empty commit message.\n")); + mention_abandoned_message = 0; exit(1); } if (template_untouched(&sb, template_file, cleanup_mode) && !allow_empty_message) { rollback_index_files(); fprintf(stderr, _("Aborting commit; you did not edit the message.\n")); + mention_abandoned_message = 0; exit(1); } @@ -1948,6 +1962,7 @@ int cmd_commit(int argc, die("%s", err.buf); } + mention_abandoned_message = 0; sequencer_post_commit_cleanup(the_repository, 0); unlink(git_path_merge_head(the_repository)); unlink(git_path_merge_msg(the_repository)); diff --git a/t/t7500-commit-template-squash-signoff.sh b/t/t7500-commit-template-squash-signoff.sh index 5e782ad8d798a0..9143aa086ef131 100755 --- a/t/t7500-commit-template-squash-signoff.sh +++ b/t/t7500-commit-template-squash-signoff.sh @@ -423,13 +423,12 @@ test_expect_success 'consecutive amend! commits remove amend! line from commit m test_expect_success 'deny to create amend! commit if its commit msg body is empty' ' commit_for_rebase_autosquash_setup && - echo "Aborting commit due to empty commit message body." >expected && ( set_fake_editor && test_must_fail env FAKE_COMMIT_MESSAGE="amend! target message subject line" \ git commit --fixup=amend:HEAD~ 2>actual ) && - test_cmp expected actual + test_grep "Aborting commit due to empty commit message body" actual ' test_expect_success 'amend! commit allows empty commit msg body with --allow-empty-message' ' From 30f8f98b5edf4f2c60dd2e92777aaac221feb057 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 29 Nov 2025 05:48:34 -0500 Subject: [PATCH 054/156] fsck: use parse_unsigned_from_buf() for parsing timestamp In 5a993593b2 (fsck: avoid parse_timestamp() on buffer that isn't NUL-terminated, 2025-11-18), we added a wrapper that copies the timestamp into a buffer before calling parse_timestamp(). Now that we have a more robust helper for parsing from a buffer, we can drop our wrapper and switch to that. We could just do so inline, but the choice of "unsigned" vs "signed" depends on the typedef of timestamp_t. So we'll wrap that in a macro that is defined alongside the rest of the timestamp abstraction. The resulting function is almost a drop-in replacement, but the new interface means we need to hold the result in a separate timestamp_t, rather than returning it directly from one function into the parameter of another. The old one did still detect overflow errors by returning TIME_MAX, since date_overflows() checks for that, but now we'll see it more directly from the return of parse_timestamp_from_buf(). The behavior should be the same. --- compat/posix.h | 2 ++ fsck.c | 20 +++----------------- 2 files changed, 5 insertions(+), 17 deletions(-) diff --git a/compat/posix.h b/compat/posix.h index 71cc7316204187..8b38abb8614053 100644 --- a/compat/posix.h +++ b/compat/posix.h @@ -267,6 +267,8 @@ char *gitdirname(char *); typedef uintmax_t timestamp_t; #define PRItime PRIuMAX #define parse_timestamp strtoumax +#define parse_timestamp_from_buf(buf, len, ep, result) \ + parse_unsigned_from_buf((buf), (len), (ep), (result), TIME_MAX) #define TIME_MAX UINTMAX_MAX #define TIME_MIN 0 diff --git a/fsck.c b/fsck.c index 94c8651c7dfa28..9e49dce7d9c17e 100644 --- a/fsck.c +++ b/fsck.c @@ -858,28 +858,13 @@ static int verify_headers(const void *data, unsigned long size, FSCK_MSG_UNTERMINATED_HEADER, "unterminated header"); } -static timestamp_t parse_timestamp_from_buf(const char **start, const char *end) -{ - const char *p = *start; - char buf[24]; /* big enough for 2^64 */ - size_t i = 0; - - while (p < end && isdigit(*p)) { - if (i >= ARRAY_SIZE(buf) - 1) - return TIME_MAX; - buf[i++] = *p++; - } - buf[i] = '\0'; - *start = p; - return parse_timestamp(buf, NULL, 10); -} - static int fsck_ident(const char **ident, const char *ident_end, const struct object_id *oid, enum object_type type, struct fsck_options *options) { const char *p = *ident; const char *nl; + timestamp_t timestamp; nl = memchr(p, '\n', ident_end - p); if (!nl) @@ -931,7 +916,8 @@ static int fsck_ident(const char **ident, const char *ident_end, "invalid author/committer line - bad date"); if (*p == '0' && p[1] != ' ') return report(options, oid, type, FSCK_MSG_ZERO_PADDED_DATE, "invalid author/committer line - zero-padded date"); - if (date_overflows(parse_timestamp_from_buf(&p, ident_end))) + if (!parse_timestamp_from_buf(p, ident_end - p, &p, ×tamp) || + date_overflows(timestamp)) return report(options, oid, type, FSCK_MSG_BAD_DATE_OVERFLOW, "invalid author/committer line - date causes integer overflow"); if (*p != ' ') return report(options, oid, type, FSCK_MSG_BAD_DATE, "invalid author/committer line - bad date"); From 064aa76fc16ab2b0c031dd530ac4f91d79745021 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Sep 2016 03:16:27 -0400 Subject: [PATCH 055/156] log: add --commit-header option This lets you stick a header right before a commit, but suppresses headers that are duplicates. This means you can do something like: git log --graph --author-date-order --commit-header='== %as ==' to get a marker in the graph whenever the day changes. This probably needs some refactoring around the setup of the pretty-print context. Signed-off-by: Jeff King --- log-tree.c | 31 +++++++++++++++++++++++++++++++ revision.c | 3 +++ revision.h | 3 +++ 3 files changed, 37 insertions(+) diff --git a/log-tree.c b/log-tree.c index 83a3c4bf9b16b9..7716c20254920c 100644 --- a/log-tree.c +++ b/log-tree.c @@ -682,6 +682,28 @@ static int show_mergetag(struct rev_info *opt, struct commit *commit) return for_each_mergetag(show_one_mergetag, commit, opt); } +static void show_commit_header(struct rev_info *opt, + struct pretty_print_context *pp, + struct commit *commit) +{ + struct strbuf out = STRBUF_INIT; + + repo_format_commit_message(the_repository, commit, opt->commit_header, &out, pp); + strbuf_complete_line(&out); + + if (!strbuf_cmp(&out, &opt->last_commit_header)) { + strbuf_release(&out); + return; + } + + graph_show_precommit(opt->graph); + graph_show_padding(opt->graph); + fwrite(out.buf, 1, out.len, opt->diffopt.file); + + strbuf_swap(&out, &opt->last_commit_header); + strbuf_release(&out); +} + static void next_commentary_block(struct rev_info *opt, struct strbuf *sb) { const char *x = opt->shown_dashes ? "\n" : "---\n"; @@ -794,6 +816,15 @@ void show_log(struct rev_info *opt) } opt->shown_one = 1; + if (opt->commit_header) { + /* + * XXX probably the initialization of the pretty ctx from "opt" + * below should happen sooner so we can use it. + */ + ctx.color = opt->diffopt.use_color; + show_commit_header(opt, &ctx, commit); + } + /* * If the history graph was requested, * print the graph, up to this commit's line diff --git a/revision.c b/revision.c index 35afe52208e710..4155ee1157cb90 100644 --- a/revision.c +++ b/revision.c @@ -1970,6 +1970,7 @@ void repo_init_revisions(struct repository *r, list_objects_filter_init(&revs->filter); init_ref_exclusions(&revs->ref_excludes); oidset_init(&revs->missing_commits, 0); + strbuf_init(&revs->last_commit_header, 0); } static void add_pending_commit_list(struct rev_info *revs, @@ -2584,6 +2585,8 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg revs->verbose_header = 1; revs->pretty_given = 1; get_commit_format(optarg, revs); + } else if (skip_prefix(arg, "--commit-header=", &arg)) { + revs->commit_header = arg; } else if (!strcmp(arg, "--expand-tabs")) { revs->expand_tabs_in_log = 8; } else if (!strcmp(arg, "--no-expand-tabs")) { diff --git a/revision.h b/revision.h index acf6d06b24126c..39a1e491221deb 100644 --- a/revision.h +++ b/revision.h @@ -308,6 +308,9 @@ struct rev_info { int show_log_size; struct string_list *mailmap; + const char *commit_header; + struct strbuf last_commit_header; + /* Filter by commit log message */ struct grep_opt grep_filter; From 7e8ff3a77615b76320ac4febd1aea113781a11f5 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 2 Jul 2026 17:31:08 -0400 Subject: [PATCH 056/156] enable --long for private-branch pushes The upstream check is overly specific to avoid running on topic branches (even though it is weirdly loose about its globbing). --- ci/lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/lib.sh b/ci/lib.sh index 6c52154eac11e9..f0b0b84a4274dc 100755 --- a/ci/lib.sh +++ b/ci/lib.sh @@ -328,7 +328,7 @@ export SKIP_DASHED_BUILT_INS=YesPlease # In order to catch bugs introduced at integration time by mismerges, # enable the long tests for pushes to the integration branches as well. case "$CI_EVENT,$CI_BRANCH" in -pull_request,*|push,*next*|push,*master*|push,*main*|push,*maint*) +pull_request,*|push,*next*|push,*master*|push,*main*|push,*maint*|push,*private*) export GIT_TEST_LONG=${GIT_TEST_LONG:-true} ;; esac From 01b6f137e0a20663413bbbe19121881b5d403423 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Oct 2021 17:02:18 -0400 Subject: [PATCH 057/156] handle %(trailers:mailmap) Signed-off-by: Jeff King --- pretty.c | 10 ++++++++++ trailer.c | 34 +++++++++++++++++++++++++++++++--- trailer.h | 1 + 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/pretty.c b/pretty.c index d8a9f370f6c2f5..900bf8b9941bac 100644 --- a/pretty.c +++ b/pretty.c @@ -1295,6 +1295,7 @@ int format_set_trailers_options(struct process_trailer_options *opts, for (;;) { const char *argval; size_t arglen; + int bool_arg; if (**arg == ')') break; @@ -1316,6 +1317,15 @@ int format_set_trailers_options(struct process_trailer_options *opts, opts->separator = expand_string_arg(sepbuf, argval, arglen); } else if (match_placeholder_arg_value(*arg, "key_value_separator", arg, &argval, &arglen)) { opts->key_value_separator = expand_string_arg(kvsepbuf, argval, arglen); + } else if (match_placeholder_bool_arg(*arg, "mailmap", arg, &bool_arg)) { + if (bool_arg) { + /* yuck but this is how mailmap_name() above does it */ + static struct string_list mailmap = STRING_LIST_INIT_DUP; + read_mailmap(the_repository, &mailmap); + opts->mailmap = &mailmap; + } else { + opts->mailmap = NULL; + } } else if (!match_placeholder_bool_arg(*arg, "only", arg, &opts->only_trailers) && !match_placeholder_bool_arg(*arg, "unfold", arg, &opts->unfold) && !match_placeholder_bool_arg(*arg, "keyonly", arg, &opts->key_only) && diff --git a/trailer.c b/trailer.c index 10b1abebfbe906..66c0199b10b6cc 100644 --- a/trailer.c +++ b/trailer.c @@ -11,6 +11,8 @@ #include "trailer.h" #include "list.h" #include "tempfile.h" +#include "mailmap.h" +#include "ident.h" /* * Copyright (c) 2013, 2014 Christian Couder @@ -1138,6 +1140,29 @@ void trailer_block_release(struct trailer_block *trailer_block) free(trailer_block); } +static int mailmap_value(struct string_list *mailmap, + struct strbuf *out, const struct strbuf *in) +{ + const char *mailbuf, *namebuf; + size_t namelen, maillen; + struct ident_split ident; + + if (split_ident_line(&ident, in->buf, in->len)) + return -1; /* not an ident */ + + namebuf = ident.name_begin; + namelen = ident.name_end - ident.name_begin; + mailbuf = ident.mail_begin; + maillen = ident.mail_end - ident.mail_begin; + + map_user(mailmap, &mailbuf, &maillen, &namebuf, &namelen); + strbuf_add(out, namebuf, namelen); + strbuf_addstr(out, " <"); + strbuf_add(out, mailbuf, maillen); + strbuf_addch(out, '>'); + return 0; +} + void format_trailers(const struct process_trailer_options *opts, struct list_head *trailers, struct strbuf *out) @@ -1179,8 +1204,11 @@ void format_trailers(const struct process_trailer_options *opts, strbuf_addf(out, "%c ", separators[0]); } } - if (!opts->key_only) - strbuf_addbuf(out, &val); + if (!opts->key_only) { + if (!opts->mailmap || + mailmap_value(opts->mailmap, out, &val) < 0) + strbuf_addbuf(out, &val); + } if (!opts->separator) strbuf_addch(out, '\n'); } @@ -1210,7 +1238,7 @@ void format_trailers_from_commit(const struct process_trailer_options *opts, /* If we want the whole block untouched, we can take the fast path. */ if (!opts->only_trailers && !opts->unfold && !opts->filter && !opts->separator && !opts->key_only && !opts->value_only && - !opts->key_value_separator) { + !opts->key_value_separator && !opts->mailmap) { strbuf_add(out, msg + trailer_block->start, trailer_block->end - trailer_block->start); } else diff --git a/trailer.h b/trailer.h index b49338858c482a..b76254630cbd6f 100644 --- a/trailer.h +++ b/trailer.h @@ -55,6 +55,7 @@ struct process_trailer_options { int no_divider; int key_only; int value_only; + struct string_list *mailmap; const struct strbuf *separator; const struct strbuf *key_value_separator; int (*filter)(const struct strbuf *, void *); From 8e7ef925411d83a5036d2b32799b57648e24d30a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 3 Apr 2024 19:11:27 -0400 Subject: [PATCH 058/156] use nodup + mkpathdup --- builtin/am.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/builtin/am.c b/builtin/am.c index e9623b8307793f..7f6b22daf3f96c 100644 --- a/builtin/am.c +++ b/builtin/am.c @@ -861,7 +861,8 @@ static int split_mail_stgit_series(struct am_state *state, const char **paths, if (*sb.buf == '#') continue; /* skip comment lines */ - strvec_push(&patches, mkpath("%s/%s", series_dir, sb.buf)); + strvec_push_nodup(&patches, + mkpathdup("%s/%s", series_dir, sb.buf)); } fclose(fp); @@ -2516,7 +2517,8 @@ int cmd_am(int argc, if (is_absolute_path(argv[i]) || !prefix) strvec_push(&paths, argv[i]); else - strvec_push(&paths, mkpath("%s/%s", prefix, argv[i])); + strvec_push_nodup(&paths, + mkpathdup("%s/%s", prefix, argv[i])); } if (state.interactive && !paths.nr) From 35ec68395f69f598c279f258b9080903f5bb0d6c Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 2 Jun 2026 04:22:43 -0400 Subject: [PATCH 059/156] test-lib: support generic environment overrides Callers of test-lib sometimes need values from their environment to take precedence over GIT-BUILD-OPTIONS. They cannot reliably restore those values after sourcing test-lib, since options such as --tee may cause test-lib to re-execute the test and exit without returning to its caller. The restoration therefore has to happen inside test-lib, after it reads GIT-BUILD-OPTIONS but before it may re-execute the test. We already do this for TEST_OUTPUT_DIRECTORY via a special-purpose override variable. Turn that into a generic list of assignments so other callers can use the same restoration point. Keep explicitly supplied assignments last in the list, so they take precedence over the TEST_OUTPUT_DIRECTORY value test-lib adds itself. The next patch will use this mechanism to preserve GIT_PERF_*. --- t/lib-subtest.sh | 7 ++----- t/test-lib.sh | 30 ++++++++++++++++-------------- 2 files changed, 18 insertions(+), 19 deletions(-) diff --git a/t/lib-subtest.sh b/t/lib-subtest.sh index 56ee927f0c49be..5c05bf44199655 100644 --- a/t/lib-subtest.sh +++ b/t/lib-subtest.sh @@ -40,11 +40,8 @@ _run_sub_test_lib_test_common () { sane_unset HARNESS_ACTIVE && export TEST_DIRECTORY && - # The child test re-sources GIT-BUILD-OPTIONS and may thus - # override the test output directory. We thus pass it as an - # explicit override to the child. - TEST_OUTPUT_DIRECTORY_OVERRIDE=$(pwd) && - export TEST_OUTPUT_DIRECTORY_OVERRIDE && + TEST_OUTPUT_DIRECTORY=$(pwd) && + export TEST_OUTPUT_DIRECTORY && GIT_SKIP_TESTS=$skip && export GIT_SKIP_TESTS && sane_unset GIT_TEST_FAIL_PREREQS && diff --git a/t/test-lib.sh b/t/test-lib.sh index 1f0505e412d8ff..5dc2e119a3546d 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -115,12 +115,21 @@ export UBSAN_OPTIONS # The TEST_OUTPUT_DIRECTORY will be overwritten via GIT-BUILD-OPTIONS. So in # case the caller has manually set up this variable via the environment we must -# make sure to not overwrite that value, and thus we save it into -# TEST_OUTPUT_DIRECTORY_OVERRIDE here. -if test -n "$TEST_OUTPUT_DIRECTORY" && test -z "$TEST_OUTPUT_DIRECTORY_OVERRIDE" -then - TEST_OUTPUT_DIRECTORY_OVERRIDE=$TEST_OUTPUT_DIRECTORY -fi +# make sure to not overwrite that value, and thus we save it here and restore +# it after sourcing GIT-BUILD-OPTIONS. +# +# It is important that existing overrides go after what we load from the +# environment. We want them to take precedence, and shell assignments +# will overwrite, giving us last-one-wins semantics. +GIT_TEST_ENV_OVERRIDES="$(env | + sed -n "/^TEST_OUTPUT_DIRECTORY=/{ + # escape all single-quotes in the value + s/'/'\\\\''/g + # turn this into an eval-able assignment + s/^\\([^=]*=\\)\\(.*\\)/\\1'\\2'/p + }") + $GIT_TEST_ENV_OVERRIDES +" if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS then @@ -137,14 +146,7 @@ then TEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY fi -# In t0000, we need to override test directories of nested testcases. In case -# the developer has TEST_OUTPUT_DIRECTORY part of his build options, then we'd -# reset this value to instead contain what the developer has specified. We thus -# have this knob to allow overriding the directory. -if test -n "${TEST_OUTPUT_DIRECTORY_OVERRIDE}" -then - TEST_OUTPUT_DIRECTORY="${TEST_OUTPUT_DIRECTORY_OVERRIDE}" -fi +eval "$GIT_TEST_ENV_OVERRIDES" # Disallow the use of abbreviated options in the test suite by default if test -z "${GIT_TEST_DISALLOW_ABBREVIATED_OPTIONS}" From c026accb5ae644cb752001c8d88e7070a73935ef Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 17 Dec 2017 23:44:10 -0500 Subject: [PATCH 060/156] rename sha1_object to process_object Signed-off-by: Jeff King --- builtin/index-pack.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 6b2a87e2d39355..755c6ff045bcb6 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -880,9 +880,9 @@ static void do_record_outgoing_links(struct object *obj) } } -static void sha1_object(const void *data, struct object_entry *obj_entry, - unsigned long size, enum object_type type, - const struct object_id *oid) +static void process_object(const void *data, struct object_entry *obj_entry, + unsigned long size, enum object_type type, + const struct object_id *oid) { void *new_data = NULL; int collision_test_needed = 0; @@ -1069,8 +1069,8 @@ static struct base_data *resolve_delta(struct object_entry *delta_obj, bad_object(delta_obj->idx.offset, _("failed to apply delta")); hash_object_file(the_hash_algo, result_data, result_size, delta_obj->real_type, &delta_obj->idx.oid); - sha1_object(result_data, NULL, result_size, delta_obj->real_type, - &delta_obj->idx.oid); + process_object(result_data, NULL, result_size, delta_obj->real_type, + &delta_obj->idx.oid); result = make_base(delta_obj, base); result->data = result_data; @@ -1281,8 +1281,8 @@ static void parse_pack_objects(unsigned char *hash) obj->real_type = OBJ_BAD; nr_delays++; } else - sha1_object(data, NULL, obj->size, obj->type, - &obj->idx.oid); + process_object(data, NULL, obj->size, obj->type, + &obj->idx.oid); free(data); display_progress(progress, i+1); } @@ -1310,8 +1310,8 @@ static void parse_pack_objects(unsigned char *hash) if (obj->real_type != OBJ_BAD) continue; obj->real_type = obj->type; - sha1_object(NULL, obj, obj->size, obj->type, - &obj->idx.oid); + process_object(NULL, obj, obj->size, obj->type, + &obj->idx.oid); nr_delays--; } if (nr_delays) From 5775b089362cd201bd6db078fd9ec6d88b35be4c Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:16:53 -0400 Subject: [PATCH 061/156] t0600: fix misplaced top-level ref We have a test that tries to manipulate the branch refs/heads/referrent, mostly using either the fully qualified refname or via git-branch (which implies refs/heads/). However, one call to update-ref uses the unqualified name, meaning we were quietly creating ".git/referrent", which was otherwise unused by the test. Fix this to specify refs/heads/referrent. I _think_ it actually doesn't affect the test outcome either way. The point of the test is that expiring reflogs for "the_symref" should not fail when we cannot lock "refs/heads/referrent" that it points to (because we have created a fake .lock file for it). And that is true even if the "referrent" file does not even exist. After all, the process holding the lock could be in the act of creating it. So I suspect this "update-ref" line could just be dropped entirely. Which you can verify by going back to its origin in 41d796ed5c (refs: on symref reflog expire, lock symref not referrent, 2016-04-07) and removing it. The test fails without the matching code change and passes with it. But I think it's worth keeping the update-ref call, as it creates a situation which is more likely to match what we'd see in the real world. Signed-off-by: Jeff King --- t/t0600-reffiles-backend.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh index bbbf6fa4228cd7..4fb6988f493994 100755 --- a/t/t0600-reffiles-backend.sh +++ b/t/t0600-reffiles-backend.sh @@ -338,7 +338,7 @@ test_expect_success SHA1 'parsing reverse reflogs at BUFSIZ boundaries' ' test_expect_success 'reflog expire operates on symref not referrent' ' git branch --create-reflog the_symref && git branch --create-reflog referrent && - git update-ref referrent HEAD && + git update-ref refs/heads/referrent HEAD && git symbolic-ref refs/heads/the_symref refs/heads/referrent && test_when_finished "rm -f .git/refs/heads/referrent.lock" && touch .git/refs/heads/referrent.lock && From b016eb3c044de29c943f41462e287cfe513937e5 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 2 Jun 2026 04:25:34 -0400 Subject: [PATCH 062/156] perf-lib: preserve GIT_PERF_* across test re-exec perf-lib saves GIT_PERF_* from the environment before sourcing GIT-BUILD-OPTIONS, then restores them after test-lib.sh returns. This usually lets values supplied by the user take precedence over those in GIT-BUILD-OPTIONS. But test-lib does not always return. Options such as --tee and --verbose-log cause it to run the test script again in a child process and then exit. At that point GIT-BUILD-OPTIONS has already overwritten the original GIT_PERF_* values, while perf-lib has not yet restored them. The child therefore inherits the overwritten values and cannot recover the user's settings. This is particularly confusing with t/perf/run. It obtains GIT_TEST_OPTS from GIT-BUILD-OPTIONS, so a --verbose-log setting left by an earlier build can silently make a run ignore variables such as GIT_PERF_LARGE_REPO. Put the saved GIT_PERF_* assignments in GIT_TEST_ENV_OVERRIDES instead. test-lib restores them immediately after sourcing GIT-BUILD-OPTIONS, before it may re-execute the test. The child then inherits the intended values, and perf-lib no longer needs a later restoration step. --- t/perf/perf-lib.sh | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 2ac007888e806a..2fe847f3ef7a4c 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -26,9 +26,9 @@ TEST_NO_CREATE_REPO=t TEST_NO_MALLOC_CHECK=t # GIT-BUILD-OPTIONS, sourced by test-lib.sh, overwrites the `GIT_PERF_*` -# values that are set by the user (if any). Let's stash them away as -# `eval`-able assignments. -git_perf_settings="$(env | +# values that are set by the user (if any). Let's stash them away +# in a form that test-lib.sh will restore automatically. +GIT_TEST_ENV_OVERRIDES="$(env | sed -n "/^GIT_PERF_/{ # escape all single-quotes in the value s/'/'\\\\''/g @@ -61,9 +61,6 @@ fi : ${TEST_OUTPUT_DIRECTORY:=$perf_dir} . "$GIT_SOURCE_DIR"/t/test-lib.sh -# Then restore GIT_PERF_* settings. -eval "$git_perf_settings" - unset GIT_CONFIG_NOSYSTEM GIT_CONFIG_SYSTEM="$TEST_DIRECTORY/perf/config" export GIT_CONFIG_SYSTEM From 50af012da7767fde6d88297d17e93d833f1bccad Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 04:43:47 -0500 Subject: [PATCH 063/156] index-pack: add --unpack option The goal is to reduce duplicate functionality from unpack-objects, so that it doesn't lag behind index-pack on features, including: - index-pack resolves deltas in parallel (maybe less important since we only use unpack-objects when there are few objects) - index-pack can stream large blobs instead of putting them in-memory A few things are not quite ideal, though: - we actually _do_ end up expanding large blobs anyway during the unpack phase, because write_sha1_file() insists on having all of the content at once (and the bulk checkin stuff insists on having a descriptor). This should be fixable with a better interface to sha1_file.c. - a normal unpack hashes the data twice (once to find out its sha1, and once on write to make sure it hasn't changed). We add a _third_ one here, since we have no way to tell write_sha1_file() that we already know the hash. This is again an interface problem. - we have to take a lock during the object write because write_sha1_file isn't thread-safe. This should be trivial to fix (the issue is the static buffer in sha1_filename) - we still spool the entire pack to disk, and then create loose objects from that spooled data. We could do better for various classes: - thin bases definitely do _not_ need copied; by definition we already have them elsewhere on disk - base objects do not need to be spooled either. We should be able to unpack them immediately and use the unpacked data later. We'd still need to create some kind of marker in the object_entry array to indicate that we got the object. One difficulty is large blobs, which we index but leave for later processing. We'd have to copy them directly into a loose object (or arguably even into a pack, similar to bulk checkin). - deltas which can be resolved immediately could do so. This is where things start to get dicey, since index-pack doesn't bother to even look for a base until the resolution phase. But in practice _most_ deltas are of this form, since the ofs format requires that bases come before deltas. - unresolved deltas (waiting on a ref_delta somewhere in the chain) must be spooled or buffered somewhere. Unpack-objects does this in memory. Spooling to disk is actually quite reasonable, though memory would be fine, too. Signed-off-by: Jeff King --- builtin/index-pack.c | 55 ++++++++++++++++++++-------- t/meson.build | 1 + t/t5336-index-pack-loose.sh | 71 +++++++++++++++++++++++++++++++++++++ 3 files changed, 112 insertions(+), 15 deletions(-) create mode 100755 t/t5336-index-pack-loose.sh diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 755c6ff045bcb6..1927d886cc5579 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -142,6 +142,7 @@ static const char *progress_title; static int show_resolving_progress; static int show_stat; static int check_self_contained_and_connected; +static int unpack_to_loose; static struct progress *progress; @@ -975,6 +976,21 @@ static void process_object(const void *data, struct object_entry *obj_entry, read_unlock(); } + if (unpack_to_loose) { + struct object_id dummy; + + /* XXX This will expand too-large objects! */ + if (!data) + data = new_data = get_data_from_pack(obj_entry); + + /* XXX extra unnecessary hash! */ + /* XXX there should be a thread-safe write_sha1_file! */ + read_lock(); + if (odb_write_object(the_repository->objects, data, size, type, &dummy) < 0) + die("failed to write object %s", oid_to_hex(oid)); + read_unlock(); + } + free(new_data); } @@ -1376,7 +1392,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha return; } - if (fix_thin_pack) { + if (fix_thin_pack || unpack_to_loose) { struct hashfile *f; unsigned char read_hash[GIT_MAX_RAWSZ], tail_hash[GIT_MAX_RAWSZ]; struct strbuf msg = STRBUF_INIT; @@ -1889,7 +1905,7 @@ int cmd_index_pack(int argc, struct repository *repo) { int i, fix_thin_pack = 0, verify = 0, stat_only = 0, rev_index; - const char *curr_index; + const char *curr_index = NULL; char *curr_rev_index = NULL; const char *index_name = NULL, *pack_name = NULL, *rev_index_name = NULL; const char *keep_msg = NULL; @@ -2008,6 +2024,8 @@ int cmd_index_pack(int argc, rev_index = 1; } else if (!strcmp(arg, "--no-rev-index")) { rev_index = 0; + } else if (!strcmp(arg, "--unpack")) { + unpack_to_loose = 1; } else usage(index_pack_usage); continue; @@ -2097,18 +2115,20 @@ int cmd_index_pack(int argc, if (show_stat) show_pack_info(stat_only); - ALLOC_ARRAY(idx_objects, nr_objects); - for (i = 0; i < nr_objects; i++) - idx_objects[i] = &objects[i].idx; - curr_index = write_idx_file(the_repository, index_name, idx_objects, - nr_objects, &opts, pack_hash); - if (rev_index) - curr_rev_index = write_rev_file(the_repository, rev_index_name, - idx_objects, nr_objects, - pack_hash, opts.flags); - free(idx_objects); - - if (!verify) + if (!unpack_to_loose) { + ALLOC_ARRAY(idx_objects, nr_objects); + for (i = 0; i < nr_objects; i++) + idx_objects[i] = &objects[i].idx; + curr_index = write_idx_file(the_repository, index_name, idx_objects, + nr_objects, &opts, pack_hash); + if (rev_index) + curr_rev_index = write_rev_file(the_repository, rev_index_name, + idx_objects, nr_objects, + pack_hash, opts.flags); + free(idx_objects); + } + + if (!verify && !unpack_to_loose) final(pack_name, curr_pack, index_name, curr_index, rev_index_name, curr_rev_index, @@ -2139,8 +2159,13 @@ int cmd_index_pack(int argc, free(objects); strbuf_release(&index_name_buf); strbuf_release(&rev_index_name_buf); - if (!pack_name) + if (!pack_name) { + if (unpack_to_loose) { + close(output_fd); + unlink_or_warn(curr_pack); + } free((void *) curr_pack); + } if (!index_name) free((void *) curr_index); free(curr_rev_index); diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..b96fffc7245b5d 100644 --- a/t/meson.build +++ b/t/meson.build @@ -639,6 +639,7 @@ integration_tests = [ 't5333-pseudo-merge-bitmaps.sh', 't5334-incremental-multi-pack-index.sh', 't5335-compact-multi-pack-index.sh', + 't5336-index-pack-loose.sh', 't5351-unpack-large-objects.sh', 't5400-send-pack.sh', 't5401-update-hooks.sh', diff --git a/t/t5336-index-pack-loose.sh b/t/t5336-index-pack-loose.sh new file mode 100755 index 00000000000000..bbf42530cdcb9a --- /dev/null +++ b/t/t5336-index-pack-loose.sh @@ -0,0 +1,71 @@ +#!/bin/sh + +test_description='index-pack --unpack' +. ./test-lib.sh + +mktmp () { + test_when_finished "rm -rf tmp" && + git init tmp +} + +verify_objects () { + git -C tmp rev-list --objects $head >actual && + test_cmp rev-list.expect actual +} + +verify_packs () { + find tmp/.git/objects/pack/ -name '*.pack' >actual && + test_line_count = $1 actual +} + +test_expect_success 'create some objects' ' + test-tool genrandom foo 4096 >file && + git add . && + git commit -m one && + + echo extra >>file && + git add . && + git commit -m two && + + head=$(git rev-parse HEAD) && + git rev-list --objects HEAD >rev-list.expect +' + +test_expect_success 'create some packs' ' + git pack-objects --all --stdout >ref-delta.pack && + git pack-objects --all --stdout --delta-base-offset >ofs-delta.pack && + + git pack-objects --revs --stdout >base.pack <<-\EOF && + HEAD^ + EOF + + git pack-objects --revs --stdout --thin >thin.pack <<-\EOF + HEAD + --not + HEAD^ + EOF +' + +test_expect_success 'unpack ref-delta.pack' ' + mktmp && + git -C tmp index-pack --stdin --unpack Date: Mon, 29 Apr 2024 04:21:29 -0400 Subject: [PATCH 064/156] t5619: use fully qualified refname for branch We use test_commit_bulk to update the ref "main". But without qualification, that is really creating ".git/main", not an actual branch in "refs/heads/". In the end it does not really matter, because after creating "main" its only purpose is for us to point HEAD to its same commit id. And that works regardless of how we named it (the later call to update-ref just calls it "main", but that is OK; we are resolving it to an oid there, so our DWIM logic will find it in "refs/heads/"). Since it seems like going outside of "refs/" was accidental here, let's do the more normal thing and just use a branch. That makes the test less confusing and will future proof us against tighter restrictions of top-level ref names. Signed-off-by: Jeff King --- t/t5619-clone-local-ambiguous-transport.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t5619-clone-local-ambiguous-transport.sh b/t/t5619-clone-local-ambiguous-transport.sh index e4218e5c7f9f50..e3464bd82b49f0 100755 --- a/t/t5619-clone-local-ambiguous-transport.sh +++ b/t/t5619-clone-local-ambiguous-transport.sh @@ -21,7 +21,7 @@ test_expect_success 'setup' ' echo "secret" >sensitive/secret && git init --bare "$REPO" && - test_commit_bulk -C "$REPO" --ref=main 1 && + test_commit_bulk -C "$REPO" --ref=refs/heads/main 1 && git -C "$REPO" update-ref HEAD main && git -C "$REPO" update-server-info && From d722ff46d81335f9dbd4b1bd26a3000d0124c1ad Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 13 Aug 2026 01:15:05 -0400 Subject: [PATCH 065/156] perf-lib: export TEST_OUTPUT_DIRECTORY This is necessary so that "env" sees it during our stash/recover. It sure would be nice if there was a way to access these variables reliably from within the shell. I guess we could use "set", but its output is potentially weird for multiline stuff. But then again, so is env. It might be nicer if we just used "set" to save and reload the whole variable state. --- t/perf/perf-lib.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 2fe847f3ef7a4c..2ced13d049f519 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -59,6 +59,7 @@ fi . "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS : ${TEST_OUTPUT_DIRECTORY:=$perf_dir} +export TEST_OUTPUT_DIRECTORY . "$GIT_SOURCE_DIR"/t/test-lib.sh unset GIT_CONFIG_NOSYSTEM From 35663e2094e5761197d08fbe23e2febb92876077 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 05:20:28 -0500 Subject: [PATCH 066/156] implement unpack-limit Signed-off-by: Jeff King --- builtin/index-pack.c | 6 ++++++ t/t5336-index-pack-loose.sh | 14 ++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 1927d886cc5579..619857fa870393 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -143,6 +143,7 @@ static int show_resolving_progress; static int show_stat; static int check_self_contained_and_connected; static int unpack_to_loose; +static unsigned long unpack_limit; static struct progress *progress; @@ -2026,6 +2027,9 @@ int cmd_index_pack(int argc, rev_index = 0; } else if (!strcmp(arg, "--unpack")) { unpack_to_loose = 1; + } else if (skip_prefix(arg, "--unpack-limit=", &arg)) { + if (!git_parse_ulong(arg, &unpack_limit)) + die("--unpack-limit expects a non-negative integer"); } else usage(index_pack_usage); continue; @@ -2098,6 +2102,8 @@ int cmd_index_pack(int argc, curr_pack = open_pack_file(pack_name); parse_pack_header(); + if (nr_objects < unpack_limit) + unpack_to_loose = 1; CALLOC_ARRAY(objects, st_add(nr_objects, 1)); if (show_stat) CALLOC_ARRAY(obj_stat, st_add(nr_objects, 1)); diff --git a/t/t5336-index-pack-loose.sh b/t/t5336-index-pack-loose.sh index bbf42530cdcb9a..05365e1845b6db 100755 --- a/t/t5336-index-pack-loose.sh +++ b/t/t5336-index-pack-loose.sh @@ -68,4 +68,18 @@ test_expect_success 'unpack base and thin packs' ' verify_packs 0 ' +test_expect_success 'unpack limit (under)' ' + mktmp && + git -C tmp index-pack --stdin --unpack-limit=10 Date: Mon, 29 Apr 2024 04:42:01 -0400 Subject: [PATCH 067/156] t0610: pass fully qualified branch name to update-ref We create several sample branches by doing "git update-ref branch-$i". But update-ref requires a fully qualified name, so this is creating "branch-$i" at the root of the ref namespace, not actual branches in "refs/heads/". The test doesn't care either way. It is about reftable compaction, and the ref creation is just to trigger the creation of new reftables. But to future-proof it against update-ref getting more picky about creating top-level refs, let's do the more normal thing and create actual branches. --- t/t0610-reftable-basics.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 35e98b43dba035..5d2d8925df0d43 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -347,11 +347,11 @@ test_expect_success 'ref transaction: env var disables compaction' ' for i in $(test_seq $iterations) do GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ - git -C repo update-ref branch-$i HEAD || return 1 + git -C repo update-ref refs/heads/branch-$i HEAD || return 1 done && test_line_count = $expected repo/.git/reftable/tables.list && - git -C repo update-ref foo HEAD && + git -C repo update-ref refs/heads/foo HEAD && test_line_count -lt $expected repo/.git/reftable/tables.list ' From 87a467b48c22a003218575535d5d8e59430036c1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 13 Aug 2026 01:21:59 -0400 Subject: [PATCH 068/156] save/restore all of GIT_TEST_ too This makes: GIT_TEST_CMP=whatever ./t1234-foo.sh work again. --- t/test-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 5dc2e119a3546d..80ec4acccb0827 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -122,7 +122,7 @@ export UBSAN_OPTIONS # environment. We want them to take precedence, and shell assignments # will overwrite, giving us last-one-wins semantics. GIT_TEST_ENV_OVERRIDES="$(env | - sed -n "/^TEST_OUTPUT_DIRECTORY=/{ + sed -n "/^\(TEST_OUTPUT_DIRECTORY=\|GIT_TEST_\)/{ # escape all single-quotes in the value s/'/'\\\\''/g # turn this into an eval-able assignment From 6810d7c9b9318602cb1eb9d05fbd3c5bb4f60dbd Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 05:56:43 -0500 Subject: [PATCH 069/156] receive-pack: never call unpack-objects We can use "index-pack --unpack-limit" instead, which avoids having to sniff the pack header. [This code has moved to object-file.c, but receive-pack is still the only caller]. Signed-off-by: Jeff King --- object-file.c | 172 +++++++++++++++++--------------------------------- 1 file changed, 57 insertions(+), 115 deletions(-) diff --git a/object-file.c b/object-file.c index a4cbf8b081df1e..d94e5c6b77f513 100644 --- a/object-file.c +++ b/object-file.c @@ -1296,32 +1296,6 @@ static int odb_transaction_files_commit(struct odb_transaction *base) return 0; } -static const char *parse_pack_header(struct pack_header *hdr, int pack_fd) -{ - switch (read_pack_header(pack_fd, hdr)) { - case PH_ERROR_EOF: - return "eof before pack header was fully read"; - - case PH_ERROR_PACK_SIGNATURE: - return "protocol error (pack signature mismatch detected)"; - - case PH_ERROR_PROTOCOL: - return "protocol error (pack version unsupported)"; - - default: - return "unknown error in parse_pack_header"; - - case 0: - return NULL; - } -} - -static void push_header_arg(struct strvec *args, struct pack_header *hdr) -{ - strvec_pushf(args, "--pack_header=%"PRIu32",%"PRIu32, - ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); -} - static unsigned int get_unpack_limit(struct repository *repo, enum odb_transaction_flags flags) { @@ -1344,18 +1318,10 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base, container_of(base, struct odb_transaction_files, base); struct repository *repo = base->source->odb->repo; struct child_process child = CHILD_PROCESS_INIT; - struct pack_header hdr; - const char *hdr_err; int err_fd = opts->err_fd; int status; - - hdr_err = parse_pack_header(&hdr, pack_fd); - if (hdr_err) { - if (err_fd > 0) - close(err_fd); - strbuf_addstr(err_msg, hdr_err); - return -1; - } + char hostname[HOST_NAME_MAX + 1]; + char *lockfile; if (opts->shallow_file) { strvec_push(&child.args, "--shallow-file"); @@ -1364,89 +1330,65 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base, odb_transaction_env(base, &child.env); - if (ntohl(hdr.hdr_entries) < get_unpack_limit(repo, transaction->flags)) { - strvec_push(&child.args, "unpack-objects"); - push_header_arg(&child.args, &hdr); - if (opts->quiet) - strvec_push(&child.args, "-q"); - if (opts->fsck_objects) - strvec_pushf(&child.args, "--strict%s", - opts->fsck_msg_types); - if (opts->max_input_size) - strvec_pushf(&child.args, "--max-input-size=%"PRIuMAX, - (uintmax_t)opts->max_input_size); - child.no_stdout = 1; - child.in = pack_fd; - child.err = err_fd; - child.git_cmd = 1; - status = run_command(&child); - if (status) { - strbuf_addstr(err_msg, "unpack-objects abnormal exit"); - return -1; - } - } else { - char hostname[HOST_NAME_MAX + 1]; - char *lockfile; - - strvec_pushl(&child.args, "index-pack", "--stdin", NULL); - push_header_arg(&child.args, &hdr); - - if (xgethostname(hostname, sizeof(hostname))) - xsnprintf(hostname, sizeof(hostname), "localhost"); - strvec_pushf(&child.args, - "--keep=receive-pack %"PRIuMAX" on %s", - (uintmax_t)getpid(), - hostname); - - if (!opts->quiet && err_fd) - strvec_push(&child.args, "--show-resolving-progress"); - if (err_fd) - strvec_push(&child.args, "--report-end-of-input"); - if (opts->fsck_objects) - strvec_pushf(&child.args, "--strict%s", - opts->fsck_msg_types); - if (!opts->reject_thin) - strvec_push(&child.args, "--fix-thin"); - if (opts->max_input_size) - strvec_pushf(&child.args, "--max-input-size=%"PRIuMAX, - (uintmax_t)opts->max_input_size); - child.out = -1; - child.in = pack_fd; - child.err = err_fd; - child.git_cmd = 1; - status = start_command(&child); - if (status) { - strbuf_addstr(err_msg, "index-pack fork failed"); - return -1; - } - - /* - * The lockfile filepath is expected to be the final location of - * the ".keep" file after being migrated to the main ODB source. - * This ensures the lockfile can be found and removed later - * after the ODB transaction has been committed. - */ - lockfile = index_pack_lockfile(base->source, child.out, NULL); - if (lockfile) { - ALLOC_GROW(transaction->pack_lockfiles, - transaction->pack_lockfiles_nr + 1, - transaction->pack_lockfiles_alloc); - transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] = - register_tempfile(lockfile); - free(lockfile); - } - close(child.out); + strvec_pushl(&child.args, "index-pack", "--stdin", NULL); + strvec_pushf(&child.args, "--unpack-limit=%d", + get_unpack_limit(repo, transaction->flags)); + + if (xgethostname(hostname, sizeof(hostname))) + xsnprintf(hostname, sizeof(hostname), "localhost"); + strvec_pushf(&child.args, + "--keep=receive-pack %"PRIuMAX" on %s", + (uintmax_t)getpid(), + hostname); + + if (!opts->quiet && err_fd) + strvec_push(&child.args, "--show-resolving-progress"); + if (err_fd) + strvec_push(&child.args, "--report-end-of-input"); + if (opts->fsck_objects) + strvec_pushf(&child.args, "--strict%s", + opts->fsck_msg_types); + if (!opts->reject_thin) + strvec_push(&child.args, "--fix-thin"); + if (opts->max_input_size) + strvec_pushf(&child.args, "--max-input-size=%"PRIuMAX, + (uintmax_t)opts->max_input_size); + child.out = -1; + child.in = pack_fd; + child.err = err_fd; + child.git_cmd = 1; + status = start_command(&child); + if (status) { + strbuf_addstr(err_msg, "index-pack fork failed"); + return -1; + } - status = finish_command(&child); - if (status) { - strbuf_addstr(err_msg, "index-pack abnormal exit"); - return -1; - } + /* + * The lockfile filepath is expected to be the final location of + * the ".keep" file after being migrated to the main ODB source. + * This ensures the lockfile can be found and removed later + * after the ODB transaction has been committed. + */ + lockfile = index_pack_lockfile(base->source, child.out, NULL); + if (lockfile) { + ALLOC_GROW(transaction->pack_lockfiles, + transaction->pack_lockfiles_nr + 1, + transaction->pack_lockfiles_alloc); + transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] = + register_tempfile(lockfile); + free(lockfile); + } + close(child.out); - odb_source_prepare(transaction->quarantine, - ODB_PREPARE_FLUSH_CACHES); + status = finish_command(&child); + if (status) { + strbuf_addstr(err_msg, "index-pack abnormal exit"); + return -1; } + odb_source_prepare(transaction->quarantine, + ODB_PREPARE_FLUSH_CACHES); + return 0; } From a425b9a4de4601de893f3a5114af3d8055e74c0a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:23:02 -0400 Subject: [PATCH 070/156] refs: disallow dash in root ref syntax Our is_root_ref_syntax() function allows upper-case letters, underscore ("_") and dash ("-"). Our glossary definition was historically vague on the allowed punctuation, but was tightened to just "upper-case characters or underscores" in 74b50a5881 (Documentation/glossary: define root refs as refs, 2024-05-15). I don't think we have ever used a root ref within Git with a dash, and the existing open-coded syntax check in refname_is_safe() allows only underscores. This logic comes from 266b18273a (refs: add ref_type function, 2015-07-31), but I couldn't find any comment on the dash in the commit message or the list discussion. It's used mostly for is_root_ref(), which further requires that the name either end in "_HEAD" or be one of a specific set of "irregular" root refs. So I don't think we'd ever see a dash in the real world (you'd need to have "FOO-BAR_HEAD"). And because of the rules in refname_is_safe(), such a ref would not be fully functional (you couldn't delete it!). Let's tighten this up now so that the function is consistent with (and can be used in) other spots. Signed-off-by: Jeff King --- refs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/refs.c b/refs.c index 92d5df5b71fa4b..534b05229d363b 100644 --- a/refs.c +++ b/refs.c @@ -904,7 +904,7 @@ static int is_root_ref_syntax(const char *refname) const char *c; for (c = refname; *c; c++) { - if (!isupper(*c) && *c != '-' && *c != '_') + if (!isupper(*c) && *c != '_') return 0; } From 34cf25144711003ba2f77c1ce5263e46ca27da65 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:37:39 -0400 Subject: [PATCH 071/156] t1401: avoid pointless use of root ref We check that symbolic-ref fails when trying to delete a symref that doesn't exist. We use the name "FOO", but that does not conform to our root-ref syntax. But there's no need for this to be a root ref. We can use a name in refs/heads/, which future-proofs us against increased enforcement of the syntax. And while we're at it, let's use a name which more obviously explains its purpose ("does-not-exist"). --- t/t1401-symbolic-ref.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh index a2a7e947164c2a..bf703498f1a2e5 100755 --- a/t/t1401-symbolic-ref.sh +++ b/t/t1401-symbolic-ref.sh @@ -60,9 +60,9 @@ test_expect_success 'symbolic-ref can delete dangling symref' ' ' reset_to_sane -test_expect_success 'symbolic-ref fails to delete missing FOO' ' - echo "fatal: Cannot delete FOO, not a symbolic ref" >expect && - test_must_fail git symbolic-ref -d FOO >actual 2>&1 && +test_expect_success 'symbolic-ref fails to delete missing ref' ' + echo "fatal: Cannot delete refs/heads/does-not-exist, not a symbolic ref" >expect && + test_must_fail git symbolic-ref -d refs/heads/does-not-exist >actual 2>&1 && test_cmp expect actual ' reset_to_sane From 6f5566a1937dc6c0fc3617b24204178b0419a341 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 13 Aug 2026 01:27:06 -0400 Subject: [PATCH 072/156] test-lib: just use set for overrides This just dumps all of the variables in a readable form, and then re-evals it after we load the build options. This will do noop writes of many variables, but that should be OK. And it's probably still faster than running a separate env/sed process. It also avoids problems with multiline variables. We wouldn't expect them, but we are open to injection based on arbitrary variables, like: UNTRUSTED_VAR='this has a newline TRUSTED_VAR=yikes that allows for environment injection' --- t/test-lib.sh | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 80ec4acccb0827..474d93ce0b654d 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -121,14 +121,11 @@ export UBSAN_OPTIONS # It is important that existing overrides go after what we load from the # environment. We want them to take precedence, and shell assignments # will overwrite, giving us last-one-wins semantics. -GIT_TEST_ENV_OVERRIDES="$(env | - sed -n "/^\(TEST_OUTPUT_DIRECTORY=\|GIT_TEST_\)/{ - # escape all single-quotes in the value - s/'/'\\\\''/g - # turn this into an eval-able assignment - s/^\\([^=]*=\\)\\(.*\\)/\\1'\\2'/p - }") +GIT_TEST_ENV_OVERRIDES="$( + unset GIT_TEST_ENV_OVERRIDES + set $GIT_TEST_ENV_OVERRIDES +) " if test ! -f "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS From a84c74179636d0f50c1a989b85e04c9e167d4fe8 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 05:57:56 -0500 Subject: [PATCH 073/156] fetch-pack: never call unpack-objects Same as the previous commit, but note that setting the unpackLimit to "0" has a different meaning here. Signed-off-by: Jeff King --- fetch-pack.c | 102 ++++++++++++++++++++------------------------------- 1 file changed, 40 insertions(+), 62 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 5d0296f970e464..fc26bbfcdb6a21 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -964,10 +964,6 @@ static int get_pack(struct fetch_pack_args *args, struct oidset *gitmodules_oids) { struct async demux; - int do_keep = args->keep_pack; - const char *cmd_name; - struct pack_header header; - int pass_header = 0; struct child_process cmd = CHILD_PROCESS_INIT; int fsck_objects = 0; int ret; @@ -988,17 +984,6 @@ static int get_pack(struct fetch_pack_args *args, else demux.out = xd[0]; - if (!args->keep_pack && unpack_limit && !index_pack_args) { - - if (read_pack_header(demux.out, &header)) - die(_("protocol error: bad pack header")); - pass_header = 1; - if (ntohl(header.hdr_entries) < unpack_limit) - do_keep = 0; - else - do_keep = 1; - } - if (alternate_shallow_file) { strvec_push(&cmd.args, "--shallow-file"); strvec_push(&cmd.args, alternate_shallow_file); @@ -1006,51 +991,44 @@ static int get_pack(struct fetch_pack_args *args, fsck_objects = fetch_pack_fsck_objects(); - if (do_keep || args->from_promisor || index_pack_args || fsck_objects) { - if (pack_lockfiles || fsck_objects) - cmd.out = -1; - cmd_name = "index-pack"; - strvec_push(&cmd.args, cmd_name); - strvec_push(&cmd.args, "--stdin"); - if (!args->quiet && !args->no_progress) - strvec_push(&cmd.args, "-v"); - if (args->use_thin_pack) - strvec_push(&cmd.args, "--fix-thin"); - if ((do_keep || index_pack_args) && (args->lock_pack || unpack_limit)) - add_index_pack_keep_option(&cmd.args); - if (!index_pack_args && args->check_self_contained_and_connected) - strvec_push(&cmd.args, "--check-self-contained-and-connected"); - else - /* - * We cannot perform any connectivity checks because - * not all packs have been downloaded; let the caller - * have this responsibility. - */ - args->check_self_contained_and_connected = 0; - - if (args->from_promisor) - /* - * create_promisor_file() may be called afterwards but - * we still need index-pack to know that this is a - * promisor pack. For example, if transfer.fsckobjects - * is true, index-pack needs to know that .gitmodules - * is a promisor object (so that it won't complain if - * it is missing). - */ - strvec_push(&cmd.args, "--promisor"); - } - else { - cmd_name = "unpack-objects"; - strvec_push(&cmd.args, cmd_name); - if (args->quiet || args->no_progress) - strvec_push(&cmd.args, "-q"); + if (pack_lockfiles || fsck_objects) + cmd.out = -1; + strvec_push(&cmd.args, "index-pack"); + strvec_push(&cmd.args, "--stdin"); + if (!args->quiet && !args->no_progress) + strvec_push(&cmd.args, "-v"); + if (args->use_thin_pack) + strvec_push(&cmd.args, "--fix-thin"); + if ((args->keep_pack || index_pack_args) && (args->lock_pack || unpack_limit)) + add_index_pack_keep_option(&cmd.args); + if (!index_pack_args && args->check_self_contained_and_connected) + strvec_push(&cmd.args, "--check-self-contained-and-connected"); + else + /* + * We cannot perform any connectivity checks because + * not all packs have been downloaded; let the caller + * have this responsibility. + */ args->check_self_contained_and_connected = 0; - } - if (pass_header) - strvec_pushf(&cmd.args, "--pack_header=%"PRIu32",%"PRIu32, - ntohl(header.hdr_version), - ntohl(header.hdr_entries)); + if (args->from_promisor) + /* + * create_promisor_file() may be called afterwards but + * we still need index-pack to know that this is a + * promisor pack. For example, if transfer.fsckobjects + * is true, index-pack needs to know that .gitmodules + * is a promisor object (so that it won't complain if + * it is missing). + */ + strvec_push(&cmd.args, "--promisor"); + + if (args->keep_pack || args->from_promisor || index_pack_args) + ; /* never --unpack */ + else if (!unpack_limit) + strvec_push(&cmd.args, "--unpack"); + else + strvec_pushf(&cmd.args, "--unpack-limit=%d", unpack_limit+1); /* hackery? */ + if (fsck_objects) { if (args->from_promisor || index_pack_args) /* @@ -1072,8 +1050,8 @@ static int get_pack(struct fetch_pack_args *args, cmd.in = demux.out; cmd.git_cmd = 1; if (start_command(&cmd)) - die(_("fetch-pack: unable to fork off %s"), cmd_name); - if (do_keep && (pack_lockfiles || fsck_objects)) { + die(_("fetch-pack: unable to fork off index-pack")); + if (args->keep_pack && (pack_lockfiles || fsck_objects)) { int is_well_formed; char *pack_lockfile = index_pack_lockfile(the_repository->objects->sources, cmd.out, @@ -1099,7 +1077,7 @@ static int get_pack(struct fetch_pack_args *args, args->check_self_contained_and_connected && ret == 0; else - die(_("%s failed"), cmd_name); + die(_("index-pack failed")); if (use_sideband && finish_async(&demux)) die(_("error in sideband demultiplexer")); @@ -1109,7 +1087,7 @@ static int get_pack(struct fetch_pack_args *args, * Now that index-pack has succeeded, write the promisor file using the * obtained .keep filename if necessary */ - if (do_keep && pack_lockfiles && pack_lockfiles->nr && args->from_promisor) + if (args->keep_pack && pack_lockfiles && pack_lockfiles->nr && args->from_promisor) create_promisor_file(pack_lockfiles->items[0].string, sought, nr_sought); return 0; From 8824b4e54bdda00b58d5f491dbd090eccb372100 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 17 Jun 2014 15:45:40 +0200 Subject: [PATCH 074/156] index-pack: set up quarantine bits [explain reasons] Signed-off-by: Jeff King --- tmp-objdir.c | 23 +++++++++++++++++++++++ tmp-objdir.h | 3 +++ 2 files changed, 26 insertions(+) diff --git a/tmp-objdir.c b/tmp-objdir.c index 0eaa79ffd7cd44..63e0372ed4385b 100644 --- a/tmp-objdir.c +++ b/tmp-objdir.c @@ -336,3 +336,26 @@ struct odb_source *tmp_objdir_replace_primary_odb(struct tmp_objdir *t, return odb_set_temporary_primary_source(t->repo->objects, t->path.buf, will_destroy, &t->prev_source); } + +FILE *fopen_quarantine(const char *filename, const char *mode) +{ + const char *root; + char *path; + FILE *ret; + + root = getenv(GIT_QUARANTINE_ENVIRONMENT); + if (!root) + die("BUG: fopen_quarantine called without %s set", + GIT_QUARANTINE_ENVIRONMENT); + + path = xstrfmt("%s/%s", root, filename); + ret = xfopen(path, mode); + + free(path); + return ret; +} + +int git_is_quarantined(void) +{ + return !!getenv(GIT_QUARANTINE_ENVIRONMENT); +} diff --git a/tmp-objdir.h b/tmp-objdir.h index 81eb9274136e69..b3943219ad158c 100644 --- a/tmp-objdir.h +++ b/tmp-objdir.h @@ -70,4 +70,7 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *); struct odb_source *tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy); +FILE *fopen_quarantine(const char *filename, const char *mode); +int git_is_quarantined(void); + #endif /* TMP_OBJDIR_H */ From 0f15faf6d41f834c950e4fc47fbd750f66f4acd3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:23:21 -0400 Subject: [PATCH 075/156] refs: use is_root_ref_syntax() in refname_is_safe() The open-coded logic in refname_is_safe() predates the addition of is_root_ref_syntax(). But now that we have it, we can make use of it. This shortens the code, but also makes sure we use a consistent definition of the syntax. Signed-off-by: Jeff King --- refs.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/refs.c b/refs.c index 534b05229d363b..b35916f32d4186 100644 --- a/refs.c +++ b/refs.c @@ -169,6 +169,8 @@ void update_ref_namespace(enum ref_namespace namespace, char *ref) info->ref_updated = 1; } +static int is_root_ref_syntax(const char *refname); + /* * Try to read one refname component from the front of refname. * Return the length of the component found, or -1 if the component is @@ -403,12 +405,7 @@ int refname_is_safe(const char *refname) return result; } - do { - if (!isupper(*refname) && *refname != '_') - return 0; - refname++; - } while (*refname); - return 1; + return is_root_ref_syntax(refname); } /* From 11ac064790ca9f698c12b006f3d7facb2ba7777b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:40:08 -0400 Subject: [PATCH 076/156] t9300: follow root ref syntax for TEMP_TAG We fast-import a ref named TEMP_TAG at the root of the refs namespace (outside of "refs/"), but this violates our usual root-ref syntax. It is intentional that this is done at the root level. The test comes from ea08a6fd19 (Actually allow TAG_FIXUP branches in fast-import, 2007-08-02), which says: New test cases have been added to make sure we still detect very wrong branch names (e.g. containing [ or starting with .) and yet still permit reasonable names (e.g. TAG_FIXUP). Our definition of "reasonable" has tightened over the years, and TAG_FIXUP (or TEMP_TAG) do not match our allowed root-ref syntax (though we do not yet enforce it consistently). It must be TAG_FIXUP_HEAD. So let's use that, which I think matches the spirit of the test. While we're here, let's also swap out "rm -f" for "update-ref -d" in the cleanup code. The former was a noop when using the reftable backend. The leftover refs don't seem to affect downstream tests, but this matches the original intent. --- t/t9300-fast-import.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh index d9de2ef0d88bc8..18ceec3fb7087d 100755 --- a/t/t9300-fast-import.sh +++ b/t/t9300-fast-import.sh @@ -378,7 +378,7 @@ test_expect_success 'B: fail on invalid blob sha1' ' test_expect_success 'B: accept branch name "TEMP_TAG"' ' cat >input <<-INPUT_END && - commit TEMP_TAG + commit TEMP_TAG_HEAD committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE data < Date: Fri, 14 Aug 2026 00:33:42 -0400 Subject: [PATCH 077/156] limit ourselves to just test vars Unfortunately a regular "set" with bash will output a bunch of readonly variables like BASH_VERSINFO, causing bash to complain. Worse, in "set -e" mode, it will cause us to bail completely. So we have to either weed those out, or restrict ourselves to known-useful quantities. --- t/test-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/test-lib.sh b/t/test-lib.sh index 474d93ce0b654d..8148e2a88aed3b 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -123,7 +123,7 @@ export UBSAN_OPTIONS # will overwrite, giving us last-one-wins semantics. GIT_TEST_ENV_OVERRIDES="$( unset GIT_TEST_ENV_OVERRIDES - set + set | grep -E '^(GIT_TEST_|TEST_OUTPUT_DIRECTORY=)' $GIT_TEST_ENV_OVERRIDES ) " From e455ff88b792bb3333f3ccb99ef5d0c1bb7484d0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 05:58:53 -0500 Subject: [PATCH 078/156] sha1_file: drop read_pack_header() There are no more callers. Signed-off-by: Jeff King --- object-file.c | 15 --------------- pack.h | 5 ----- 2 files changed, 20 deletions(-) diff --git a/object-file.c b/object-file.c index d94e5c6b77f513..b49458a740b151 100644 --- a/object-file.c +++ b/object-file.c @@ -1023,21 +1023,6 @@ int index_path(struct index_state *istate, struct object_id *oid, return rc; } -int read_pack_header(int fd, struct pack_header *header) -{ - if (read_in_full(fd, header, sizeof(*header)) != sizeof(*header)) - /* "eof before pack header was fully read" */ - return PH_ERROR_EOF; - - if (header->hdr_signature != htonl(PACK_SIGNATURE)) - /* "protocol error (pack signature mismatch detected)" */ - return PH_ERROR_PACK_SIGNATURE; - if (!pack_version_ok(header->hdr_version)) - /* "protocol error (pack version unsupported)" */ - return PH_ERROR_PROTOCOL; - return 0; -} - int for_each_file_in_obj_subdir(unsigned int subdir_nr, struct strbuf *path, const struct git_hash_algo *algop, diff --git a/pack.h b/pack.h index ada506b5c5d0db..08c6003058a2cb 100644 --- a/pack.h +++ b/pack.h @@ -134,11 +134,6 @@ char *write_rev_file_order(struct repository *repo, int encode_in_pack_object_header(unsigned char *hdr, int hdr_len, enum object_type, uintmax_t); -#define PH_ERROR_EOF (-1) -#define PH_ERROR_PACK_SIGNATURE (-2) -#define PH_ERROR_PROTOCOL (-3) -int read_pack_header(int fd, struct pack_header *); - struct packing_data; struct hashfile *create_tmp_packfile(struct repository *repo, From 2afa84d4a673f38782b8a95461dd2ca8ac864453 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 18 Jun 2014 18:33:21 +0200 Subject: [PATCH 079/156] index-pack: Implement `--warn-object-size` Signed-off-by: Jeff King --- builtin/index-pack.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 6b2a87e2d39355..4ae2c07ed097e9 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -31,6 +31,7 @@ #include "run-command.h" #include "setup.h" #include "strvec.h" +#include "tmp-objdir.h" static const char index_pack_usage[] = "git index-pack [-v] [-o ] [--keep | --keep=] [--[no-]rev-index] [--verify] [--strict[==...]] [--fsck-objects[==...]] ( | --stdin [--fix-thin] [])"; @@ -162,6 +163,8 @@ static const char *curr_pack; static struct oidset outgoing_links = OIDSET_INIT; static int record_outgoing_links; +static unsigned long warn_object_size; + static struct thread_local_data *thread_data; static int nr_dispatched; static int threads_active; @@ -182,6 +185,10 @@ static pthread_mutex_t deepest_delta_mutex; #define deepest_delta_lock() lock_mutex(&deepest_delta_mutex) #define deepest_delta_unlock() unlock_mutex(&deepest_delta_mutex) +static pthread_mutex_t large_obj_mutex; +#define large_obj_lock() lock_mutex(&large_obj_mutex) +#define large_obj_unlock() unlock_mutex(&large_obj_mutex) + static pthread_key_t key; static inline void lock_mutex(pthread_mutex_t *mutex) @@ -205,6 +212,8 @@ static void init_thread(void) init_recursive_mutex(&read_mutex); pthread_mutex_init(&counter_mutex, NULL); pthread_mutex_init(&work_mutex, NULL); + if (warn_object_size) + pthread_mutex_init(&large_obj_mutex, NULL); if (show_stat) pthread_mutex_init(&deepest_delta_mutex, NULL); pthread_key_create(&key, NULL); @@ -225,6 +234,8 @@ static void cleanup_thread(void) pthread_mutex_destroy(&read_mutex); pthread_mutex_destroy(&counter_mutex); pthread_mutex_destroy(&work_mutex); + if (warn_object_size) + pthread_mutex_destroy(&large_obj_mutex); if (show_stat) pthread_mutex_destroy(&deepest_delta_mutex); for (i = 0; i < nr_threads; i++) @@ -233,6 +244,23 @@ static void cleanup_thread(void) free(thread_data); } +static void warn_large_object(const struct object_id *oid, unsigned long size) +{ + static FILE *logfile = NULL; + + large_obj_lock(); + + if (!logfile) + logfile = git_is_quarantined() ? fopen_quarantine(".large-objects", "w") : stderr; + + if (logfile == stderr) + fprintf(logfile, "large-object: "); + + fprintf(logfile, "%s %"PRIuMAX"\n", oid_to_hex(oid), (uintmax_t)size); + + large_obj_unlock(); +} + static int mark_link(struct object *obj, enum object_type type, void *data UNUSED, struct fsck_options *options UNUSED) @@ -889,6 +917,9 @@ static void sha1_object(const void *data, struct object_entry *obj_entry, assert(data || obj_entry); + if (warn_object_size && size > warn_object_size) + warn_large_object(oid, size); + if (startup_info->have_repository) { read_lock(); collision_test_needed = odb_has_object(the_repository->objects, oid, @@ -2008,6 +2039,8 @@ int cmd_index_pack(int argc, rev_index = 1; } else if (!strcmp(arg, "--no-rev-index")) { rev_index = 0; + } else if (skip_prefix(arg, "--warn-object-size=", &arg)) { + warn_object_size = strtoul(arg, NULL, 10); } else usage(index_pack_usage); continue; From bba8378446b1abab1071f630512a862c558df9d1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:33:26 -0400 Subject: [PATCH 080/156] check_refname_format(): add FULLY_QUALIFIED flag Before operating on a refname we get from a user, we usually check that it's syntactically valid. As a general rule, refs should be in the "refs/" namespace, the exception being root refs like HEAD, REBASE_HEAD, and so on (including pseudorefs like FETCH_HEAD). Those root refs should consist only of all-caps and underscore, but the syntactic rules are not enforced by check_refname_format(). So for example we will happily operate on a ref "foo/bar" that will use the file ".git/foo/bar" under the hood (when using the files backend, of course). Making things even more complicated, refname_is_safe() does enforce these syntax restrictions! When that function was added in d0f810f0bc (refs.c: allow listing and deleting badly named refs, 2014-09-03), we would have refused to work with such refs entirely. But we stopped calling it in every code path in 03afcbee9b (read_packed_refs: avoid double-checking sane refs, 2015-04-16). The rationale given in that commit is that check_refname_format() is supposed to contain a superset of the checks of refname_is_safe(). So the idea is that we usually would rely on the more-strict check_refname_format(), but for certain operations (e.g., deleting a ref) we want to allow invalid names as long as they are not unsafe (e.g., not escaping the on-disk "refs/" hierarchy). But the root ref handling violates that logic; check_refname_format() is more lenient than refname_is_safe(). So you can create "foo/bar" and read it, but you cannot delete it: $ git update-ref foo/bar HEAD $ git rev-parse foo/bar 747a29934757b7e695781e13e2511c43b951da2 $ git update-ref -d foo/bar error: refusing to update ref with bad name 'foo/bar' So we probably want check_ref_format() to learn the same syntactic restrictions that refname_is_safe() uses (namely insisting that anything outside of "refs/" matches the root ref syntax). The most obvious way to do that is simply to call refname_is_safe(). But the point of 03afcbee9b is that doing so is expensive. Without the syntactic restrictions of check_refname_format(), refname_is_safe() has to actually normalize the pathname to make sure it does not escape "refs/". That's redundant for us in check_refname_format(); we just need to make sure it either starts with "refs/" or uses the all-caps root-ref syntax. But wait, it gets more complicated! We also allow some special worktree refnames: "worktrees/foo/$X" and "main-worktree/$X". In that case we should only be checking "$X" (which should be either a root ref or start with "refs/"). We can use parse_worktree_ref(), which fairly efficiently gives us the "bare" refname (even for a non-worktree ref, where it returns the original name). And now when should this new logic kick in? Unfortunately we can't just do it all the time, because many callers pass in partial ref components to check_ref_format(). E.g., if they are thinking about making "refs/heads/foo", they'll pass us "foo". This is usually accompanied by the ALLOW_ONELEVEL flag. But we likewise can't take the absence of ALLOW_ONELEVEL as a hint that the name is fully qualified, because that flag is also used to indicate that root refs should be allowed! We need a new flag to tell these two situations apart. So let's add a FULLY_QUALIFIED flag that callers can use to ask us to enforce these syntactic rules. There are no callers yet, but we should be able to examine users of ALLOW_ONELEVEL, figure out which semantics they wanted, and convert as needed. Signed-off-by: Jeff King --- refs.c | 14 +++++++++++++- refs.h | 1 + 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/refs.c b/refs.c index b35916f32d4186..90b4dbffb53d95 100644 --- a/refs.c +++ b/refs.c @@ -279,6 +279,15 @@ static int check_or_sanitize_refname(const char *refname, int flags, { int component_len, component_count = 0; + if ((flags & REFNAME_FULLY_QUALIFIED)) { + const char *bare_ref; + + parse_worktree_ref(refname, NULL, NULL, &bare_ref); + if (!starts_with(bare_ref, "refs/") && + !is_root_ref_syntax(bare_ref)) + return -1; + } + if (!strcmp(refname, "@")) { /* Refname is a single character '@'. */ if (sanitized) @@ -313,8 +322,11 @@ static int check_or_sanitize_refname(const char *refname, int flags, else return -1; } - if (!(flags & REFNAME_ALLOW_ONELEVEL) && component_count < 2) + + if (!(flags & (REFNAME_ALLOW_ONELEVEL | REFNAME_FULLY_QUALIFIED)) && + component_count < 2) return -1; /* Refname has only one component. */ + return 0; } diff --git a/refs.h b/refs.h index 9979446d15fd3b..dbd1b5b810d071 100644 --- a/refs.h +++ b/refs.h @@ -695,6 +695,7 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat #define REFNAME_ALLOW_ONELEVEL 1 #define REFNAME_REFSPEC_PATTERN 2 +#define REFNAME_FULLY_QUALIFIED 4 /* * Return 0 iff refname has the correct format for a refname according From 564572f4d4ed404b842d63a2c2333be82b8370d3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:34:35 -0400 Subject: [PATCH 081/156] t1400: swap out PSEUDOREF for ROOT_HEAD Some old tests in t1400 use the name PSEUDOREF for some root refs outside of the "refs/" hierarchy. This term was refined in 6fd8037564 (Documentation/glossary: redefine pseudorefs as special refs, 2024-05-15) and these are no longer actually pseudorefs. Moreover, the name PSEUDOREF does not match our root ref syntax (but we do not yet enforce it consistently). Let's call this ROOT_HEAD, which future proofs against tighter enforcement and gives a more glossary-accurate name. --- t/t1400-update-ref.sh | 66 +++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index 269fdaa3ed71aa..cff410ec8910ba 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -503,61 +503,61 @@ test_expect_success 'git cat-file blob main@{2005-05-26 23:42}:F (expect OTHER)' test OTHER = $(git cat-file blob "main@{2005-05-26 23:42}:F") ' -# Test adding and deleting pseudorefs +# Test adding and deleting root refs -test_expect_success 'given old value for missing pseudoref, do not create' ' - test_must_fail git update-ref PSEUDOREF $A $B 2>err && - test_must_fail git rev-parse PSEUDOREF && +test_expect_success 'given old value for missing root ref, do not create' ' + test_must_fail git update-ref ROOT_HEAD $A $B 2>err && + test_must_fail git rev-parse ROOT_HEAD && test_grep "unable to resolve reference" err ' -test_expect_success 'create pseudoref' ' - git update-ref PSEUDOREF $A && - test $A = $(git show-ref -s --verify PSEUDOREF) +test_expect_success 'create root ref' ' + git update-ref ROOT_HEAD $A && + test $A = $(git show-ref -s --verify ROOT_HEAD) ' -test_expect_success 'overwrite pseudoref with no old value given' ' - git update-ref PSEUDOREF $B && - test $B = $(git show-ref -s --verify PSEUDOREF) +test_expect_success 'overwrite root ref with no old value given' ' + git update-ref ROOT_HEAD $B && + test $B = $(git show-ref -s --verify ROOT_HEAD) ' -test_expect_success 'overwrite pseudoref with correct old value' ' - git update-ref PSEUDOREF $C $B && - test $C = $(git show-ref -s --verify PSEUDOREF) +test_expect_success 'overwrite root ref with correct old value' ' + git update-ref ROOT_HEAD $C $B && + test $C = $(git show-ref -s --verify ROOT_HEAD) ' -test_expect_success 'do not overwrite pseudoref with wrong old value' ' - test_must_fail git update-ref PSEUDOREF $D $E 2>err && - test $C = $(git show-ref -s --verify PSEUDOREF) && +test_expect_success 'do not overwrite root ref with wrong old value' ' + test_must_fail git update-ref ROOT_HEAD $D $E 2>err && + test $C = $(git show-ref -s --verify ROOT_HEAD) && test_grep "cannot lock ref.*expected" err ' -test_expect_success 'delete pseudoref' ' - git update-ref -d PSEUDOREF && - test_must_fail git show-ref -s --verify PSEUDOREF +test_expect_success 'delete root ref' ' + git update-ref -d ROOT_HEAD && + test_must_fail git show-ref -s --verify ROOT_HEAD ' -test_expect_success 'do not delete pseudoref with wrong old value' ' - git update-ref PSEUDOREF $A && - test_must_fail git update-ref -d PSEUDOREF $B 2>err && - test $A = $(git show-ref -s --verify PSEUDOREF) && +test_expect_success 'do not delete root ref with wrong old value' ' + git update-ref ROOT_HEAD $A && + test_must_fail git update-ref -d ROOT_HEAD $B 2>err && + test $A = $(git show-ref -s --verify ROOT_HEAD) && test_grep "cannot lock ref.*expected" err ' -test_expect_success 'delete pseudoref with correct old value' ' - git update-ref -d PSEUDOREF $A && - test_must_fail git show-ref -s --verify PSEUDOREF +test_expect_success 'delete root ref with correct old value' ' + git update-ref -d ROOT_HEAD $A && + test_must_fail git show-ref -s --verify ROOT_HEAD ' -test_expect_success 'create pseudoref with old OID zero' ' - git update-ref PSEUDOREF $A $Z && - test $A = $(git show-ref -s --verify PSEUDOREF) +test_expect_success 'create root ref with old OID zero' ' + git update-ref ROOT_HEAD $A $Z && + test $A = $(git show-ref -s --verify ROOT_HEAD) ' -test_expect_success 'do not overwrite pseudoref with old OID zero' ' - test_when_finished git update-ref -d PSEUDOREF && - test_must_fail git update-ref PSEUDOREF $B $Z 2>err && - test $A = $(git show-ref -s --verify PSEUDOREF) && +test_expect_success 'do not overwrite root ref with old OID zero' ' + test_when_finished git update-ref -d ROOT_HEAD && + test_must_fail git update-ref ROOT_HEAD $B $Z 2>err && + test $A = $(git show-ref -s --verify ROOT_HEAD) && test_grep "already exists" err ' From 17021e61d7d1c6d5aebd85a51197c0be1fbff293 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 13 Aug 2026 01:31:09 -0400 Subject: [PATCH 082/156] just use set for perf-lib --- t/perf/perf-lib.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 2ced13d049f519..3f0a0e2db7f7a9 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -28,13 +28,7 @@ TEST_NO_MALLOC_CHECK=t # GIT-BUILD-OPTIONS, sourced by test-lib.sh, overwrites the `GIT_PERF_*` # values that are set by the user (if any). Let's stash them away # in a form that test-lib.sh will restore automatically. -GIT_TEST_ENV_OVERRIDES="$(env | - sed -n "/^GIT_PERF_/{ - # escape all single-quotes in the value - s/'/'\\\\''/g - # turn this into an eval-able assignment - s/^\\([^=]*=\\)\\(.*\\)/\\1'\\2'/p - }")" +GIT_TEST_ENV_OVERRIDES="$(set)" # While test-lib.sh computes the build directory for us, we also have to do the # same thing in order to locate the script via GIT-BUILD-OPTIONS in the first From 952190330ef3b93821b1a2747534789bddfcd2ae Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 04:42:02 -0500 Subject: [PATCH 083/156] index-pack: allow verifying over --stdin This obviously skips verification of the index, but allows you to see if the pack itself is valid. This is similar to unpack-objects' --dry-run. Signed-off-by: Jeff King --- builtin/index-pack.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 619857fa870393..e436ff93ff43bd 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -2071,9 +2071,7 @@ int cmd_index_pack(int argc, &rev_index_name_buf); } - if (verify) { - if (!index_name) - die(_("--verify with no packfile name given")); + if (verify && index_name) { read_idx_option(&opts, index_name); opts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT; } From 8a52df18367db58c9aebcfc72cca62db78cb1e12 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 10 Oct 2020 09:58:57 -0400 Subject: [PATCH 084/156] move sizep computation Signed-off-by: Jeff King --- packfile.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/packfile.c b/packfile.c index 4fa5fd67c8497f..87b454da94b4a9 100644 --- a/packfile.c +++ b/packfile.c @@ -1318,7 +1318,8 @@ int packed_object_info_with_index_pos(struct odb_source_packed *source, /* * We always get the representation type, but only convert it to - * a "real" type later if the caller is interested. + * a "real" type later if the caller is interested. Likewise... + * tbd. */ if (oi->contentp) { *oi->contentp = cache_or_unpack_entry(p->repo, p, obj_offset, @@ -1327,24 +1328,24 @@ int packed_object_info_with_index_pos(struct odb_source_packed *source, type = OBJ_BAD; } else if (oi->sizep || oi->typep || oi->delta_base_oid) { type = unpack_object_header(p, &w_curs, &curpos, &size); - } - if (!oi->contentp && oi->sizep) { - if (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) { - off_t tmp_pos = curpos; - off_t base_offset = get_delta_base(p, &w_curs, &tmp_pos, - type, obj_offset); - if (!base_offset) { - ret = -1; - goto out; - } - size = get_size_from_delta(p, &w_curs, tmp_pos); - if (size == 0) { - ret = -1; - goto out; + if (oi->sizep) { + if (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) { + off_t tmp_pos = curpos; + off_t base_offset = get_delta_base(p, &w_curs, &tmp_pos, + type, obj_offset); + if (!base_offset) { + ret = -1; + goto out; + } + size = get_size_from_delta(p, &w_curs, tmp_pos); + if (size == 0) { + ret = -1; + goto out; + } } + *oi->sizep = size; } - *oi->sizep = size; } if (oi->disk_sizep || (oi->mtimep && p->is_cruft)) { From 3a701a4a68f28238acb89ff58b3b1cbc94c199ec Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 26 Jun 2014 13:23:53 -0400 Subject: [PATCH 085/156] receive-pack: pass receive.warnobjectsize to index-pack This plumbs the config option through to index-pack, where it will actually take effect. Our tests install a pre-receive hook which parses the large-object log from index-pack and acts on it. [this should probably be called softObjectSizeLimit, and there should probably be a corresponding "hard"] Signed-off-by: Jeff King --- builtin/receive-pack.c | 6 +++ t/t5546-receive-limits.sh | 87 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index e6e54ba55f7a8c..9b9891332f6ce0 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -64,6 +64,7 @@ static int advertise_atomic_push = 1; static int advertise_push_options; static int advertise_sid; static off_t max_input_size; +static off_t warn_object_size; static int report_status; static int report_status_v2; static int use_sideband; @@ -255,6 +256,11 @@ static int receive_pack_config(const char *var, const char *value, return 0; } + if (strcmp(var, "receive.warnobjectsize") == 0) { + warn_object_size = git_config_ulong(var, value, ctx->kvi); + return 0; + } + return git_default_config(var, value, ctx, cb); } diff --git a/t/t5546-receive-limits.sh b/t/t5546-receive-limits.sh index f1e61c9f09572e..46e3a22302e3a1 100755 --- a/t/t5546-receive-limits.sh +++ b/t/t5546-receive-limits.sh @@ -83,3 +83,90 @@ test_pack_input_limit index test_pack_input_limit unpack test_done + +# XXX need to refactor +create_dest() { + rm -rf dest && + git init --bare dest && + git --git-dir=dest config receive.unpacklimit 1 +} + +create_hook() { + write_script dest/hooks/pre-receive <<-\EOF + log="$GIT_QUARANTINE_PATH/.large-objects" + test -s "$log" || exit 0 + + tips= + while read old new ref; do + test "$new" = "$_z40" && continue + tips="$tips $new" + done + + echo >&2 "blocking push due to size" + while read sha1 size; do + path=$(git rev-list --find="$sha1" $tips) + echo >&2 "too large ($size): ${path:-$sha1}" + done <"$log" + exit 1 + EOF +} + +test_expect_success 'index-pack can warn about large object' ' + git pack-objects --all --stdout tmp.pack && + git index-pack --warn-object-size=512 --stdin stderr && + echo "large-object: $blob 1024" >expect && + test_cmp expect stderr +' + +test_expect_success 'receive.warnobjectsize does not block large object' ' + create_dest && + git --git-dir=dest config receive.warnobjectsize 512 && + git push dest HEAD +' + +test_expect_success 'receive.warnobjectsize writes log' ' + create_dest && + create_hook && + git --git-dir=dest config receive.warnobjectsize 512 && + test_must_fail git push dest HEAD 2>stderr +' + +test_expect_success 'rejected push cleans up object' ' + ( + cd dest && + test_must_fail git cat-file -t $blob + ) +' + +test_expect_success 'pre-receive hook parsed large-object log' ' + perl -lne "print \$1 if /remote: (.*?)\\s+\$/" stderr >actual && + cat >expect <<-EOF && + blocking push due to size + too large (1024): $blob $commit/file + EOF + test_cmp expect actual +' + +test_expect_success 'bumping limit allows push' ' + git --git-dir=dest config receive.warnobjectsize 4k && + git push dest HEAD +' + +test_expect_success 'create delta-capable commit' ' + echo foo >>file && + git add file && + test_commit delta +' + +test_expect_success 'warnobjectsize catches deltified objects' ' + create_dest && + create_hook && + git push dest HEAD^:refs/heads/master && + git --git-dir=dest config receive.maxsize 512 && + git --git-dir=dest config receive.warnobjectsize 512 && + test_must_fail git push dest HEAD && + git --git-dir=dest config receive.warnobjectsize 4k && + git push dest HEAD +' + +test_done From d0d0d600767ace9fc8dccb3a8a416f1fe5e9c369 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:34:01 -0400 Subject: [PATCH 086/156] refs: check refnames as fully qualified when writing When a ref update is queued via ref_transaction_update(), we call check_refname_format() to make sure the name is acceptable. We pass REFNAME_ALLOW_ONELEVEL, which allows pseudorefs like MERGE_HEAD. But that's not enough to forbid names outside of refs/ like "foo/bar" or even scary stuff like "config" (though fortunately I think that should never work because we cannot resolve "config" to read the old value). Let's instead pass REFNAME_FULLY_QUALIFIED, which tells the checking function that we really do have a full refname, and it can enforce it as such. This means that "git update-ref foo/bar HEAD" will now be rejected. Note that _deleting_ such a ref is already forbidden (and there is a test in t1430 for that already), due to some confusing differences between check_refname_format() and refname_is_safe(). See the previous commit for more details. And that case is already tested via t1430's "update-ref -d cannot delete non-ref in .git dir" test, so we only need to add tests for our newly-changed behavior. Signed-off-by: Jeff King --- refs.c | 2 +- t/t1430-bad-ref-name.sh | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/refs.c b/refs.c index 90b4dbffb53d95..6158ff6adc9da5 100644 --- a/refs.c +++ b/refs.c @@ -1390,7 +1390,7 @@ static int transaction_refname_valid(const char *refname, strbuf_addf(err, refusal_msg, refname); return 0; } else if ((new_oid && !is_null_oid(new_oid)) ? - check_refname_format(refname, REFNAME_ALLOW_ONELEVEL) : + check_refname_format(refname, REFNAME_FULLY_QUALIFIED) : !refname_is_safe(refname)) { const char *refusal_msg; if (flags & REF_LOG_ONLY) diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index 1ed4c7d2c233a8..d6df1e12c58080 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -389,4 +389,14 @@ test_expect_success 'branch -m can rename refs/heads/-dash' ' git show-ref refs/heads/dash ' +test_expect_success 'update-ref refuses lowercase outside of refs/' ' + test_must_fail git update-ref lowercase HEAD 2>err && + test_grep "refusing to update ref with bad name" err +' + +test_expect_success 'update-ref refuses non-underscore punctuation outside of refs/' ' + test_must_fail git update-ref FOO/HEAD HEAD 2>err && + test_grep "refusing to update ref with bad name" err +' + test_done From fe5b0b4c006d1ed89f3965539cee826fbd313e5b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:35:54 -0400 Subject: [PATCH 087/156] t: adjust root refs that violate syntax rules The syntax of root refs (outside of "refs/") generally requires that names end in "_HEAD" (both in the glossary, and also in code via the is_root_ref_syntax() function). We don't yet enforce that consistently, and there are several tests that use names that violate the rules. In preparation for tighter enforcement, let's switch these to syntactically valid names. In many of these cases we probably could just move the refs into refs/heads/ or elsewhere. But without investigating each test, that runs the risk of violating what the test is trying to do (e.g., like the case in t9300 that we adjusted in an earlier commit). So let's just adjust the names, which ensures we aren't ruining any tests, but future-proofs us. --- t/t1400-update-ref.sh | 62 +++++++++---------- t/t1401-symbolic-ref.sh | 28 ++++----- t/t1405-main-ref-store.sh | 16 ++--- t/t1407-worktree-ref-store.sh | 10 +-- t/t4013-diff-various.sh | 4 +- ...iff.diff-tree_--cc_--summary_REVERSE_HEAD} | 2 +- 6 files changed, 61 insertions(+), 61 deletions(-) rename t/t4013/{diff.diff-tree_--cc_--summary_REVERSE => diff.diff-tree_--cc_--summary_REVERSE_HEAD} (73%) diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index cff410ec8910ba..147c89c53ca985 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -216,11 +216,11 @@ test_expect_success 'delete symref without dereference' ' echo foo >foo.c && git add foo.c && git commit -m foo && - git symbolic-ref SYMREF $m && - git update-ref --no-deref -d SYMREF && + git symbolic-ref SYMREF_HEAD $m && + git update-ref --no-deref -d SYMREF_HEAD && git show-ref --verify -q $m && - test_must_fail git show-ref --verify -q SYMREF && - test_must_fail git symbolic-ref SYMREF + test_must_fail git show-ref --verify -q SYMREF_HEAD && + test_must_fail git symbolic-ref SYMREF_HEAD ' test_expect_success 'delete symref without dereference when the referred ref is packed' ' @@ -228,12 +228,12 @@ test_expect_success 'delete symref without dereference when the referred ref is echo foo >foo.c && git add foo.c && git commit -m foo && - git symbolic-ref SYMREF $m && + git symbolic-ref SYMREF_HEAD $m && git pack-refs --all && - git update-ref --no-deref -d SYMREF && + git update-ref --no-deref -d SYMREF_HEAD && git show-ref --verify -q $m && - test_must_fail git show-ref --verify -q SYMREF && - test_must_fail git symbolic-ref SYMREF + test_must_fail git show-ref --verify -q SYMREF_HEAD && + test_must_fail git symbolic-ref SYMREF_HEAD ' test_expect_success 'update-ref -d is not confused by self-reference' ' @@ -826,13 +826,13 @@ test_expect_success 'stdin delete ref fails with zero old value' ' ' test_expect_success 'stdin update symref works option no-deref' ' - git symbolic-ref TESTSYMREF $b && + git symbolic-ref TESTSYMREF_HEAD $b && cat >stdin <<-EOF && option no-deref - update TESTSYMREF $a $b + update TESTSYMREF_HEAD $a $b EOF git update-ref --stdin expect && + git rev-parse TESTSYMREF_HEAD >expect && git rev-parse $a >actual && test_cmp expect actual && git rev-parse $m~1 >expect && @@ -841,27 +841,27 @@ test_expect_success 'stdin update symref works option no-deref' ' ' test_expect_success 'stdin delete symref works option no-deref' ' - git symbolic-ref TESTSYMREF $b && + git symbolic-ref TESTSYMREF_HEAD $b && cat >stdin <<-EOF && option no-deref - delete TESTSYMREF $b + delete TESTSYMREF_HEAD $b EOF git update-ref --stdin expect && git rev-parse $b >actual && test_cmp expect actual ' test_expect_success 'stdin update symref works flag --no-deref' ' - git symbolic-ref TESTSYMREFONE $b && - git symbolic-ref TESTSYMREFTWO $b && + git symbolic-ref TESTSYMREFONE_HEAD $b && + git symbolic-ref TESTSYMREFTWO_HEAD $b && cat >stdin <<-EOF && - update TESTSYMREFONE $a $b - update TESTSYMREFTWO $a $b + update TESTSYMREFONE_HEAD $a $b + update TESTSYMREFTWO_HEAD $a $b EOF git update-ref --no-deref --stdin expect && + git rev-parse TESTSYMREFONE_HEAD TESTSYMREFTWO_HEAD >expect && git rev-parse $a $a >actual && test_cmp expect actual && git rev-parse $m~1 >expect && @@ -870,15 +870,15 @@ test_expect_success 'stdin update symref works flag --no-deref' ' ' test_expect_success 'stdin delete symref works flag --no-deref' ' - git symbolic-ref TESTSYMREFONE $b && - git symbolic-ref TESTSYMREFTWO $b && + git symbolic-ref TESTSYMREFONE_HEAD $b && + git symbolic-ref TESTSYMREFTWO_HEAD $b && cat >stdin <<-EOF && - delete TESTSYMREFONE $b - delete TESTSYMREFTWO $b + delete TESTSYMREFONE_HEAD $b + delete TESTSYMREFTWO_HEAD $b EOF git update-ref --no-deref --stdin expect && git rev-parse $b >actual && test_cmp expect actual @@ -1252,10 +1252,10 @@ test_expect_success 'stdin -z delete ref fails with zero old value' ' ' test_expect_success 'stdin -z update symref works option no-deref' ' - git symbolic-ref TESTSYMREF $b && - printf $F "option no-deref" "update TESTSYMREF" "$a" "$b" >stdin && + git symbolic-ref TESTSYMREF_HEAD $b && + printf $F "option no-deref" "update TESTSYMREF_HEAD" "$a" "$b" >stdin && git update-ref -z --stdin expect && + git rev-parse TESTSYMREF_HEAD >expect && git rev-parse $a >actual && test_cmp expect actual && git rev-parse $m~1 >expect && @@ -1264,10 +1264,10 @@ test_expect_success 'stdin -z update symref works option no-deref' ' ' test_expect_success 'stdin -z delete symref works option no-deref' ' - git symbolic-ref TESTSYMREF $b && - printf $F "option no-deref" "delete TESTSYMREF" "$b" >stdin && + git symbolic-ref TESTSYMREF_HEAD $b && + printf $F "option no-deref" "delete TESTSYMREF_HEAD" "$b" >stdin && git update-ref -z --stdin expect && git rev-parse $b >actual && test_cmp expect actual diff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh index bf703498f1a2e5..b78dcac7a96c3a 100755 --- a/t/t1401-symbolic-ref.sh +++ b/t/t1401-symbolic-ref.sh @@ -45,18 +45,18 @@ test_expect_success 'HEAD cannot be removed' ' reset_to_sane test_expect_success 'symbolic-ref can be deleted' ' - git symbolic-ref NOTHEAD refs/heads/foo && - git symbolic-ref -d NOTHEAD && + git symbolic-ref NOT_HEAD refs/heads/foo && + git symbolic-ref -d NOT_HEAD && git rev-parse refs/heads/foo && - test_must_fail git symbolic-ref NOTHEAD + test_must_fail git symbolic-ref NOT_HEAD ' reset_to_sane test_expect_success 'symbolic-ref can delete dangling symref' ' - git symbolic-ref NOTHEAD refs/heads/missing && - git symbolic-ref -d NOTHEAD && + git symbolic-ref NOT_HEAD refs/heads/missing && + git symbolic-ref -d NOT_HEAD && test_must_fail git rev-parse refs/heads/missing && - test_must_fail git symbolic-ref NOTHEAD + test_must_fail git symbolic-ref NOT_HEAD ' reset_to_sane @@ -190,34 +190,34 @@ test_expect_success 'symbolic-ref pointing at another' ' test_expect_success 'symbolic-ref --short handles complex utf8 case' ' name="测试-加-增加-加-增加" && - git symbolic-ref TEST_SYMREF "refs/heads/$name" && + git symbolic-ref TEST_SYMREF_HEAD "refs/heads/$name" && # In the real world, we saw problems with this case only # when the locale includes UTF-8. Set it here to try to make things as # hard as possible for us to pass, but in practice we should do the # right thing regardless (and of course some platforms may not even # have this locale). - LC_ALL=en_US.UTF-8 git symbolic-ref --short TEST_SYMREF >actual && + LC_ALL=en_US.UTF-8 git symbolic-ref --short TEST_SYMREF_HEAD >actual && echo "$name" >expect && test_cmp expect actual ' test_expect_success 'symbolic-ref --short handles name with suffix' ' - git symbolic-ref TEST_SYMREF "refs/remotes/origin/HEAD" && - git symbolic-ref --short TEST_SYMREF >actual && + git symbolic-ref TEST_SYMREF_HEAD "refs/remotes/origin/HEAD" && + git symbolic-ref --short TEST_SYMREF_HEAD >actual && echo "origin" >expect && test_cmp expect actual ' test_expect_success 'symbolic-ref --short handles almost-matching name' ' - git symbolic-ref TEST_SYMREF "refs/headsXfoo" && - git symbolic-ref --short TEST_SYMREF >actual && + git symbolic-ref TEST_SYMREF_HEAD "refs/headsXfoo" && + git symbolic-ref --short TEST_SYMREF_HEAD >actual && echo "headsXfoo" >expect && test_cmp expect actual ' test_expect_success 'symbolic-ref --short handles name with percent' ' - git symbolic-ref TEST_SYMREF "refs/heads/%foo" && - git symbolic-ref --short TEST_SYMREF >actual && + git symbolic-ref TEST_SYMREF_HEAD "refs/heads/%foo" && + git symbolic-ref --short TEST_SYMREF_HEAD >actual && echo "%foo" >expect && test_cmp expect actual ' diff --git a/t/t1405-main-ref-store.sh b/t/t1405-main-ref-store.sh index 6d8f401a2a9827..347c53e642b6a7 100755 --- a/t/t1405-main-ref-store.sh +++ b/t/t1405-main-ref-store.sh @@ -14,21 +14,21 @@ test_expect_success 'setup' ' test_commit one ' -test_expect_success 'create_symref(FOO, refs/heads/main)' ' - $RUN create-symref FOO refs/heads/main nothing && +test_expect_success 'create_symref(FOO_HEAD, refs/heads/main)' ' + $RUN create-symref FOO_HEAD refs/heads/main nothing && echo refs/heads/main >expected && - git symbolic-ref FOO >actual && + git symbolic-ref FOO_HEAD >actual && test_cmp expected actual ' -test_expect_success 'delete_refs(FOO, refs/tags/new-tag)' ' +test_expect_success 'delete_refs(FOO_HEAD, refs/tags/new-tag)' ' git tag -a -m new-tag new-tag HEAD && - git rev-parse FOO -- && + git rev-parse FOO_HEAD -- && git rev-parse refs/tags/new-tag -- && m=$(git rev-parse main) && - $RUN delete-refs REF_NO_DEREF nothing FOO refs/tags/new-tag && - test_must_fail git rev-parse --symbolic-full-name FOO && - test_must_fail git rev-parse FOO -- && + $RUN delete-refs REF_NO_DEREF nothing FOO_HEAD refs/tags/new-tag && + test_must_fail git rev-parse --symbolic-full-name FOO_HEAD && + test_must_fail git rev-parse FOO_HEAD -- && test_must_fail git rev-parse refs/tags/new-tag -- ' diff --git a/t/t1407-worktree-ref-store.sh b/t/t1407-worktree-ref-store.sh index 9d8e1a13432b83..a845d8e691388b 100755 --- a/t/t1407-worktree-ref-store.sh +++ b/t/t1407-worktree-ref-store.sh @@ -40,15 +40,15 @@ test_expect_success 'resolve_ref()' ' test_cmp expected actual ' -test_expect_success 'create_symref(FOO, refs/heads/main)' ' - $RWT create-symref FOO refs/heads/main nothing && +test_expect_success 'create_symref(FOO_HEAD, refs/heads/main)' ' + $RWT create-symref FOO_HEAD refs/heads/main nothing && echo refs/heads/main >expected && - git -C wt symbolic-ref FOO >actual && + git -C wt symbolic-ref FOO_HEAD >actual && test_cmp expected actual && - $RMAIN create-symref FOO refs/heads/wt-main nothing && + $RMAIN create-symref FOO_HEAD refs/heads/wt-main nothing && echo refs/heads/wt-main >expected && - git symbolic-ref FOO >actual && + git symbolic-ref FOO_HEAD >actual && test_cmp expected actual ' diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh index b7a382c88130ec..0dcdb4dd51e503 100755 --- a/t/t4013-diff-various.sh +++ b/t/t4013-diff-various.sh @@ -118,7 +118,7 @@ test_expect_success setup ' # pseudo-ref to avoid impacting tests with --all. commit=$(echo reverse | git commit-tree -p main^2 -p main^1 main^{tree}) && - git update-ref REVERSE $commit && + git update-ref REVERSE_HEAD $commit && git config diff.renames false && @@ -328,7 +328,7 @@ diff-tree --cc --stat --summary main diff-tree -c --stat --summary side diff-tree --cc --stat --summary side diff-tree --cc --shortstat main -diff-tree --cc --summary REVERSE +diff-tree --cc --summary REVERSE_HEAD # improved by Timo's patch diff-tree --cc --patch-with-stat main # improved by Timo's patch diff --git a/t/t4013/diff.diff-tree_--cc_--summary_REVERSE b/t/t4013/diff.diff-tree_--cc_--summary_REVERSE_HEAD similarity index 73% rename from t/t4013/diff.diff-tree_--cc_--summary_REVERSE rename to t/t4013/diff.diff-tree_--cc_--summary_REVERSE_HEAD index e208dd56826e74..0f86ae60db4a17 100644 --- a/t/t4013/diff.diff-tree_--cc_--summary_REVERSE +++ b/t/t4013/diff.diff-tree_--cc_--summary_REVERSE_HEAD @@ -1,4 +1,4 @@ -$ git diff-tree --cc --summary REVERSE +$ git diff-tree --cc --summary REVERSE_HEAD 2562325a7ee916efb2481da93073b82cec801cbc create mode 100644 file1 delete mode 100644 file2 From 8ca9e0c08def39cb0b54520d30f8a2d9e817b1a0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 13 Aug 2026 01:32:16 -0400 Subject: [PATCH 088/156] we should not need our magic export anymore --- t/perf/perf-lib.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 3f0a0e2db7f7a9..c086eece76f2bc 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -53,7 +53,6 @@ fi . "$GIT_BUILD_DIR"/GIT-BUILD-OPTIONS : ${TEST_OUTPUT_DIRECTORY:=$perf_dir} -export TEST_OUTPUT_DIRECTORY . "$GIT_SOURCE_DIR"/t/test-lib.sh unset GIT_CONFIG_NOSYSTEM From c3f2e8e8035810bba6edca412a4e25cea0f9e5b0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 19 Dec 2017 04:58:34 -0500 Subject: [PATCH 089/156] unpack-objects: drop -r option It doesn't actually work. Signed-off-by: Jeff King --- builtin/unpack-objects.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c index 351948724ab703..76d156f9fad933 100644 --- a/builtin/unpack-objects.c +++ b/builtin/unpack-objects.c @@ -22,7 +22,7 @@ #include "fsck.h" #include "packfile.h" -static int dry_run, quiet, recover, has_errors, strict; +static int dry_run, quiet, has_errors, strict; static const char unpack_usage[] = "git unpack-objects [-n] [-q] [-r] [--strict]"; static unsigned char buffer[DEFAULT_IO_BUFFER_SIZE]; @@ -136,8 +136,6 @@ static void *get_data(unsigned long size) if (ret != Z_OK) { error("inflate returned %d", ret); FREE_AND_NULL(buf); - if (!recover) - exit(1); has_errors = 1; break; } @@ -522,8 +520,6 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size, if (!base) { error("failed to read delta-pack base object %s", oid_to_hex(&base_oid)); - if (!recover) - exit(1); has_errors = 1; return; } @@ -576,8 +572,6 @@ static void unpack_one(unsigned nr) default: error("bad object type %d", type); has_errors = 1; - if (recover) - return; exit(1); } } @@ -646,7 +640,7 @@ int cmd_unpack_objects(int argc, continue; } if (!strcmp(arg, "-r")) { - recover = 1; + warning("option -r is deprecated and does nothing"); continue; } if (!strcmp(arg, "--strict")) { From 501d83f7eca9c319d8fc8a55c5a44a45ddb036d8 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 13 Oct 2020 08:43:26 -0400 Subject: [PATCH 090/156] allow content-limit Signed-off-by: Jeff King --- builtin/cat-file.c | 18 ++++++++++++++++-- odb.h | 1 + odb/source-loose.c | 12 ++++++++---- packfile.c | 13 ++++++++++++- 4 files changed, 37 insertions(+), 7 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 8870a210ec2e94..7e3e38adb91301 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -306,6 +306,7 @@ struct expand_data { off_t disk_size; const char *rest; struct object_id delta_base_oid; + void *content; /* * If mark_query is true, we do not expand anything, but rather @@ -440,7 +441,10 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d assert(data->info.typep); - if (data->type == OBJ_BLOB) { + if (data->content) { + batch_write(opt, data->content, data->size); + FREE_AND_NULL(data->content); + } else if (data->type == OBJ_BLOB) { if (opt->buffer_output) fflush(stdout); if (opt->transform_mode) { @@ -1058,9 +1062,19 @@ static int batch_objects(struct batch_options *opt) /* * If we are printing out the object, then always fill in the type, * since we will want to decide whether or not to stream. + * + * Likewise, grab the content in the initial request if it's small + * and we're not planning to filter it. */ - if (opt->batch_mode == BATCH_MODE_CONTENTS) + if (opt->batch_mode == BATCH_MODE_CONTENTS) { data.info.typep = &data.type; + if (!opt->transform_mode) { + data.info.sizep = &data.size; + data.info.contentp = &data.content; + data.info.content_limit = + repo_settings_get_big_file_threshold(the_repository); + } + } if (opt->all_objects) { struct object_cb_data cb; diff --git a/odb.h b/odb.h index 248ee9cdfaa99c..5ff35ceaa909cf 100644 --- a/odb.h +++ b/odb.h @@ -370,6 +370,7 @@ struct object_info { /* The object contents. Ownership of memory goes over to the caller. */ void **contentp; + size_t content_limit; /* * The time the given looked-up object has been last modified. diff --git a/odb/source-loose.c b/odb/source-loose.c index 524405445bd3c2..99741d4f69b7aa 100644 --- a/odb/source-loose.c +++ b/odb/source-loose.c @@ -171,10 +171,14 @@ static int read_object_info_from_path(struct odb_source_loose *loose, die(_("invalid object type")); if (oi->contentp) { - *oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid); - if (!*oi->contentp) { - ret = -1; - goto out; + if (!oi->content_limit || *oi->sizep <= oi->content_limit) { + *oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid); + if (!*oi->contentp) { + ret = -1; + goto out; + } + } else { + *oi->contentp = NULL; } } diff --git a/packfile.c b/packfile.c index 87b454da94b4a9..7cb9ff5ffb6c84 100644 --- a/packfile.c +++ b/packfile.c @@ -1321,7 +1321,7 @@ int packed_object_info_with_index_pos(struct odb_source_packed *source, * a "real" type later if the caller is interested. Likewise... * tbd. */ - if (oi->contentp) { + if (oi->contentp && !oi->content_limit) { *oi->contentp = cache_or_unpack_entry(p->repo, p, obj_offset, oi->sizep, &type); if (!*oi->contentp) @@ -1346,6 +1346,17 @@ int packed_object_info_with_index_pos(struct odb_source_packed *source, } *oi->sizep = size; } + + if (oi->contentp) { + if (oi->sizep && *oi->sizep < oi->content_limit) { + *oi->contentp = cache_or_unpack_entry(p->repo, p, obj_offset, + oi->sizep, &type); + if (!*oi->contentp) + type = OBJ_BAD; + } else { + *oi->contentp = NULL; + } + } } if (oi->disk_sizep || (oi->mtimep && p->is_cruft)) { From a3a42013a6a21014d770b284b2365ac7d147ba8b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Sep 2016 17:29:42 -0400 Subject: [PATCH 091/156] index-pack: summarily reject non-blobs over warn-object-size The point of this option being "warn" and not a strict maximum is that we want to write a nicer message to the user. We get the complete list of large objects, and then we walk the rejected history to show their path names. This is a good idea for blobs, where people accidentally add large files, and where the cost of walking the history is low (because we don't really do anything with blobs except write them to disk, there's little harm in things like overflows). But the equation is different for other objects, like trees. Anybody with a 100MB tree is almost certainly doing something malicious. And we are at much higher risk, because we actually parse those trees. For example, the is_ntfs_dotgit() function (itself a safety check!) is run even on large trees, but can experience integer overflow with a 2-gigabyte pathname. This shouldn't be an exploitable vulnerability (we end up reading NULL, but the function does not write at all), but it still causes us to segfault. Instead, let's summarily die when we see a large object that isn't a blob. Signed-off-by: Jeff King --- builtin/index-pack.c | 5 ++++- t/t5546-receive-limits.sh | 6 ++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 4ae2c07ed097e9..85674cc012480e 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -917,8 +917,11 @@ static void sha1_object(const void *data, struct object_entry *obj_entry, assert(data || obj_entry); - if (warn_object_size && size > warn_object_size) + if (warn_object_size && size > warn_object_size) { + if (type != OBJ_BLOB) + die("non-blob object size limit exceeded"); warn_large_object(oid, size); + } if (startup_info->have_repository) { read_lock(); diff --git a/t/t5546-receive-limits.sh b/t/t5546-receive-limits.sh index 46e3a22302e3a1..2010273ac1ee45 100755 --- a/t/t5546-receive-limits.sh +++ b/t/t5546-receive-limits.sh @@ -169,4 +169,10 @@ test_expect_success 'warnobjectsize catches deltified objects' ' git push dest HEAD ' +test_expect_success 'receive.warnobjectsize does block large non-blobs' ' + create_dest && + git --git-dir=dest config receive.warnobjectsize 10 && + test_must_fail git push dest HEAD +' + test_done From 5f93a934a76bcbcea910d90748f85f898c01660d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 10 Aug 2015 00:35:34 -0400 Subject: [PATCH 092/156] pager_in_use: make sure output is still going to pager When we start a pager, we set GIT_PAGER_IN_USE=1 in the environment. This lets sub-processes know that even though isatty(1) is not true, it is because it is connected to a pager (and we should still turn on human-readable niceties like auto-color). Unfortunately, this is too inclusive for scripts which invoke git sub-programs with their stdout going somewhere else. For example, if you run "git -p pull rebase", git-pull will invoke "git rebase", which invokes: git format-patch ... >rebased-patches This format-patch process knows that its stdout is not a tty, but because of GIT_PAGER_IN_USE it assumes this is because stdout is going to a pager. As a result, it writes colorized output, and the matching "git am" invocation chokes on it, causing the rebase to fail. We could work around this by passing "--no-color" to format-patch, or by removing GIT_PAGER_IN_USE from the environment. But we should not have to do so; format-patch should be able to realize that even though GIT_PAGER_IN_USE is set, its stdout is not actually going to that pager. For this simple case, format-patch could see that its output is not even a pipe. But that would not catch a case like: git format-patch | some-program >rebased-patches where it cannot distinguish between the pipe to the pager and the pipe to some-program. This patch solves it by actually noting the inode of the pipe to the pager in the environment, which readers of GIT_PAGER_IN_USE can check against their stdout. This technically makes GIT_PAGER_IN_USE redundant (we can just check the new GIT_PAGER_PIPE_ID), but we keep using both variables for compatibility with external scripts: - scripts which check GIT_PAGER_IN_USE can continue to do so, and will just ignore the new pipe-id variable. Meaning they may accidentally turn on colors if their output is redirected to a file, but that is the same as today and we cannot fix that. We do not actively break them from showing colors when their stdout _does_ go to the pager. - scripts which set GIT_PAGER_IN_USE but not GIT_PAGER_PIPE_ID will continue to turn on colorization for git sub-commands (again, they do not benefit from the new code, but we are not making anything worse). The inode-retrieval code itself is abstracted into compat/, as different platforms may represent the pipe id differently. These ids do not need to be portable across systems, only within processes on the same system. Note that there is an existing test in t7006 which tests for the exact _opposite_ of what we are trying to achieve (namely, that GIT_PAGER_IN_USE does _not_ cause us to write colors to a random file). This test comes from a battery of tests added by 60b6e22 (tests: Add tests for automatic use of pager, 2010-02-19), and I think is simply misguided, as evidenced by the real "git pull" bug above. If you want to ensure colors in a file, you do it with "--color", not by pretending you have a pager. Rather than delete the test, though, we simply re-title it here. It actually makes a good check of the "scripts which set PAGER_IN_USE but not PAGER_PIPE_ID" historical compatibility mentioned above. Signed-off-by: Jeff King --- Makefile | 1 + compat/pipe-id.c | 38 ++++++++++++++++++++++++++++++++++++++ compat/pipe-id.h | 27 +++++++++++++++++++++++++++ meson.build | 1 + pager.c | 16 +++++++++++++++- t/t7006-pager.sh | 20 +++++++++++++++++++- 6 files changed, 101 insertions(+), 2 deletions(-) create mode 100644 compat/pipe-id.c create mode 100644 compat/pipe-id.h diff --git a/Makefile b/Makefile index d4b775953d3842..bfcbb17cd8f76a 100644 --- a/Makefile +++ b/Makefile @@ -1133,6 +1133,7 @@ LIB_OBJS += common-init.o LIB_OBJS += compat/nonblock.o LIB_OBJS += compat/obstack.o LIB_OBJS += compat/open.o +LIB_OBJS += compat/pipe-id.o LIB_OBJS += compat/terminal.o LIB_OBJS += compiler-tricks/not-constant.o LIB_OBJS += config.o diff --git a/compat/pipe-id.c b/compat/pipe-id.c new file mode 100644 index 00000000000000..a117c98f0418e0 --- /dev/null +++ b/compat/pipe-id.c @@ -0,0 +1,38 @@ +#include "git-compat-util.h" +#include "compat/pipe-id.h" +#include "strbuf.h" + +const char *pipe_id_get(int fd) +{ + static struct strbuf id = STRBUF_INIT; + struct stat st; + + if (fstat(fd, &st) < 0 || !S_ISFIFO(st.st_mode)) + return NULL; + + strbuf_reset(&id); + strbuf_addf(&id, "%lu:%lu", + (unsigned long)st.st_dev, + (unsigned long)st.st_ino); + return id.buf; +} + +int pipe_id_match(int fd, const char *id) +{ + struct stat st; + const char *end; + unsigned long dev, ino; + + if (fstat(fd, &st) < 0 || !S_ISFIFO(st.st_mode)) + return 0; + + dev = strtoul(id, (char **)&end, 10); + if (*end++ != ':') + return 0; + ino = strtoul(end, (char **)&end, 10); + if (*end) + return 0; + + return dev == (unsigned long)st.st_dev && + ino == (unsigned long)st.st_ino; +} diff --git a/compat/pipe-id.h b/compat/pipe-id.h new file mode 100644 index 00000000000000..5ddff2c115c97f --- /dev/null +++ b/compat/pipe-id.h @@ -0,0 +1,27 @@ +#ifndef PIPE_ID_H +#define PIPE_ID_H + +/** + * This module allows callers to save a string pipe identifier, and later find + * out whether a file descriptor refers to the same pipe. + * + * The ids should be opaque to the callers, as their implementation may be + * system dependent. The generated ids can be used between processes on the + * same system, but are not portable between systems, or even between different + * versions of git. + */ + +/** + * Returns a string representing the pipe-id of the file descriptor `fd`, or + * NULL if an error occurs. Note that the return value may be invalidated by + * subsequent calls to pipe_id_get. + */ +const char *pipe_id_get(int fd); + +/** + * Returns 1 if the pipe at `fd` matches the id `id`, or 0 otherwise (or if an + * error occurs). + */ +int pipe_id_match(int fd, const char *id); + +#endif /* PIPE_ID_H */ diff --git a/meson.build b/meson.build index d86f2acd2b2a46..7df75926679b61 100644 --- a/meson.build +++ b/meson.build @@ -290,6 +290,7 @@ compat_sources = [ 'compat/nonblock.c', 'compat/obstack.c', 'compat/open.c', + 'compat/pipe-id.c', 'compat/terminal.c', ] diff --git a/pager.c b/pager.c index 543ef129366a19..7c7a847ce35ce3 100644 --- a/pager.c +++ b/pager.c @@ -7,6 +7,7 @@ #include "alias.h" #include "repository.h" #include "environment.h" +#include "compat/pipe-id.h" int pager_use_color = 1; @@ -156,6 +157,7 @@ void setup_pager(struct repository *r) { static int once = 0; const char *pager = git_pager(r, isatty(1)); + const char *pipe_id; if (!pager) return; @@ -192,6 +194,10 @@ void setup_pager(struct repository *r) } close(pager_process.in); + pipe_id = pipe_id_get(1); + if (pipe_id) + setenv("GIT_PAGER_PIPE_ID", pipe_id, 1); + sigchain_push_common(wait_for_pager_signal); if (!once) { @@ -202,7 +208,15 @@ void setup_pager(struct repository *r) int pager_in_use(void) { - return git_env_bool("GIT_PAGER_IN_USE", 0); + const char *pipe_id; + + if (!git_env_bool("GIT_PAGER_IN_USE", 0)) + return 0; + + pipe_id = getenv("GIT_PAGER_PIPE_ID"); + if (!pipe_id) /* historical compatibility */ + return 1; + return pipe_id_match(1, pipe_id); } /* diff --git a/t/t7006-pager.sh b/t/t7006-pager.sh index 75ca7a25daed91..9bab578c637bc0 100755 --- a/t/t7006-pager.sh +++ b/t/t7006-pager.sh @@ -324,7 +324,7 @@ test_expect_success TTY 'colors are suppressed by color.pager' ' ! colorful paginated.out ' -test_expect_success 'color when writing to a file intended for a pager' ' +test_expect_success 'color goes to files when GIT_PAGER_PIPE_ID is not used' ' rm -f colorful.log && test_config color.ui auto && ( @@ -343,6 +343,24 @@ test_expect_success TTY 'colors are sent to pager for external commands' ' colorful paginated.out ' +test_expect_success TTY 'no color when paged program writes to file' ' + test_config alias.externallog "!git log >log.out" && + test_config color.ui auto && + test_terminal env TERM=vt100 git -p externallog && + test_line_count = 0 paginated.out && + test -s log.out && + ! colorful log.out +' + +test_expect_success TTY 'no color when paged program writes to pipe' ' + test_config alias.externallog "!git log | cat >log.out" && + test_config color.ui auto && + test_terminal env TERM=vt100 git -p externallog && + test_line_count = 0 paginated.out && + test -s log.out && + ! colorful log.out +' + # Use this helper to make it easy for the caller of your # terminal-using function to specify whether it should fail. # If you write From 0789d594cbbf9ae214c6c2f717eccb8c482cf3e3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 3 Jan 2018 03:13:08 -0500 Subject: [PATCH 093/156] receive-pack: disable hook advice in "quiet" mode If the client asked us to be quiet, we should avoid printing warnings like the ones ...is this a good idea? Signed-off-by: Jeff King --- advice.c | 8 ++++++++ advice.h | 6 ++++++ builtin/receive-pack.c | 5 ++++- t/t7520-ignored-hook-warning.sh | 15 +++++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/advice.c b/advice.c index 63bf8b0c5f0481..5f3a5d6bc2cc57 100644 --- a/advice.c +++ b/advice.c @@ -313,3 +313,11 @@ void advise_on_moving_dirty_path(struct string_list *pathspec_list) "* Use \"git add --sparse \" to update the index\n" "* Use \"git sparse-checkout reapply\" to apply the sparsity rules")); } + +void disable_advice(void) +{ + size_t i; + + for (i = 0; i < ARRAY_SIZE(advice_setting); i++) + advice_setting[i].level = ADVICE_LEVEL_DISABLED; +} diff --git a/advice.h b/advice.h index 66f6cd6a772d8c..f12b6185d9ec04 100644 --- a/advice.h +++ b/advice.h @@ -85,4 +85,10 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list); void detach_advice(const char *new_name); void advise_on_moving_dirty_path(struct string_list *pathspec_list); +/* + * Turn off all advice flags; this can be used to centrally enforce a --quiet + * option. + */ +void disable_advice(void); + #endif /* ADVICE_H */ diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index e6e54ba55f7a8c..5eb935663d2289 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -39,6 +39,7 @@ #include "trace2.h" #include "version.h" #include "worktree.h" +#include "advice.h" static const char * const receive_pack_usage[] = { N_("git receive-pack "), @@ -2222,8 +2223,10 @@ static struct command *read_head_info(struct packet_reader *reader, report_status_v2 = 1; if (parse_feature_request(feature_list, "side-band-64k")) use_sideband = LARGE_PACKET_MAX; - if (parse_feature_request(feature_list, "quiet")) + if (parse_feature_request(feature_list, "quiet")) { quiet = 1; + disable_advice(); + } if (advertise_atomic_push && parse_feature_request(feature_list, "atomic")) use_atomic = 1; diff --git a/t/t7520-ignored-hook-warning.sh b/t/t7520-ignored-hook-warning.sh index bcfe15d51d0706..551e3a21ff436e 100755 --- a/t/t7520-ignored-hook-warning.sh +++ b/t/t7520-ignored-hook-warning.sh @@ -3,6 +3,7 @@ test_description='ignored hook warning' . ./test-lib.sh +. "$TEST_DIRECTORY"/lib-terminal.sh test_expect_success setup ' test_hook --setup pre-commit <<-\EOF @@ -35,4 +36,18 @@ test_expect_success 'no warning if unset advice.ignoredHook and hook removed' ' test_grep ! -e "hook was ignored" message ' +test_expect_success TTY,POSIXPERM 'push --quiet silences remote hook warnings' ' + git init --bare dst.git && + echo "exit 0" >dst.git/hooks/update && + chmod -x dst.git/hooks/update && + + git commit --allow-empty -m one && + test_terminal git push dst.git HEAD 2>message && + test_grep "hook was ignored" message && + + git commit --allow-empty -m two && + test_terminal git push --quiet dst.git HEAD 2>message && + test_grep ! "hook was ignored" message +' + test_done From 349b499471ccaad7c923d810a311bb18fefb9b0f Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 29 Apr 2024 04:35:33 -0400 Subject: [PATCH 094/156] refs: check refnames as fully qualified when resolving Most code paths for resolving refs end up in refs_resolve_ref_unsafe(), which checks the names using check_refname_format(). The names we have at this stage are always full refnames, but we pass the ALLOW_ONELEVEL flag so that the function allows root refs like MERGE_HEAD. We should instead pass the FULLY_QUALIFIED flag, which lets check_refname_format() do some extra syntactic checks on those ref names. With this patch we'll refuse to read anything outside of refs/ that does not match the usual root ref syntax (all caps plus underscore). This should not be a loss of functionality (since such refs cannot be written as of the previous commit), but may protect us from mischief. For example, you can ask for silly things that look vaguely like object ids, like "rr-cache//postimage", "info/refs", or "objects/info/commit-graphs/commit-graph-chain". It's doubtful you can really do anything _too_ terrible there, but it seems like peeking at random files in .git in response to possibly untrusted input is something we should avoid. Our tests are a bit tricky. We'll try to read a ref with an invalid name and make sure it fails. Ideally we could realize that the failure is caused by the invalid name, and not that the ref doesn't exist. But the "bad name" error is not passed up the stack from the ref code, so the two look the same. The best we can do is create the ref and confirm that we don't accidentally read it. But there's a problem: we don't allow writing refs with invalid names either! For the files backend, we can hack around this by munging the filesystem manually. That means with the files backend, the new test fails without this patch and succeeds with it. But for other backends like reftable, the test would have passed anyway (because the invalid ref does not exist). This is probably OK for now. At least one backend is performing the full test, and other backends won't erroneously fail. In the long run it might be nice to surface the error via stderr, but that can come later. The second test here, for "main-worktree/bad", actually succeeds even without this patch. The worktree-ref code enforces the root-ref syntax itself via is_current_worktree_ref(), so the extra checks for this in check_refname_format() are redundant (though the parsing there is still necessary so we know _not_ to reject "main-worktree/HEAD"). But either way it's good to have a test which makes sure this remains the case. Signed-off-by: Jeff King --- refs.c | 4 ++-- t/t1430-bad-ref-name.sh | 10 ++++++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/refs.c b/refs.c index 6158ff6adc9da5..62295d1502d0fb 100644 --- a/refs.c +++ b/refs.c @@ -2137,7 +2137,7 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs, *flags = 0; - if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) { + if (check_refname_format(refname, REFNAME_FULLY_QUALIFIED)) { if (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) || !refname_is_safe(refname)) return NULL; @@ -2196,7 +2196,7 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs, oidclr(oid, refs->repo->hash_algo); return refname; } - if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) { + if (check_refname_format(refname, REFNAME_FULLY_QUALIFIED)) { if (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) || !refname_is_safe(refname)) return NULL; diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index d6df1e12c58080..bcad6edf219a0a 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -399,4 +399,14 @@ test_expect_success 'update-ref refuses non-underscore punctuation outside of re test_grep "refusing to update ref with bad name" err ' +test_expect_success 'rev-parse refuses non-root-ref outside of refs/' ' + git rev-parse HEAD >.git/bad && + test_must_fail git rev-parse --verify bad +' + +test_expect_success 'rev-parse recognizes non-root-ref via worktree' ' + git rev-parse HEAD >.git/bad && + test_must_fail git rev-parse --verify main-worktree/bad +' + test_done From 2102476f66d4bd443a4fcabd1b7f58f0b8edb11d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 16 Jun 2025 00:35:54 -0400 Subject: [PATCH 095/156] refs: take transaction flags for symref updates The ref_transaction_update() function (and its siblings) all take a ref_transaction_flag argument. Now that symref updates are built on top of ref transactions, let's likewise accept those flags to pass along to the transaction functions. Regular callers generally won't need to use these flags, but we pass them via "test-tool ref-store" to do things like circumvent refname checks. So let's likewise plumb them through that test helper. For now all callers will pass "0" for the flags, so there should be no change in behavior. A few implementation notes: - We'll take the new argument only in refs_update_symref_extended(), to avoid disrupting most callers (there are a few which need updating to pass the empty flag set, but that seems less bad than introducing yet another variant of this function). - Sadly we can't easily make this an optional argument of the test-tool "create-symref" command, since the final "logmsg" argument is already optional. One alternative would be to do real option parsing, like "-f FLAGS". But the other sub-commands don't bother, and this is a hacky test-only interface anyway. So let's just keep it simple and update the callers to all pass 0. - Having both a "create_only" flag and now transaction flags is kind of ugly. Possibly we could represent create_only with a flag, but we'd need to figure out the least-bad way to add it to the transaction flags (non-symref updates don't need such a flag, since they use the null oid in the "old oid" field to indicate this). I've punted on it for now. --- builtin/fetch.c | 2 +- builtin/remote.c | 2 +- refs.c | 11 +++++++---- refs.h | 7 ++++++- t/helper/test-ref-store.c | 20 +++++++++++--------- t/t0600-reffiles-backend.sh | 2 +- t/t1405-main-ref-store.sh | 2 +- t/t1407-worktree-ref-store.sh | 4 ++-- t/t1430-bad-ref-name.sh | 24 ++++++++++++------------ t/t1450-fsck.sh | 2 +- t/t2402-worktree-list.sh | 2 +- 11 files changed, 44 insertions(+), 34 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index ab7db2be06d14c..73ffb439cb4347 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1800,7 +1800,7 @@ static int set_head(const struct ref *remote_refs, struct remote *remote, goto cleanup; } was_detached = refs_update_symref_extended(refs, b_head.buf, b_remote_head.buf, - "fetch", &b_local_head, create_only); + "fetch", &b_local_head, create_only, 0); if (was_detached == -1) { result = 1; goto cleanup; diff --git a/builtin/remote.c b/builtin/remote.c index de989ea3ba9691..b2d12c03deebc7 100644 --- a/builtin/remote.c +++ b/builtin/remote.c @@ -1599,7 +1599,7 @@ static int set_head(int argc, const char **argv, const char *prefix, goto cleanup; } was_detached = refs_update_symref_extended(refs, b_head.buf, b_remote_head.buf, - "remote set-head", &b_local_head, 0); + "remote set-head", &b_local_head, 0, 0); if (was_detached == -1) { result |= error(_("Could not set up %s"), b_head.buf); goto cleanup; diff --git a/refs.c b/refs.c index ee14bdebaff172..5a6eb522d5932b 100644 --- a/refs.c +++ b/refs.c @@ -2522,17 +2522,20 @@ int reference_get_peeled_oid(struct repository *repo, int refs_update_symref(struct ref_store *refs, const char *ref, const char *target, const char *logmsg) { - return refs_update_symref_extended(refs, ref, target, logmsg, NULL, 0); + return refs_update_symref_extended(refs, ref, target, logmsg, NULL, 0, 0); } int refs_update_symref_extended(struct ref_store *refs, const char *ref, const char *target, const char *logmsg, - struct strbuf *referent, int create_only) + struct strbuf *referent, int create_only, + unsigned int flags) { struct ref_transaction *transaction; struct strbuf err = STRBUF_INIT; int ret = 0, prepret = 0; + flags |= REF_NO_DEREF; + transaction = ref_store_transaction_begin(refs, 0, &err); if (!transaction) { error_return: @@ -2541,14 +2544,14 @@ int refs_update_symref_extended(struct ref_store *refs, const char *ref, } if (create_only) { if (ref_transaction_create(transaction, ref, NULL, target, - REF_NO_DEREF, logmsg, &err)) + flags, logmsg, &err)) goto error_return; prepret = ref_transaction_prepare(transaction, &err); if (prepret && prepret != REF_TRANSACTION_ERROR_CREATE_EXISTS) goto error_return; } else { if (ref_transaction_update(transaction, ref, NULL, NULL, - target, NULL, REF_NO_DEREF, + target, NULL, flags, logmsg, &err) || ref_transaction_prepare(transaction, &err)) goto error_return; diff --git a/refs.h b/refs.h index dbd1b5b810d071..ecc8995710e897 100644 --- a/refs.h +++ b/refs.h @@ -755,9 +755,14 @@ int refs_copy_existing_ref(struct ref_store *refs, const char *oldref, int refs_update_symref(struct ref_store *refs, const char *refname, const char *target, const char *logmsg); +/* + * This takes the same ref_transaction_flags as ref_transaction_update(), + * but assumes REF_NO_DEREF (so there is no need for callers to pass it in). + */ int refs_update_symref_extended(struct ref_store *refs, const char *refname, const char *target, const char *logmsg, - struct strbuf *referent, int create_only); + struct strbuf *referent, int create_only, + unsigned int flags); enum action_on_err { UPDATE_REFS_MSG_ON_ERR, diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c index 5a9a3053d9d81a..17db74fe3f70da 100644 --- a/t/helper/test-ref-store.c +++ b/t/helper/test-ref-store.c @@ -112,15 +112,6 @@ static const char **get_store(const char **argv, struct ref_store **refs) return argv + 1; } -static int cmd_create_symref(struct ref_store *refs, const char **argv) -{ - const char *refname = notnull(*argv++, "refname"); - const char *target = notnull(*argv++, "target"); - const char *logmsg = *argv++; - - return refs_update_symref(refs, refname, target, logmsg); -} - static struct flag_definition transaction_flags[] = { FLAG_DEF(REF_NO_DEREF), FLAG_DEF(REF_FORCE_CREATE_REFLOG), @@ -130,6 +121,17 @@ static struct flag_definition transaction_flags[] = { { NULL, 0 } }; +static int cmd_create_symref(struct ref_store *refs, const char **argv) +{ + unsigned int flags = arg_flags(*argv++, "flags", transaction_flags); + const char *refname = notnull(*argv++, "refname"); + const char *target = notnull(*argv++, "target"); + const char *logmsg = *argv++; + + return refs_update_symref_extended(refs, refname, target, logmsg, NULL, + 0, flags); +} + static int cmd_delete_refs(struct ref_store *refs, const char **argv) { unsigned int flags = arg_flags(*argv++, "flags", transaction_flags); diff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh index 4fb6988f493994..1b5ed44415f261 100755 --- a/t/t0600-reffiles-backend.sh +++ b/t/t0600-reffiles-backend.sh @@ -369,7 +369,7 @@ test_expect_success 'log diagnoses bogus HEAD hash' ' test_expect_success 'log diagnoses bogus HEAD symref' ' git init empty && - test-tool -C empty ref-store main create-symref HEAD refs/heads/invalid.lock && + test-tool -C empty ref-store main create-symref 0 HEAD refs/heads/invalid.lock && test_must_fail git -C empty log 2>stderr && test_grep broken stderr && test_must_fail git -C empty log --default totally-bogus 2>stderr && diff --git a/t/t1405-main-ref-store.sh b/t/t1405-main-ref-store.sh index 347c53e642b6a7..df8e62c31160ff 100755 --- a/t/t1405-main-ref-store.sh +++ b/t/t1405-main-ref-store.sh @@ -15,7 +15,7 @@ test_expect_success 'setup' ' ' test_expect_success 'create_symref(FOO_HEAD, refs/heads/main)' ' - $RUN create-symref FOO_HEAD refs/heads/main nothing && + $RUN create-symref 0 FOO_HEAD refs/heads/main nothing && echo refs/heads/main >expected && git symbolic-ref FOO_HEAD >actual && test_cmp expected actual diff --git a/t/t1407-worktree-ref-store.sh b/t/t1407-worktree-ref-store.sh index a845d8e691388b..22f1462822880d 100755 --- a/t/t1407-worktree-ref-store.sh +++ b/t/t1407-worktree-ref-store.sh @@ -41,12 +41,12 @@ test_expect_success 'resolve_ref()' ' ' test_expect_success 'create_symref(FOO_HEAD, refs/heads/main)' ' - $RWT create-symref FOO_HEAD refs/heads/main nothing && + $RWT create-symref 0 FOO_HEAD refs/heads/main nothing && echo refs/heads/main >expected && git -C wt symbolic-ref FOO_HEAD >actual && test_cmp expected actual && - $RMAIN create-symref FOO_HEAD refs/heads/wt-main nothing && + $RMAIN create-symref 0 FOO_HEAD refs/heads/wt-main nothing && echo refs/heads/wt-main >expected && git symbolic-ref FOO_HEAD >actual && test_cmp expected actual diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index 13faaf1d84c0d5..bd3af4b8fd26b4 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -152,7 +152,7 @@ test_expect_success 'rev-parse skips symref pointing to broken name' ' test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && git branch shadow one && test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && - test-tool ref-store main create-symref refs/tags/shadow refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/tags/shadow refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/tags/shadow" && git rev-parse --verify one >expect && git rev-parse --verify shadow >actual 2>err && @@ -163,9 +163,9 @@ test_expect_success 'rev-parse skips symref pointing to broken name' ' test_expect_success 'for-each-ref emits warnings for broken names' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && - test-tool ref-store main create-symref refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && git for-each-ref >output 2>error && test_grep ! -e "broken\.\.\.ref" output && @@ -202,7 +202,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref to broken name' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git update-ref --no-deref -d refs/heads/badname >output 2>error && @@ -214,7 +214,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref to broken name' test_expect_success 'branch -d can delete symref to broken name' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git branch -d badname >output 2>error && @@ -224,7 +224,7 @@ test_expect_success 'branch -d can delete symref to broken name' ' ' test_expect_success 'update-ref --no-deref -d can delete dangling symref to broken name' ' - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git update-ref --no-deref -d refs/heads/badname >output 2>error && @@ -234,7 +234,7 @@ test_expect_success 'update-ref --no-deref -d can delete dangling symref to brok ' test_expect_success 'branch -d can delete dangling symref to broken name' ' - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git branch -d badname >output 2>error && @@ -246,7 +246,7 @@ test_expect_success 'branch -d can delete dangling symref to broken name' ' test_expect_success 'update-ref -d can delete broken name through symref' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/broken...ref && git update-ref -d refs/heads/badname >output 2>error && @@ -256,7 +256,7 @@ test_expect_success 'update-ref -d can delete broken name through symref' ' ' test_expect_success 'update-ref --no-deref -d can delete symref with broken name' ' - test-tool ref-store main create-symref refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git update-ref --no-deref -d refs/heads/broken...symref >output 2>error && @@ -266,7 +266,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref with broken name ' test_expect_success 'branch -d can delete symref with broken name' ' - test-tool ref-store main create-symref refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git branch -d broken...symref >output 2>error && @@ -276,7 +276,7 @@ test_expect_success 'branch -d can delete symref with broken name' ' ' test_expect_success 'update-ref --no-deref -d can delete dangling symref with broken name' ' - test-tool ref-store main create-symref refs/heads/broken...symref refs/heads/idonotexist && + test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/idonotexist && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git update-ref --no-deref -d refs/heads/broken...symref >output 2>error && @@ -286,7 +286,7 @@ test_expect_success 'update-ref --no-deref -d can delete dangling symref with br ' test_expect_success 'branch -d can delete dangling symref with broken name' ' - test-tool ref-store main create-symref refs/heads/broken...symref refs/heads/idonotexist && + test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/idonotexist && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git branch -d broken...symref >output 2>error && diff --git a/t/t1450-fsck.sh b/t/t1450-fsck.sh index 77cd96de78eced..319308add3ea14 100755 --- a/t/t1450-fsck.sh +++ b/t/t1450-fsck.sh @@ -110,7 +110,7 @@ test_expect_success REFFILES 'HEAD link pointing at a funny object' ' test_expect_success 'HEAD link pointing at a funny place' ' test_when_finished "git update-ref --no-deref HEAD $orig_head" && - test-tool ref-store main create-symref HEAD refs/funny/place && + test-tool ref-store main create-symref 0 HEAD refs/funny/place && # avoid corrupt/broken HEAD from interfering with repo discovery test_must_fail env GIT_DIR=.git git fsck 2>out && test_grep "HEAD: badHeadTarget: HEAD points to non-branch ${SQ}refs/funny/place${SQ}" out diff --git a/t/t2402-worktree-list.sh b/t/t2402-worktree-list.sh index eeaf800d74fc5f..845bf352106717 100755 --- a/t/t2402-worktree-list.sh +++ b/t/t2402-worktree-list.sh @@ -260,7 +260,7 @@ test_expect_success 'broken main worktree still at the top' ' EOF cd linked && echo "worktree $(pwd)" >expected && - (cd ../ && test-tool ref-store main create-symref HEAD .broken ) && + (cd ../ && test-tool ref-store main create-symref 0 HEAD .broken ) && git worktree list --porcelain >out && head -n 3 out >actual && test_cmp ../expected actual && From 0e523f7a045bb724bb6b10a6b8c5cf601aa99126 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 1 May 2024 21:31:59 -0400 Subject: [PATCH 096/156] refs: check syntax of symbolic ref names Historically, neither the ref subsytem nor the symbolic-ref command checked the syntax of refnames at all. And you could run garbage like: git symbolic-ref refs/../../../another-repo/.git/HEAD refs/heads/foo to escape the refs/ namespace and write into another repository (assuming the files backend, of course). That improved a little bit in 300b38e46f (refs: use transaction in `refs_create_symref()`, 2024-05-07). Since then we're built on top of the ref transaction subsystem, which does try to check the validity of the names. But there's a bug! There are two levels of checks. For a regular ref creation or update, we use check_refname_format(), which enforces all of the rules. But for a deletion, we use only refname_is_safe(), which is meant to allow deleting malformed cruft without allowing too much mischief (like the refs/ hierarchy escape above). But those checks in the ref transaction code don't understand symref updates, and mistake a symref update for a deletion (since it has no "new oid" field). As a result, since 300b38e46f we'd check refname_is_safe(), preventing the "../../" escape above, but still allow violations of the stricter check_refname_format() rules. For example, running: git symbolic-ref refs/.broken refs/heads/foo would happily write to that invalid refname. We can fix that by teaching transaction_refname_valid() to consider both new_oid and new_target when looking for a deletion. Two new tests at the end of t1430 cover this area. The "unsafe name" test passes even without this patch, but failed before 300b38e46f, demonstrating the switch to calling refname_is_safe() in that commit. And the "invalid name" test failed before this patch, showing that we were not properly calling check_refname_format(). And it now passes. We also have to update a few other tests in t1430 that use "test-tool ref-store" to intentionally create invalid refnames. They can pass the SKIP_REFNAME_VERIFICATION flag to loosen the rules, just like the nearby non-symref tests do. --- refs.c | 8 +++++--- t/t1430-bad-ref-name.sh | 18 +++++++++++++----- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/refs.c b/refs.c index 5a6eb522d5932b..25c52e3a4f1a83 100644 --- a/refs.c +++ b/refs.c @@ -1377,6 +1377,7 @@ struct ref_update *ref_transaction_add_update( static int transaction_refname_valid(const char *refname, const struct object_id *new_oid, + const char *new_target, unsigned int flags, struct strbuf *err) { if (flags & REF_SKIP_REFNAME_VERIFICATION) @@ -1390,7 +1391,7 @@ static int transaction_refname_valid(const char *refname, refusal_msg = _("refusing to update pseudoref '%s'"); strbuf_addf(err, refusal_msg, refname); return 0; - } else if ((new_oid && !is_null_oid(new_oid)) ? + } else if (((new_oid && !is_null_oid(new_oid)) || new_target) ? check_refname_format(refname, REFNAME_FULLY_QUALIFIED) : !refname_is_safe(refname)) { const char *refusal_msg; @@ -1424,7 +1425,8 @@ enum ref_transaction_error ref_transaction_update(struct ref_transaction *transa return REF_TRANSACTION_ERROR_GENERIC; } - if (!transaction_refname_valid(refname, new_oid, flags, err)) + if (!transaction_refname_valid(refname, new_oid, new_target, flags, + err)) return REF_TRANSACTION_ERROR_GENERIC; if (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS) @@ -1488,7 +1490,7 @@ int ref_transaction_update_reflog(struct ref_transaction *transaction, flags = REF_HAVE_OLD | REF_HAVE_NEW | REF_LOG_ONLY | REF_FORCE_CREATE_REFLOG | REF_NO_DEREF | REF_LOG_USE_PROVIDED_OIDS; - if (!transaction_refname_valid(refname, new_oid, flags, err)) + if (!transaction_refname_valid(refname, new_oid, NULL, flags, err)) return -1; update = ref_transaction_add_update(transaction, refname, flags, diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index bd3af4b8fd26b4..5f2b0492c00107 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -165,7 +165,7 @@ test_expect_success 'for-each-ref emits warnings for broken names' ' test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && - test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && git for-each-ref >output 2>error && test_grep ! -e "broken\.\.\.ref" output && @@ -256,7 +256,7 @@ test_expect_success 'update-ref -d can delete broken name through symref' ' ' test_expect_success 'update-ref --no-deref -d can delete symref with broken name' ' - test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git update-ref --no-deref -d refs/heads/broken...symref >output 2>error && @@ -266,7 +266,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref with broken name ' test_expect_success 'branch -d can delete symref with broken name' ' - test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/main && + test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git branch -d broken...symref >output 2>error && @@ -276,7 +276,7 @@ test_expect_success 'branch -d can delete symref with broken name' ' ' test_expect_success 'update-ref --no-deref -d can delete dangling symref with broken name' ' - test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/idonotexist && + test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/idonotexist && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git update-ref --no-deref -d refs/heads/broken...symref >output 2>error && @@ -286,7 +286,7 @@ test_expect_success 'update-ref --no-deref -d can delete dangling symref with br ' test_expect_success 'branch -d can delete dangling symref with broken name' ' - test-tool ref-store main create-symref 0 refs/heads/broken...symref refs/heads/idonotexist && + test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/idonotexist && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && test_ref_exists refs/heads/broken...symref && git branch -d broken...symref >output 2>error && @@ -419,4 +419,12 @@ test_expect_success 'rev-parse enforces root ref naming convention' ' test_must_fail git rev-parse --verify BAD_NAME ' +test_expect_success 'refuse invalid symbolic ref name' ' + test_must_fail git symbolic-ref refs/.broken refs/heads/ok +' + +test_expect_success 'refuse unsafe symbolic ref name' ' + test_must_fail git symbolic-ref refs/../../escape refs/heads/ok +' + test_done From 46ab7c9300bac7cea443d8eb1bdaac23bbe4a1d3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:48:44 -0400 Subject: [PATCH 097/156] t3200: fully qualify symref destination We create a dangling symref that points to "nowhere", which is not by itself a valid refname (it is not in the refs/ hierarchy, and it does not meet the all-caps syntactic requirement for a root ref). We don't enforce these rules yet, but let's update the test in preparation. --- t/t3200-branch.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh index cdb6c6a6342a0c..cc330da4608a54 100755 --- a/t/t3200-branch.sh +++ b/t/t3200-branch.sh @@ -821,9 +821,9 @@ test_expect_success 'deleting a symref' ' ' test_expect_success 'deleting a dangling symref' ' - git symbolic-ref refs/heads/dangling-symref nowhere && + git symbolic-ref refs/heads/dangling-symref refs/does/not/exist && git symbolic-ref --no-recurse refs/heads/dangling-symref && - echo "Deleted branch dangling-symref (was nowhere)." >expect && + echo "Deleted branch dangling-symref (was refs/does/not/exist)." >expect && git branch -d dangling-symref >actual && test_ref_missing refs/heads/dangling-symref && test_cmp expect actual From 58bc858532ac38a71dd85874be6a1084f1811db7 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 10 Jan 2026 03:59:54 -0500 Subject: [PATCH 098/156] t0614: use test-tool to create garbage symref We use "git symbolic-ref" to create a symref that points to an invalid refname, "garbage", so we can see how fsck reacts. Doing so is not currently forbidden, but will become so in a future patch. Let's instead use the test-tool helper that lets us specify that we wish to disable checks. --- t/t0614-reftable-fsck.sh | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/t/t0614-reftable-fsck.sh b/t/t0614-reftable-fsck.sh index d24b87f9611975..6d2d30e0cfcab8 100755 --- a/t/t0614-reftable-fsck.sh +++ b/t/t0614-reftable-fsck.sh @@ -91,7 +91,10 @@ test_expect_success 'invalid symref gets reported' ' test_when_finished "rm -rf repo" && git init repo && test_commit -C repo initial && - git -C repo symbolic-ref refs/heads/symref garbage && + ( + cd repo && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/symref garbage + ) && test_must_fail git -C repo refs verify 2>err && cat >expect <<-EOF && error: refs/heads/symref: badReferentName: points to invalid refname ${SQ}garbage${SQ} From 2054c735224ffbd5ef5ce1943427fb05df12f3ad Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 15 Jun 2025 23:51:54 -0400 Subject: [PATCH 099/156] symbolic-ref: tighten syntax check of target refname When creating or updating a symref, we pass the destination refname to check_refname_format() to make sure it is a valid refname. But because we pass the ALLOW_ONELEVEL flag, that function can't enforce the rules for root refs (it doesn't know if we are passing a fully qualified ref and are OK with it being a top-level ref, or if we are passing a partial component that resides within the refs/ namespace). We know that symrefs always store fully qualified refnames. So we can pass the new FULLY_QUALIFIED flag to let check_refname_format() know that it can enforce the root-ref syntax rules. The test shows off the change in behavior: we'll now forbid pointing to a root ref named FOO (it would have to be FOO_HEAD or similar). --- builtin/symbolic-ref.c | 2 +- t/t1401-symbolic-ref.sh | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/builtin/symbolic-ref.c b/builtin/symbolic-ref.c index 231e41e715b95a..868d4df65c0029 100644 --- a/builtin/symbolic-ref.c +++ b/builtin/symbolic-ref.c @@ -86,7 +86,7 @@ int cmd_symbolic_ref(int argc, if (!strcmp(argv[0], "HEAD") && !starts_with(argv[1], "refs/")) die("Refusing to point HEAD outside of refs/"); - if (check_refname_format(argv[1], REFNAME_ALLOW_ONELEVEL) < 0) + if (check_refname_format(argv[1], REFNAME_FULLY_QUALIFIED) < 0) die("Refusing to set '%s' to invalid ref '%s'", argv[0], argv[1]); ret = !!refs_update_symref(get_main_ref_store(the_repository), argv[0], argv[1], msg); diff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh index b78dcac7a96c3a..f9c6300b0b5091 100755 --- a/t/t1401-symbolic-ref.sh +++ b/t/t1401-symbolic-ref.sh @@ -174,6 +174,11 @@ test_expect_success 'symbolic-ref allows top-level target for non-HEAD' ' test_cmp_rev top-level HEAD ' +test_expect_success 'symbolic-ref rejects invalid root ref as target' - <<\EOT + test_must_fail git symbolic-ref refs/heads/invalid FOO 2>err && + test_grep "Refusing to set 'refs/heads/invalid' to invalid ref 'FOO'" err +EOT + test_expect_success 'symbolic-ref pointing at another' ' git update-ref refs/heads/maint-2.37 HEAD && git symbolic-ref refs/heads/maint refs/heads/maint-2.37 && From b223d1ce1adf203601bb3ac10554035d24853d03 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 8 Aug 2024 20:18:43 -0400 Subject: [PATCH 100/156] pager: save the original stderr when redirecting to pager When we redirect stdout to the pager, we also redirect stderr (if it would otherwise go to the terminal) so that error messages do not get overwritten by the pager. However, some stderr output may still want to go to the terminal, because they are time-sensitive (like progress reports) and should be overwritten by the pager. This patch stashes away the original stderr descriptor and creates a new stdio buffer for it. Signed-off-by: Jeff King --- pager.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pager.c b/pager.c index 543ef129366a19..fb07857652ef64 100644 --- a/pager.c +++ b/pager.c @@ -14,6 +14,7 @@ int pager_use_color = 1; #define DEFAULT_PAGER "less" #endif +FILE *original_stderr; static struct child_process pager_process; static int old_fd1 = -1, old_fd2 = -1; @@ -188,6 +189,7 @@ void setup_pager(struct repository *r) dup2(pager_process.in, 1); if (isatty(2)) { old_fd2 = dup(2); + original_stderr = fdopen(old_fd2, "w"); dup2(pager_process.in, 2); } close(pager_process.in); From 022a039cc0dbe03d9c2c5b2447dcc74c63c15a60 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 9 Jul 2025 03:02:28 -0400 Subject: [PATCH 101/156] t1400: fix broken target of symref update While testing "update-ref --stdin --batch-updates", we invoke it with this input: update refs/heads/ref1 symref-update refs/heads/ref2 ref refs/heads/nonexistent With the idea being that both ref1 and ref2 are regular, non-symref refs, and we should refuse to update ref2 because it doesn't match the expected "ref refs/heads/nonexistent" value. But there's an oddity here: we are asking to do a symref-update to an oid, not to a valid symref value. This is nonsense, and would point to a bogus symref (a refname with the literal value of the oid). But because we don't yet enforce ref syntax rules for symref targets, the test is able to continue and rejects the ref for the reason it expects. However, once a future patch starts enforcing those syntax rules, we'll start rejecting the update of "ref2" for that reason instead. Let's switch out the oid for a plausible ref value. The point of the test is that we should never get as far as actually updating ref2 to that value. By giving it a syntactically valid target, we future proof against tighter enforcement of the rules. --- t/t1400-update-ref.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index 147c89c53ca985..5efb57af7d5059 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -2223,7 +2223,7 @@ do git update-ref refs/heads/ref2 $head && format_command $type "update refs/heads/ref1" "$old_head" "$head" >stdin && - format_command $type "symref-update refs/heads/ref2" "$old_head" "ref" "refs/heads/nonexistent" >>stdin && + format_command $type "symref-update refs/heads/ref2" "refs/heads/new" "ref" "refs/heads/nonexistent" >>stdin && git update-ref $type --no-deref --stdin --batch-updates stdout 2>err && echo $old_head >expect && git rev-parse refs/heads/ref1 >actual && From e1719798817b7934bc64ec35536a3284bdb03b3b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 23 Mar 2011 14:07:52 -0400 Subject: [PATCH 102/156] progress: use pager's original_stderr if available If we are outputting to a pager, stderr is redirected to the pager. However, progress messages should not be part of that stream, as they are time-sensitive and should end up being hidden once we actually have output. Signed-off-by: Jeff King --- environment.h | 2 ++ progress.c | 12 +++++++----- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/environment.h b/environment.h index e7ec5b0437342d..13648b6b46b642 100644 --- a/environment.h +++ b/environment.h @@ -84,6 +84,8 @@ */ extern const char * const local_repo_env[]; +extern FILE *original_stderr; + struct strvec; struct repository; diff --git a/progress.c b/progress.c index 8315bdc3d4a77a..487cd5005c20ca 100644 --- a/progress.c +++ b/progress.c @@ -20,6 +20,7 @@ #include "trace2.h" #include "utf8.h" #include "parse.h" +#include "environment.h" #define TP_IDX_MAX 8 @@ -111,6 +112,7 @@ static int is_foreground_fd(int fd) static void display(struct progress *progress, uint64_t n, const char *done) { + FILE *out = original_stderr ? original_stderr : stderr; const char *tp; struct strbuf *counters_sb = &progress->counters_sb; int show_update = 0; @@ -143,7 +145,7 @@ static void display(struct progress *progress, uint64_t n, const char *done) } if (show_update) { - if (is_foreground_fd(fileno(stderr)) || done) { + if (is_foreground_fd(fileno(out)) || done) { const char *eol = done ? done : "\r"; size_t clear_len = counters_sb->len < last_count_len ? last_count_len - counters_sb->len + 1 : @@ -154,20 +156,20 @@ static void display(struct progress *progress, uint64_t n, const char *done) int cols = term_columns(); if (progress->split) { - fprintf(stderr, " %s%*s", counters_sb->buf, + fprintf(out, " %s%*s", counters_sb->buf, (int) clear_len, eol); } else if (!done && cols < progress_line_len) { clear_len = progress->title_len + 1 < cols ? cols - progress->title_len - 1 : 0; - fprintf(stderr, "%s:%*s\n %s%s", + fprintf(out, "%s:%*s\n %s%s", progress->title, (int) clear_len, "", counters_sb->buf, eol); progress->split = 1; } else { - fprintf(stderr, "%s: %s%*s", progress->title, + fprintf(out, "%s: %s%*s", progress->title, counters_sb->buf, (int) clear_len, eol); } - fflush(stderr); + fflush(out); } } } From 7e9a04aee2e07aab90d6458198fc8254267e4a5a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 2 May 2024 00:51:19 -0400 Subject: [PATCH 103/156] refs: check syntax of symref target The symbolic-ref command checks that the target a symref points to is syntactically well-formed. But it does so itself, and direct calls to refs_update_symref() from other commands get no protection. In theory this could allow you to do bad things, like pointing a symref at refs/../../../some-file and reading the result. But in practice I don't think this is too bad: - On the resolving side, we make sure that any targets we read are syntactically valid. So even if you can make such a bogus symref, it can't be used to read or write outside of refs/. - Looking over the other callers, I don't see any that can obviously be fed bogus input. They usually take the names of existing branches (e.g., to stick into HEAD). The biggest exception I found is "init --initial-branch", but it manually checks the validity of the branch name. So I don't think there are any exploitable bugs here. But I think it's better to enforce these rules within the ref code. That makes sure that we haven't missed any cases. And even though we'd refuse to read the refs, it's more helpful to detect the problem before even writing them. Let's move the check from the symbolic-ref command into the refs code. We can confirm via the existing tests that the bogus refnames are still rejected. However, we do need a few test updates: - I kept the same error text as the symbolic-ref message, but I lower-cased the first word to match our usual style. As a result, the test in t1401 needs to be updated to match. - A few tests (especially in t1430) use "test-tool ref-store" to intentionally create symrefs pointing to bogus targets. We can pass a flag there to disable verification. There's an interesting question here: which flag do we use? There is a flag for SKIP_REFNAME_VERIFICATION, but it is about the refname we are writing, not the destination. There is also SKIP_OID_VERIFICATION, which loosens checks on the destination. But of course there is no oid here at all, just a target refname. I went with SKIP_OID_VERIFICATION, as that seemed to be the moral equivalent for symrefs. We could add a new flag, but I don't think it's worth doing so. None of this is user-facing (in fact, most of these flags aren't even triggerable by real Git commands, and are only for the ref-store test-tool). --- builtin/symbolic-ref.c | 2 -- refs.c | 8 ++++++++ t/t0600-reffiles-backend.sh | 2 +- t/t1401-symbolic-ref.sh | 2 +- t/t1430-bad-ref-name.sh | 14 +++++++------- t/t2402-worktree-list.sh | 2 +- 6 files changed, 18 insertions(+), 12 deletions(-) diff --git a/builtin/symbolic-ref.c b/builtin/symbolic-ref.c index 868d4df65c0029..6274f204e80650 100644 --- a/builtin/symbolic-ref.c +++ b/builtin/symbolic-ref.c @@ -86,8 +86,6 @@ int cmd_symbolic_ref(int argc, if (!strcmp(argv[0], "HEAD") && !starts_with(argv[1], "refs/")) die("Refusing to point HEAD outside of refs/"); - if (check_refname_format(argv[1], REFNAME_FULLY_QUALIFIED) < 0) - die("Refusing to set '%s' to invalid ref '%s'", argv[0], argv[1]); ret = !!refs_update_symref(get_main_ref_store(the_repository), argv[0], argv[1], msg); break; diff --git a/refs.c b/refs.c index 25c52e3a4f1a83..07fd04ba39c18b 100644 --- a/refs.c +++ b/refs.c @@ -1429,6 +1429,14 @@ enum ref_transaction_error ref_transaction_update(struct ref_transaction *transa err)) return REF_TRANSACTION_ERROR_GENERIC; + if (new_target && !(flags & REF_SKIP_OID_VERIFICATION) && + check_refname_format(new_target, REFNAME_FULLY_QUALIFIED) < 0) { + strbuf_addf(err, + _("refusing to set '%s' to invalid ref '%s'"), + refname, new_target); + return -1; + } + if (flags & ~REF_TRANSACTION_UPDATE_ALLOWED_FLAGS) BUG("illegal flags 0x%x passed to ref_transaction_update()", flags); diff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh index 1b5ed44415f261..cd1d01efdbd532 100755 --- a/t/t0600-reffiles-backend.sh +++ b/t/t0600-reffiles-backend.sh @@ -369,7 +369,7 @@ test_expect_success 'log diagnoses bogus HEAD hash' ' test_expect_success 'log diagnoses bogus HEAD symref' ' git init empty && - test-tool -C empty ref-store main create-symref 0 HEAD refs/heads/invalid.lock && + test-tool -C empty ref-store main create-symref REF_SKIP_OID_VERIFICATION HEAD refs/heads/invalid.lock && test_must_fail git -C empty log 2>stderr && test_grep broken stderr && test_must_fail git -C empty log --default totally-bogus 2>stderr && diff --git a/t/t1401-symbolic-ref.sh b/t/t1401-symbolic-ref.sh index f9c6300b0b5091..7e3b7768d58244 100755 --- a/t/t1401-symbolic-ref.sh +++ b/t/t1401-symbolic-ref.sh @@ -176,7 +176,7 @@ test_expect_success 'symbolic-ref allows top-level target for non-HEAD' ' test_expect_success 'symbolic-ref rejects invalid root ref as target' - <<\EOT test_must_fail git symbolic-ref refs/heads/invalid FOO 2>err && - test_grep "Refusing to set 'refs/heads/invalid' to invalid ref 'FOO'" err + test_grep "refusing to set 'refs/heads/invalid' to invalid ref 'FOO'" err EOT test_expect_success 'symbolic-ref pointing at another' ' diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index 5f2b0492c00107..5f46b45b398294 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -152,7 +152,7 @@ test_expect_success 'rev-parse skips symref pointing to broken name' ' test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && git branch shadow one && test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && - test-tool ref-store main create-symref 0 refs/tags/shadow refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/tags/shadow refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/tags/shadow" && git rev-parse --verify one >expect && git rev-parse --verify shadow >actual 2>err && @@ -163,7 +163,7 @@ test_expect_success 'rev-parse skips symref pointing to broken name' ' test_expect_success 'for-each-ref emits warnings for broken names' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test-tool ref-store main create-symref REF_SKIP_REFNAME_VERIFICATION refs/heads/broken...symref refs/heads/main && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...symref" && @@ -202,7 +202,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref to broken name' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git update-ref --no-deref -d refs/heads/badname >output 2>error && @@ -214,7 +214,7 @@ test_expect_success 'update-ref --no-deref -d can delete symref to broken name' test_expect_success 'branch -d can delete symref to broken name' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git branch -d badname >output 2>error && @@ -224,7 +224,7 @@ test_expect_success 'branch -d can delete symref to broken name' ' ' test_expect_success 'update-ref --no-deref -d can delete dangling symref to broken name' ' - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git update-ref --no-deref -d refs/heads/badname >output 2>error && @@ -234,7 +234,7 @@ test_expect_success 'update-ref --no-deref -d can delete dangling symref to brok ' test_expect_success 'branch -d can delete dangling symref to broken name' ' - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/badname && git branch -d badname >output 2>error && @@ -246,7 +246,7 @@ test_expect_success 'branch -d can delete dangling symref to broken name' ' test_expect_success 'update-ref -d can delete broken name through symref' ' test-tool ref-store main update-ref msg "refs/heads/broken...ref" $main_sha1 $ZERO_OID REF_SKIP_REFNAME_VERIFICATION && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/broken...ref" && - test-tool ref-store main create-symref 0 refs/heads/badname refs/heads/broken...ref msg && + test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION refs/heads/badname refs/heads/broken...ref msg && test_when_finished "test-tool ref-store main delete-refs REF_NO_DEREF msg refs/heads/badname" && test_ref_exists refs/heads/broken...ref && git update-ref -d refs/heads/badname >output 2>error && diff --git a/t/t2402-worktree-list.sh b/t/t2402-worktree-list.sh index 845bf352106717..12c9107c657dff 100755 --- a/t/t2402-worktree-list.sh +++ b/t/t2402-worktree-list.sh @@ -260,7 +260,7 @@ test_expect_success 'broken main worktree still at the top' ' EOF cd linked && echo "worktree $(pwd)" >expected && - (cd ../ && test-tool ref-store main create-symref 0 HEAD .broken ) && + (cd ../ && test-tool ref-store main create-symref REF_SKIP_OID_VERIFICATION HEAD .broken ) && git worktree list --porcelain >out && head -n 3 out >actual && test_cmp ../expected actual && From 93fc87e012e3edd11d81e35cc7990075c9772132 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 23 Mar 2011 14:09:46 -0400 Subject: [PATCH 104/156] show: turn on rename detection progress reporting For large commits, it is nice to have some eye candy for the rename detection. However, because show can display multiple commits, we have to be careful not to clutter existing output. We show the progress report only before we have generated any actual output; once we have sent output to the terminal or pager, we turn off progress reporting. This also makes it safe to use with "git log", though it will only be useful if the first commit is the slow one. So this patch actually enables it for all of the log/whatchanged/show/reflog family. We also handle the usual --{no-}progress option and check that stderr goes to a terminal before turning on progress. Signed-off-by: Jeff King --- Documentation/git-log.adoc | 7 +++++++ builtin/log.c | 15 ++++++++++++++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/Documentation/git-log.adoc b/Documentation/git-log.adoc index fb3ac112839cf7..810f2cf0d1e79b 100644 --- a/Documentation/git-log.adoc +++ b/Documentation/git-log.adoc @@ -117,6 +117,13 @@ __:: Paths may need to be prefixed with `--` to separate them from options or the revision range, when confusion arises. +`--progress`:: +`--no-progress`:: + Enable or disable progress reporting during long computations; + the default is to enable progress reporting when stderr is a + terminal. Currently the only computation with progress support + is inexact rename detection. + include::rev-list-options.adoc[] include::pretty-formats.adoc[] diff --git a/builtin/log.c b/builtin/log.c index 350b35c556362d..a09eff376de61e 100644 --- a/builtin/log.c +++ b/builtin/log.c @@ -59,6 +59,7 @@ static unsigned int force_in_body_from; static int stdout_mboxrd; static int format_no_prefix; +static int progress = -1; static const char * const builtin_log_usage[] = { N_("git log [] [] [[--] ...]"), @@ -299,6 +300,8 @@ static void cmd_log_init_finish(int argc, const char **argv, const char *prefix, OPT_CALLBACK('L', NULL, &line_cb, "range:file", N_("trace the evolution of line range , or function : in "), log_line_range_callback), + OPT_SET_INT(0, "progress", &progress, + "force progress reporting", 1), OPT_END() }; @@ -345,6 +348,9 @@ static void cmd_log_init_finish(int argc, const char **argv, const char *prefix, read_mailmap(the_repository, rev->mailmap); } + if (progress == -1) + progress = isatty(2); + if (rev->pretty_given && rev->commit_format == CMIT_FMT_RAW) { /* * "log --pretty=raw" is special; ignore UI oriented @@ -406,18 +412,25 @@ static int cmd_log_walk_no_free(struct rev_info *rev) if (prepare_revision_walk(rev)) die(_("revision walk setup failed")); + if (progress) + rev->diffopt.show_rename_progress = 1; + /* * For --check and --exit-code, the exit code is based on CHECK_FAILED * and HAS_CHANGES being accumulated in rev->diffopt, so be careful to * retain that state information if replacing rev->diffopt in this loop */ while ((commit = get_revision(rev)) != NULL) { - if (!log_tree_commit(rev, commit) && rev->max_count >= 0) + int showed = log_tree_commit(rev, commit); + if (!showed && rev->max_count >= 0) /* * We decremented max_count in get_revision, * but we didn't actually show the commit. */ rev->max_count++; + /* Once we have output, progress will clutter the terminal. */ + if (showed) + rev->diffopt.show_rename_progress = 0; if (!rev->reflog_info && !rev->remerge_diff) { /* * We may show a given commit multiple times when From f177ac66379fb629545f60772b2c299b21d44c5a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sat, 1 Jun 2024 04:24:45 -0400 Subject: [PATCH 105/156] refs: enforce name conventions for root refs We recently taught check_refname_format() to insist that any ref outside of "refs/" match the is_root_ref_syntax() rules. This can reduce the chance of accidentally reading or writing a non-ref file in ".git" when the files backend is in use. For case-sensitive filesystems, this should mostly work. We do not generally create all-caps files under .git/ unless they are meant to be root refs. But for case-insensitive filesystems, it's less clear. Asking to write to "CONFIG" would try to access the actual ".git/config" file (though fortunately this does not work, as we refuse to write anything we can't parse as a ref). We already have functions that catalog the pattern of allowable names (i.e., ending in "_HEAD" or one of a set of historical exceptions). So even if we allowed creating such a ref, we'd skip over it while iterating over the root refs. But we don't enforce those rules; we check only the "all caps and underscore" syntax rule, not the full root-ref rules. Let's teach check_refname_format() to use those functions to further restrict what it will allow at the root level. That should make things safer and more consistent with ref iteration. Note that while is_root_ref_syntax() covers the syntax for both regular root refs and pseudo-refs (FETCH_HEAD and MERGE_HEAD), the is_root_ref() function does not include pseudo-refs. So we have to check the two classes separately. This patch doesn't touch refname_is_safe(), which generally tries to be a bit more loose (e.g., to allow deletion of bogus names). I've left it loose here, though arguably it would benefit from some of the same protection (you wouldn't want to delete ".git/config" either, though again, we'd refuse to delete something we can't parse). --- refs.c | 3 ++- t/t1430-bad-ref-name.sh | 10 ++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/refs.c b/refs.c index 62295d1502d0fb..ee14bdebaff172 100644 --- a/refs.c +++ b/refs.c @@ -284,7 +284,8 @@ static int check_or_sanitize_refname(const char *refname, int flags, parse_worktree_ref(refname, NULL, NULL, &bare_ref); if (!starts_with(bare_ref, "refs/") && - !is_root_ref_syntax(bare_ref)) + !is_pseudo_ref(bare_ref) && + !is_root_ref(bare_ref)) return -1; } diff --git a/t/t1430-bad-ref-name.sh b/t/t1430-bad-ref-name.sh index bcad6edf219a0a..13faaf1d84c0d5 100755 --- a/t/t1430-bad-ref-name.sh +++ b/t/t1430-bad-ref-name.sh @@ -399,6 +399,11 @@ test_expect_success 'update-ref refuses non-underscore punctuation outside of re test_grep "refusing to update ref with bad name" err ' +test_expect_success 'update-ref enforces root ref naming convention' ' + test_must_fail git update-ref FOO_BAR HEAD 2>err && + test_grep "refusing to update ref with bad name" err +' + test_expect_success 'rev-parse refuses non-root-ref outside of refs/' ' git rev-parse HEAD >.git/bad && test_must_fail git rev-parse --verify bad @@ -409,4 +414,9 @@ test_expect_success 'rev-parse recognizes non-root-ref via worktree' ' test_must_fail git rev-parse --verify main-worktree/bad ' +test_expect_success 'rev-parse enforces root ref naming convention' ' + git rev-parse HEAD >.git/BAD_NAME && + test_must_fail git rev-parse --verify BAD_NAME +' + test_done From 23a1924783225c18fb6f292a6617b0850fcd342b Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 9 Jul 2025 03:25:32 -0400 Subject: [PATCH 106/156] update-ref: verify stdin refnames as fully qualified When parsing refnames on stdin, we try to make sure they are syntactically valid before handing them off to the ref transaction. But we only check with the ALLOW_ONELEVEL flag, but they really should be fully-qualified refs. The ref code will eventually complain about these names, but if we are going to bother checking them up front, we should do so using the FULLY_QUALIFIED flag, which catches names like "invalid-at-root" which are valid "onelevel" names (like "refs/heads/invalid-at-root") but not actual root names. --- builtin/update-ref.c | 4 ++-- t/t1400-update-ref.sh | 8 ++++++++ 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/builtin/update-ref.c b/builtin/update-ref.c index 6355c3dd3e7fc5..e3f4b1c28aff08 100644 --- a/builtin/update-ref.c +++ b/builtin/update-ref.c @@ -83,7 +83,7 @@ static char *parse_refname(const char **next) return NULL; } - if (check_refname_format(ref.buf, REFNAME_ALLOW_ONELEVEL)) + if (check_refname_format(ref.buf, REFNAME_FULLY_QUALIFIED)) die("invalid ref format: %s", ref.buf); return strbuf_detach(&ref, NULL); @@ -371,7 +371,7 @@ static void parse_cmd_symref_update(struct ref_transaction *transaction, have_old_oid = 1; } else if (!strcmp(old_arg, "ref")) { - if (check_refname_format(old_target, REFNAME_ALLOW_ONELEVEL)) + if (check_refname_format(old_target, REFNAME_FULLY_QUALIFIED)) die("symref-update %s: invalid ref: %s", refname, old_target); } else { die("symref-update %s: invalid arg '%s' for old value", refname, old_arg); diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index 5efb57af7d5059..45f1a856614589 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -2485,4 +2485,12 @@ test_expect_success 'dangling symref overwritten without old oid' ' test_must_fail git rev-parse --verify refs/heads/does-not-exist ' +test_expect_success 'update-ref validates refs as fully qualified' ' + git symbolic-ref refs/heads/valid refs/heads/one && + test_must_fail git update-ref --no-deref --stdin <<-\EOF 2>err && + symref-update refs/heads/valid refs/heads/two ref invalid-at-root + EOF + test_grep "invalid ref: invalid-at-root" err +' + test_done From c905af10153f9b835f49a0e9669b429e912bc7f9 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 24 Mar 2011 12:41:00 -0400 Subject: [PATCH 107/156] diff: turn on rename detection progress reporting Since all of the progress happens before we generate any output, this looks OK, even when output goes to a pager. We do the usual --progress/--no-progress options and check isatty(2) to enable the feature. The argument parsing is a little ad-hoc, but we currently have no parse-options infrastructure here at all. However, it should be safe to parse like this, because the prior call to setup_revisions will have removed any options that take an argument, and our parsing removes --progress from argv for later parsers. The one exception is diff_no_index, which may get called early, and needs to learn to ignore --progress. Signed-off-by: Jeff King --- Documentation/git-diff.adoc | 7 +++++++ builtin/diff.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/Documentation/git-diff.adoc b/Documentation/git-diff.adoc index 272331afbaec73..388b9110588f41 100644 --- a/Documentation/git-diff.adoc +++ b/Documentation/git-diff.adoc @@ -152,6 +152,13 @@ section "3-Way Merge" for detailed information. "Unmerged". Can be used only when comparing the working tree with the index. +`--no-progress`:: +`--progress`:: + Disable or enable progress reporting during long computations; + the default is to enable progress reporting when stderr is a + terminal. Currently the only computation with progress support + is inexact rename detection. + `...`:: The __ parameters, when given, are used to limit the diff to the named paths (you can give directory diff --git a/builtin/diff.c b/builtin/diff.c index 18b1083e984a35..e4cfeef104fed3 100644 --- a/builtin/diff.c +++ b/builtin/diff.c @@ -410,6 +410,8 @@ int cmd_diff(int argc, int nongit = 0, no_index = 0; int result; struct symdiff sdiff; + int progress = -1; + int unknown_argc, parsed_argc; /* * We could get N tree-ish in the rev.pending_objects list. @@ -539,6 +541,32 @@ int cmd_diff(int argc, diff_setup_done(&rev.diffopt); } + parsed_argc = 0; + for (unknown_argc = i = 1; i < argc; i++) { + const char *arg = argv[i]; + if (!strcmp(arg, "--") || arg[0] != '-') { + int j; + for (j = i; j < argc; j++) + argv[unknown_argc++] = argv[j]; + break; + } + else if (!strcmp(argv[i], "--progress")) + progress = 1; + else if (!strcmp(argv[i], "--no-progress")) + progress = 0; + else { + argv[unknown_argc++] = argv[i]; + continue; + } + parsed_argc++; + } + argc -= parsed_argc; + + if (progress == -1) + progress = isatty(2); + if (progress) + rev.diffopt.show_rename_progress = 1; + rev.diffopt.flags.recursive = 1; rev.diffopt.rotate_to_strict = 1; From 5b68a2f0056f5c7b9553a6756f23620abf039df1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 1 Sep 2026 01:41:55 -0400 Subject: [PATCH 108/156] revision: hang on to "freed" argv elements In setup_revisions() we rewrite the incoming argv array, losing references to the strings it contains. For a synthetic argv array constructed from heap strings, that traditionally meant we leaked those allocated strings. We fixed the leak in cd43948798 (revision: manage memory ownership of argv in setup_revisions(), 2025-09-19). Now callers can tell the revision code that argv entries are allocated and should be freed, which it will do before overwriting them. But this introduced a new bug! The overwritten entries go away as soon as option parsing is finished, but a few options may actually create new references to those strings. And once we free the strings, those stale references become use-after-free bugs. For example, running: git stash show --src-prefix=foo/ demonstrates the problem: 1. The stash command generates its own synthetic argv (because it has to treat the stash specifiers specially) which it then passes to setup_revisions(). 2. Parsing will create a reference to the partial string "foo/" in revs.diffopt.a_prefix. 3. When setup_revisions() finishes, we rewrite argv to throw away parsed strings. This frees the entry holding "--src-prefix=foo", at which point we have a dangling reference in revs.diffopt. 4. We generate an actual diff, accessing garbage memory via revs.diffopt.a_prefix. The output is usually garbled, but ASan also detects this reliably. One obvious fix here is to allocate new strings when we pull data out of the argv array. But doing so is error prone (every string option must remember to do it or risk a subtle bug), and creates more questions about memory ownership (e.g., some callers assign string literals directly to a_prefix, and we would not want to free those). Instead we can fix this centrally by delaying the free() calls. We'll collect any "freed" strings in a new array, hold on to it for the life of the rev_info struct, and then release it at the end. We can easily use a strvec for this, since it handles growth and cleanup for us. This fixes the prefix case above (which is now tested in t3903), and should fix any other stray cases. Though I could not find any; we use OPT_STRING only in the prefix diff options, and very few revision opts store strings. Those that do (like --format and --encoding) already make a copy of the string. They do not need for us to hold on to the memory longer, but it does not hurt them if we do. One may note that combined with cd43948798 we have approached a simpler solution in a roundabout way. We are still hacking up argv, but now carefully constructing a parallel argv of old strings we've overwritten (and will eventually free). In an alternate universe, we could instead leave the original argv pristine and return a new reduced-size argv. This is conceptually simpler, though it does mean that every caller must free that new argv array itself (not the entries). That's not something they traditionally had to do, so it would mean tweaking every caller. So even though the combination of this cd43948798 and this patch is a little convoluted, it should make things just work (no leaks and no use-after-free) without modifying any callers. Reported-by: Nicolas Le Cam --- revision.c | 36 ++++++++++++++++++++++++++++-------- revision.h | 9 +++++++++ t/t3903-stash.sh | 17 +++++++++++++++++ 3 files changed, 54 insertions(+), 8 deletions(-) diff --git a/revision.c b/revision.c index 35afe52208e710..f90cc00361fb82 100644 --- a/revision.c +++ b/revision.c @@ -2307,9 +2307,27 @@ static timestamp_t parse_age(const char *arg) return num; } +/* + * When asked to free argv strings, we should not do so immediately. Some + * option parsing may have stored a reference to the string (either the whole + * thing, or a substring inside it). We should keep it valid until the rev_info + * struct itself is freed. + * + * Note that we take a const str for the convenience of callers (who have the + * usual const argv array, even when opt->free_removed_argv_elements is set). + * We cast away the const on their behalf. + */ +static void mark_argv_for_free(struct rev_info *revs, const char *str) +{ + if (!str) + return; + strvec_push_nodup(&revs->argv_to_free, (char *)str); +} + static void overwrite_argv(int *argc, const char **argv, const char **value, - const struct setup_revision_opt *opt) + const struct setup_revision_opt *opt, + struct rev_info *revs) { /* * Detect the case when we are overwriting ourselves. The assignment @@ -2318,7 +2336,7 @@ static void overwrite_argv(int *argc, const char **argv, */ if (*value != argv[*argc]) { if (opt && opt->free_removed_argv_elements) - free((char *)argv[*argc]); + mark_argv_for_free(revs, argv[*argc]); argv[*argc] = *value; *value = NULL; } @@ -2346,7 +2364,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg starts_with(arg, "--branches=") || starts_with(arg, "--tags=") || starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk=")) { - overwrite_argv(unkc, unkv, &argv[0], opt); + overwrite_argv(unkc, unkv, &argv[0], opt, revs); return 1; } @@ -2738,7 +2756,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg } else { int opts = diff_opt_parse(&revs->diffopt, argv, argc, revs->prefix); if (!opts) - overwrite_argv(unkc, unkv, &argv[0], opt); + overwrite_argv(unkc, unkv, &argv[0], opt, revs); return opts; } @@ -3038,7 +3056,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s if (strcmp(arg, "--")) continue; if (opt && opt->free_removed_argv_elements) - free((char *)argv[i]); + mark_argv_for_free(revs, argv[i]); argv[i] = NULL; argc = i; if (argv[i + 1]) @@ -3068,7 +3086,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s if (!strcmp(arg, "--stdin")) { if (revs->disable_stdin) { - overwrite_argv(&left, argv, &argv[i], opt); + overwrite_argv(&left, argv, &argv[i], + opt, revs); continue; } if (revs->read_from_stdin++) @@ -3242,7 +3261,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s if (argv) { if (opt && opt->free_removed_argv_elements) - free((char *)argv[left]); + mark_argv_for_free(revs, argv[left]); argv[left] = NULL; } @@ -3264,7 +3283,7 @@ void setup_revisions_from_strvec(struct strvec *argv, struct rev_info *revs, ret = setup_revisions(argv->nr, argv->v, revs, opt); for (size_t i = ret; i < argv->nr; i++) - free((char *)argv->v[i]); + mark_argv_for_free(revs, argv->v[i]); argv->nr = ret; } @@ -3326,6 +3345,7 @@ void release_revisions(struct rev_info *revs) oidset_clear(&revs->missing_commits); release_revisions_bloom_keyvecs(revs); release_follow_pathspec_slab(revs); + strvec_clear(&revs->argv_to_free); } static void add_child(struct rev_info *revs, struct commit *parent, struct commit *child) diff --git a/revision.h b/revision.h index acf6d06b24126c..e5dabd18ce1ecb 100644 --- a/revision.h +++ b/revision.h @@ -396,6 +396,14 @@ struct rev_info { /* Missing commits to be tracked without failing traversal. */ struct oidset missing_commits; + + /* + * Strings whose ownership has been handed over to us, but which + * we may be referencing in any of the above options (including + * within the diffopt struct). These will remain valid until + * release_revisions() is called. + */ + struct strvec argv_to_free; }; /** @@ -433,6 +441,7 @@ struct rev_info { .commit_format = CMIT_FMT_DEFAULT, \ .expand_tabs_in_log_default = 8, \ .rdiff_log_arg = STRVEC_INIT, \ + .argv_to_free = STRVEC_INIT, \ } /** diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh index da27a6599a6a79..260c809f994bc6 100755 --- a/t/t3903-stash.sh +++ b/t/t3903-stash.sh @@ -780,6 +780,23 @@ test_expect_success 'stash show --patience shows diff' ' diff_cmp expected actual ' +test_expect_success 'stash show supports prefixes' ' + git reset --hard && + echo foo >>file && + git stash && + cat >expected <<-\EOF && + diff --git foo/file bar/file + index 7601807..71b52c4 100644 + --- foo/file + +++ bar/file + @@ -1 +1,2 @@ + baz + +foo + EOF + git stash show --src-prefix=foo/ --dst-prefix=bar/ >actual && + diff_cmp expected actual +' + test_expect_success 'drop: fail early if specified stash is not a stash ref' ' git stash clear && test_when_finished "git reset --hard HEAD && git stash clear" && From 9c4a2a0542da38dae296a5b660f0680c973af31f Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 04:46:41 -0500 Subject: [PATCH 109/156] shortlog: drop redundant shortlog struct initialization We converted the "struct shortlog" from a static variable to a local one in 64093fc06a (blame,shortlog: don't make local option variables static, 2016-06-13). That commit added an initializer, since we'd no longer get the static zero-initialization. But doing so was pointless: the first thing we do with the struct is pass it to shortlog_init(), which zeroes it already. Even worse, the redundant initializer used STRING_LIST_INIT_NODUP for the "list" field (matching the zero-initialization). But shortlog_init() will initialize it in the opposite way, with the strdup_strings field set to 1. This didn't cause a bug since the NODUP state was never used, but it certainly added to the confusion. Let's drop the redundant initialization, making it clear that the call to shortlog_init() is doing the actual work. --- builtin/shortlog.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index 4c78d2e5badb82..eb0dbd641315a3 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -385,7 +385,7 @@ int cmd_shortlog(int argc, const char *prefix, struct repository *repo UNUSED) { - struct shortlog log = { STRING_LIST_INIT_NODUP }; + struct shortlog log; struct rev_info rev; int nongit = !startup_info->have_repository; From 2fb89ee6b389bf16654cc136e72f71531aaa4013 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 03:33:46 -0500 Subject: [PATCH 110/156] shortlog: drop redundant assignment of strdup_strings Before clearing the string_list in a shortlog struct, we assign "1" to its strdup_strings field. But this is pointless; the struct was initialized with this field enabled originally in shortlog_init(). This extra line goes all the way back to b8ec59234b (Build in shortlog, 2006-10-22). It was redundant even then, as the list (a "path_list" at the time) was still initialized with its strdup_paths field enabled. Let's drop the redundant assignment for simplicity and clarity. --- builtin/shortlog.c | 1 - 1 file changed, 1 deletion(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index eb0dbd641315a3..e72e7511eb6f83 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -529,7 +529,6 @@ void shortlog_output(struct shortlog *log) } strbuf_release(&sb); - log->list.strdup_strings = 1; string_list_clear(&log->list, 1); clear_mailmap(&log->mailmap); string_list_clear(&log->format, 0); From aea9d9db6ae5641c25469030ba553d9ad72e8766 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 03:37:22 -0500 Subject: [PATCH 111/156] shortlog: avoid tricky strdup-before-clear string_list construct There's a trick we sometimes do with string_list structs: 1. We declare them as "nodup", not owning their strings. 2. We construct allocated strings elsewhere, and then insert them into the list, handing over memory ownership. 3. We turn on the strdup_strings field before calling string_list_clear(), causing it to free the now-owned strings. This construct isn't wrong per se, but it is error prone. We've had leaks and double-frees in the past that come from trying to tweak the ownership semantics on the fly (I won't catalog them here, but "git log --grep=strdup_strings" will find many complaints from me). Instead, a less confusing way to do this is: 1. Declare the list as "dup" if we intend for it to own the strings. 2. If a caller wishes to append an already-allocated string, use the append_nodup() function to do so. 3. There is no step 3. You can clear as normal. We still have to remember to do the right thing (using the nodup append function), but now it happens right where we are doing the specialized allocation, which is much harder to get wrong. The instance here in shortlog is not wrong or leaky, but I think it is worth switching to the append_nodup() form to make that more obvious. --- builtin/shortlog.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index e72e7511eb6f83..d38cdaa728e6ef 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -87,9 +87,9 @@ static void insert_one_record(struct shortlog *log, if (!item->util) { item->util = xmalloc(sizeof(struct string_list)); - string_list_init_nodup(item->util); + string_list_init_dup(item->util); } - string_list_append(item->util, buffer); + string_list_append_nodup(item->util, buffer); } } @@ -520,7 +520,6 @@ void shortlog_output(struct shortlog *log) fprintf(log->file, " %s\n", msg); } putc('\n', log->file); - onelines->strdup_strings = 1; string_list_clear(onelines, 0); free(onelines); } From f8a142567ccb6391736deebfaa7307c585b9a780 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 03:33:54 -0500 Subject: [PATCH 112/156] shortlog: use initializer function for string_lists We initialize the string_list fields of a shortlog struct by zero-ing the whole struct and then setting their strdup_strings fields. This is OK in practice, as the string_list initializer just zeroes all of the other fields anyway. But it provides a cleaner boundary for us to use the actual string_list initializer function. That will future proof us against the details of string_list changing. As a bonus, removing direct assignments of strdup_strings also makes it easier to audit the code base the confusing strdup-before-clear trick. --- builtin/shortlog.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index d38cdaa728e6ef..4453fe9b6ec761 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -359,13 +359,13 @@ void shortlog_init(struct shortlog *log) read_mailmap(the_repository, &log->mailmap); - log->list.strdup_strings = 1; + string_list_init_dup(&log->list); log->wrap = DEFAULT_WRAPLEN; log->in1 = DEFAULT_INDENT1; log->in2 = DEFAULT_INDENT2; - log->trailers.strdup_strings = 1; + string_list_init_dup(&log->trailers); log->trailers.cmp = strcasecmp; - log->format.strdup_strings = 1; + string_list_init_dup(&log->format); } void shortlog_finish_setup(struct shortlog *log) From d7508676ddeb79cb0091c3c4552f599e5c4080e4 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 04:51:04 -0500 Subject: [PATCH 113/156] convert shortlog to hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The use of string_list here is quadratic, since we insert into a sorted string-list. This seems to give a 1-3% speed up of "git shortlog -s" in linux.git, with ~30k unique authors. It gives a 75% speedup on a more pathological case with ~200k authors. This will also allow us to use a real union instead of the gross void casting. Notes: - do not need stable qsort - initialization of string_list can happen earlier - do not need to NULL oneline lists Benchmark 1: ./git.old shortlog -ns HEAD Time (mean ± σ): 11.533 s ± 0.086 s [User: 11.399 s, System: 0.133 s] Range (min … max): 11.428 s … 11.687 s 10 runs Benchmark 2: ./git.new shortlog -ns HEAD Time (mean ± σ): 11.244 s ± 0.237 s [User: 11.112 s, System: 0.132 s] Range (min … max): 11.049 s … 11.641 s 10 runs Summary ./git.new shortlog -ns HEAD ran 1.03 ± 0.02 times faster than ./git.old shortlog -ns HEAD --- builtin/shortlog.c | 87 +++++++++++++++++++++++++++++++++++----------- shortlog.h | 4 ++- 2 files changed, 69 insertions(+), 22 deletions(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index 4453fe9b6ec761..c5dcee1f7f55a1 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -15,6 +15,7 @@ #include "parse-options.h" #include "trailer.h" #include "strmap.h" +#include "khash.h" static char const * const shortlog_usage[] = { N_("git shortlog [] [] [[--] ...]"), @@ -23,7 +24,7 @@ static char const * const shortlog_usage[] = { }; /* - * The util field of our string_list_items will contain one of two things: + * The util field of our shortlog_items will contain one of two things: * * - if --summary is not in use, it will point to a string list of the * oneline subjects assigned to this author @@ -35,15 +36,22 @@ static char const * const shortlog_usage[] = { */ #define UTIL_TO_INT(x) ((intptr_t)(x)->util) +struct shortlog_item { + const char *string; + void *util; +}; + +KHASH_INIT(shortlog, const char *, void *, 1, kh_str_hash_func, kh_str_hash_equal) + static int compare_by_counter(const void *a1, const void *a2) { - const struct string_list_item *i1 = a1, *i2 = a2; + const struct shortlog_item *i1 = a1, *i2 = a2; return UTIL_TO_INT(i2) - UTIL_TO_INT(i1); } static int compare_by_list(const void *a1, const void *a2) { - const struct string_list_item *i1 = a1, *i2 = a2; + const struct shortlog_item *i1 = a1, *i2 = a2; const struct string_list *l1 = i1->util, *l2 = i2->util; if (l1->nr < l2->nr) @@ -54,17 +62,36 @@ static int compare_by_list(const void *a1, const void *a2) return -1; } +static int compare_by_string(const void *a1, const void *a2) +{ + const struct shortlog_item *i1 = a1, *i2 = a2; + return strcmp(i1->string, i2->string); +} + static void insert_one_record(struct shortlog *log, const char *ident, const char *oneline) { - struct string_list_item *item; - - item = string_list_insert(&log->list, ident); + int hash_ret; + khiter_t pos; + + pos = kh_put_shortlog(log->entries, ident, &hash_ret); + if (hash_ret) { + kh_key(log->entries, pos) = xstrdup(ident); + if (log->summary) + kh_value(log->entries, pos) = NULL; + else { + struct string_list *onelines = + xmalloc(sizeof(struct string_list)); + string_list_init_dup(onelines); + kh_value(log->entries, pos) = onelines; + } + } - if (log->summary) - item->util = (void *)(UTIL_TO_INT(item) + 1); - else { + if (log->summary) { + intptr_t cur = (intptr_t)kh_value(log->entries, pos); + kh_value(log->entries, pos) = (void *)(cur + 1); + } else { char *buffer; struct strbuf subject = STRBUF_INIT; const char *eol; @@ -85,11 +112,7 @@ static void insert_one_record(struct shortlog *log, format_subject(&subject, oneline, " "); buffer = strbuf_detach(&subject, NULL); - if (!item->util) { - item->util = xmalloc(sizeof(struct string_list)); - string_list_init_dup(item->util); - } - string_list_append_nodup(item->util, buffer); + string_list_append_nodup(kh_value(log->entries, pos), buffer); } } @@ -359,7 +382,7 @@ void shortlog_init(struct shortlog *log) read_mailmap(the_repository, &log->mailmap); - string_list_init_dup(&log->list); + log->entries = kh_init_shortlog(); log->wrap = DEFAULT_WRAPLEN; log->in1 = DEFAULT_INDENT1; log->in2 = DEFAULT_INDENT2; @@ -495,12 +518,31 @@ void shortlog_output(struct shortlog *log) { size_t i, j; struct strbuf sb = STRBUF_INIT; + size_t sorted_nr = kh_size(log->entries); + struct shortlog_item *sorted; + const char *key; + void *value; + + ALLOC_ARRAY(sorted, sorted_nr); + i = 0; + kh_foreach(log->entries, key, value, { + if (i >= sorted_nr) + BUG("unexpected extra hash entries"); + sorted[i].string = key; + sorted[i].util = value; + i++; + }); + if (i != sorted_nr) + BUG("unexpected missing hashmap entries"); if (log->sort_by_number) - STABLE_QSORT(log->list.items, log->list.nr, + STABLE_QSORT(sorted, sorted_nr, log->summary ? compare_by_counter : compare_by_list); - for (i = 0; i < log->list.nr; i++) { - const struct string_list_item *item = &log->list.items[i]; + else + QSORT(sorted, sorted_nr, compare_by_string); + + for (i = 0; i < sorted_nr; i++) { + struct shortlog_item *item = &sorted[i]; if (log->summary) { fprintf(log->file, "%6d\t%s\n", (int)UTIL_TO_INT(item), item->string); @@ -523,12 +565,15 @@ void shortlog_output(struct shortlog *log) string_list_clear(onelines, 0); free(onelines); } - - log->list.items[i].util = NULL; } strbuf_release(&sb); - string_list_clear(&log->list, 1); + free(sorted); + kh_foreach(log->entries, key, value, { + free((void *)key); + }); + kh_destroy_shortlog(log->entries); + log->entries = NULL; clear_mailmap(&log->mailmap); string_list_clear(&log->format, 0); string_list_clear(&log->trailers, 0); diff --git a/shortlog.h b/shortlog.h index 28d04f951af73d..c7bec6495d6254 100644 --- a/shortlog.h +++ b/shortlog.h @@ -5,9 +5,11 @@ #include "date.h" struct commit; +struct kh_shortlog; struct shortlog { - struct string_list list; + struct kh_shortlog *entries; + int summary; int wrap_lines; int sort_by_number; From 31e9cb9523c6aa8b493750bff72516bfb711bf88 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 29 Jan 2026 05:36:25 -0500 Subject: [PATCH 114/156] convert to union The main goal here is better readability and less pointer-tweaking. But it also seems to speed things up a little (maybe 1% on linux, --- builtin/shortlog.c | 46 +++++++++++++++++++--------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index c5dcee1f7f55a1..dd46c4e109258c 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -24,35 +24,32 @@ static char const * const shortlog_usage[] = { }; /* - * The util field of our shortlog_items will contain one of two things: - * - * - if --summary is not in use, it will point to a string list of the - * oneline subjects assigned to this author - * - * - if --summary is in use, we don't need that list; we only need to know - * its size. So we abuse the pointer slot to store our integer counter. - * - * This macro accesses the latter. + * If --summary is in use, we only need to store a count for each group. + * Otherwise, we store the a list of the actual oneline subjects. */ -#define UTIL_TO_INT(x) ((intptr_t)(x)->util) +union shortlog_value { + int counter; + struct string_list onelines; +}; struct shortlog_item { const char *string; - void *util; + union shortlog_value value; }; -KHASH_INIT(shortlog, const char *, void *, 1, kh_str_hash_func, kh_str_hash_equal) +KHASH_INIT(shortlog, const char *, union shortlog_value, 1, kh_str_hash_func, kh_str_hash_equal) static int compare_by_counter(const void *a1, const void *a2) { const struct shortlog_item *i1 = a1, *i2 = a2; - return UTIL_TO_INT(i2) - UTIL_TO_INT(i1); + return i2->value.counter - i1->value.counter; } static int compare_by_list(const void *a1, const void *a2) { const struct shortlog_item *i1 = a1, *i2 = a2; - const struct string_list *l1 = i1->util, *l2 = i2->util; + const struct string_list *l1 = &i1->value.onelines, + *l2 = &i2->value.onelines; if (l1->nr < l2->nr) return 1; @@ -79,18 +76,14 @@ static void insert_one_record(struct shortlog *log, if (hash_ret) { kh_key(log->entries, pos) = xstrdup(ident); if (log->summary) - kh_value(log->entries, pos) = NULL; + kh_value(log->entries, pos).counter = 0; else { - struct string_list *onelines = - xmalloc(sizeof(struct string_list)); - string_list_init_dup(onelines); - kh_value(log->entries, pos) = onelines; + string_list_init_dup(&kh_value(log->entries, pos).onelines); } } if (log->summary) { - intptr_t cur = (intptr_t)kh_value(log->entries, pos); - kh_value(log->entries, pos) = (void *)(cur + 1); + kh_value(log->entries, pos).counter++; } else { char *buffer; struct strbuf subject = STRBUF_INIT; @@ -112,7 +105,7 @@ static void insert_one_record(struct shortlog *log, format_subject(&subject, oneline, " "); buffer = strbuf_detach(&subject, NULL); - string_list_append_nodup(kh_value(log->entries, pos), buffer); + string_list_append_nodup(&kh_value(log->entries, pos).onelines, buffer); } } @@ -521,7 +514,7 @@ void shortlog_output(struct shortlog *log) size_t sorted_nr = kh_size(log->entries); struct shortlog_item *sorted; const char *key; - void *value; + union shortlog_value value; ALLOC_ARRAY(sorted, sorted_nr); i = 0; @@ -529,7 +522,7 @@ void shortlog_output(struct shortlog *log) if (i >= sorted_nr) BUG("unexpected extra hash entries"); sorted[i].string = key; - sorted[i].util = value; + sorted[i].value = value; i++; }); if (i != sorted_nr) @@ -545,9 +538,9 @@ void shortlog_output(struct shortlog *log) struct shortlog_item *item = &sorted[i]; if (log->summary) { fprintf(log->file, "%6d\t%s\n", - (int)UTIL_TO_INT(item), item->string); + item->value.counter, item->string); } else { - struct string_list *onelines = item->util; + struct string_list *onelines = &item->value.onelines; fprintf(log->file, "%s (%"PRIuMAX"):\n", item->string, (uintmax_t)onelines->nr); for (j = onelines->nr; j >= 1; j--) { @@ -563,7 +556,6 @@ void shortlog_output(struct shortlog *log) } putc('\n', log->file); string_list_clear(onelines, 0); - free(onelines); } } From 723c1202a7812a766cdc70db785e3b1754e8711a Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 02:07:26 -0400 Subject: [PATCH 115/156] drop a funky strdup_strings-then-clear case This incidentally also fixes a bug where label.buf is examined after being detached (in the error message for duplicates). --- t/helper/test-parse-options.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/t/helper/test-parse-options.c b/t/helper/test-parse-options.c index f181f0c02d355a..8ae9c7f82cd229 100644 --- a/t/helper/test-parse-options.c +++ b/t/helper/test-parse-options.c @@ -69,10 +69,11 @@ static int collect_expect(const struct option *opt, const char *arg, int unset) if (!colon) die("malformed --expect option, lacking a colon"); strbuf_add(&label, arg, colon - arg); - item = string_list_insert(expect, strbuf_detach(&label, NULL)); + item = string_list_insert(expect, label.buf); if (item->util) die("malformed --expect option, duplicate %s", label.buf); item->util = (void *)arg; + strbuf_release(&label); return 0; } @@ -118,7 +119,7 @@ int cmd__parse_options(int argc, const char **argv) "A helper function for the parse-options API.", NULL }; - struct string_list expect = STRING_LIST_INIT_NODUP; + struct string_list expect = STRING_LIST_INIT_DUP; struct string_list list = STRING_LIST_INIT_NODUP; uint16_t u16 = 0; int16_t i16 = 0; @@ -246,7 +247,6 @@ int cmd__parse_options(int argc, const char **argv) for (int i = 0; i < argc; i++) show(&expect, &ret, "arg %02d: %s", i, argv[i]); - expect.strdup_strings = 1; string_list_clear(&expect, 0); string_list_clear(&list, 0); free(file); From 8f690a733830316dde0f22cfa81813e29928ba8e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 11 Mar 2026 19:07:53 -0400 Subject: [PATCH 116/156] notes: put string_list directly in struct The pointer here is not doing anything useful. This also switches to using the correct initializer. --- notes-utils.c | 17 +++++++---------- notes-utils.h | 2 +- 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/notes-utils.c b/notes-utils.c index 5c1c75d5b8099a..74a6cf10d0a539 100644 --- a/notes-utils.c +++ b/notes-utils.c @@ -123,7 +123,7 @@ static int notes_rewrite_config(const char *k, const char *v, /* note that a refs/ prefix is implied in the * underlying for_each_glob_ref */ if (starts_with(v, "refs/notes/")) - string_list_add_refs_by_glob(c->refs, v); + string_list_add_refs_by_glob(&c->refs, v); else warning(_("Refusing to rewrite notes in %s" " (outside of refs/notes/)"), v); @@ -142,8 +142,7 @@ struct notes_rewrite_cfg *init_copy_notes_for_rewrite(const char *cmd) c->cmd = cmd; c->enabled = 1; c->combine = combine_notes_concatenate; - CALLOC_ARRAY(c->refs, 1); - c->refs->strdup_strings = 1; + string_list_init_dup(&c->refs); c->refs_from_env = 0; c->mode_from_env = 0; if (rewrite_mode_env) { @@ -160,18 +159,16 @@ struct notes_rewrite_cfg *init_copy_notes_for_rewrite(const char *cmd) } if (rewrite_refs_env) { c->refs_from_env = 1; - string_list_add_refs_from_colon_sep(c->refs, rewrite_refs_env); + string_list_add_refs_from_colon_sep(&c->refs, rewrite_refs_env); } repo_config(the_repository, notes_rewrite_config, c); - if (!c->enabled || !c->refs->nr) { - string_list_clear(c->refs, 0); - free(c->refs); + if (!c->enabled || !c->refs.nr) { + string_list_clear(&c->refs, 0); free(c); return NULL; } - c->trees = load_notes_trees(c->refs, NOTES_INIT_WRITABLE); - string_list_clear(c->refs, 0); - free(c->refs); + c->trees = load_notes_trees(&c->refs, NOTES_INIT_WRITABLE); + string_list_clear(&c->refs, 0); return c; } diff --git a/notes-utils.h b/notes-utils.h index c54b1fe141f4bf..733d50c697b387 100644 --- a/notes-utils.h +++ b/notes-utils.h @@ -39,7 +39,7 @@ struct notes_rewrite_cfg { const char *cmd; int enabled; combine_notes_fn combine; - struct string_list *refs; + struct string_list refs; int refs_from_env; int mode_from_env; }; From c9946442be8544915a5c98b242f13eeab5ca1662 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 11 Mar 2026 19:10:58 -0400 Subject: [PATCH 117/156] a few more initializer uses The notes one is particularly egregious, and comes from 2721ce21e4 (use string_list initializer consistently, 2016-06-13) which just assumed initializations were correct. Note also calloc-vs-alloc; calloc not necessary since we are using the full initializer. --- fmt-merge-msg.c | 8 ++++---- notes.c | 3 +-- reflog-walk.c | 2 +- resolve-undo.c | 8 ++++---- wt-status.c | 6 +++--- 5 files changed, 13 insertions(+), 14 deletions(-) diff --git a/fmt-merge-msg.c b/fmt-merge-msg.c index 14441f23ae7b3b..f7cba997ae6bf3 100644 --- a/fmt-merge-msg.c +++ b/fmt-merge-msg.c @@ -64,10 +64,10 @@ struct origin_data { static void init_src_data(struct src_data *data) { - data->branch.strdup_strings = 1; - data->tag.strdup_strings = 1; - data->r_branch.strdup_strings = 1; - data->generic.strdup_strings = 1; + string_list_init_dup(&data->branch); + string_list_init_dup(&data->tag); + string_list_init_dup(&data->r_branch); + string_list_init_dup(&data->generic); } static struct string_list srcs = STRING_LIST_INIT_DUP; diff --git a/notes.c b/notes.c index ec9c2cb150d4e3..fd23707d6230e1 100644 --- a/notes.c +++ b/notes.c @@ -80,7 +80,7 @@ struct non_note { struct notes_tree default_notes_tree; -static struct string_list display_notes_refs = STRING_LIST_INIT_NODUP; +static struct string_list display_notes_refs = STRING_LIST_INIT_DUP; static struct notes_tree **display_notes_trees; static void load_subtree(struct notes_tree *t, struct leaf_node *subtree, @@ -1111,7 +1111,6 @@ void load_display_notes(struct display_notes_opt *opt) { char *display_ref_env; int load_config_refs = 0; - display_notes_refs.strdup_strings = 1; assert(!display_notes_trees); diff --git a/reflog-walk.c b/reflog-walk.c index 4dbeaa93a7703f..df99c43c51d68c 100644 --- a/reflog-walk.c +++ b/reflog-walk.c @@ -132,7 +132,7 @@ struct reflog_walk_info { void init_reflog_walk(struct reflog_walk_info **info) { CALLOC_ARRAY(*info, 1); - (*info)->complete_reflogs.strdup_strings = 1; + string_list_init_dup(&(*info)->complete_reflogs); } void reflog_walk_info_release(struct reflog_walk_info *info) diff --git a/resolve-undo.c b/resolve-undo.c index 52c45e5a494636..1e3cf73b94d288 100644 --- a/resolve-undo.c +++ b/resolve-undo.c @@ -20,8 +20,8 @@ void record_resolve_undo(struct index_state *istate, struct cache_entry *ce) return; if (!istate->resolve_undo) { - CALLOC_ARRAY(resolve_undo, 1); - resolve_undo->strdup_strings = 1; + ALLOC_ARRAY(resolve_undo, 1); + string_list_init_dup(resolve_undo); istate->resolve_undo = resolve_undo; } resolve_undo = istate->resolve_undo; @@ -64,8 +64,8 @@ struct string_list *resolve_undo_read(const char *data, unsigned long size, int i; const unsigned rawsz = algop->rawsz; - CALLOC_ARRAY(resolve_undo, 1); - resolve_undo->strdup_strings = 1; + ALLOC_ARRAY(resolve_undo, 1); + string_list_init_dup(resolve_undo); while (size) { struct string_list_item *lost; diff --git a/wt-status.c b/wt-status.c index 57772c7501fdba..0d354dfc404ee4 100644 --- a/wt-status.c +++ b/wt-status.c @@ -155,9 +155,9 @@ void wt_status_prepare(struct repository *r, struct wt_status *s) s->reference = "HEAD"; s->fp = stdout; s->index_file = repo_get_index_file(r); - s->change.strdup_strings = 1; - s->untracked.strdup_strings = 1; - s->ignored.strdup_strings = 1; + string_list_init_dup(&s->change); + string_list_init_dup(&s->untracked); + string_list_init_dup(&s->ignored); s->show_branch = -1; /* unspecified */ s->show_stash = 0; s->ahead_behind_flags = AHEAD_BEHIND_UNSPECIFIED; From c646ced4f855211bd266649e2159a96506087fb0 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 02:09:20 -0400 Subject: [PATCH 118/156] another initializer use This one is a little funky because it's a big array of string_lists. --- unpack-trees.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 154d6d40a15934..58de489d1e3863 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -204,7 +204,7 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts, opts->internal.show_all_errors = 1; /* rejected paths may not have a static buffer */ for (i = 0; i < ARRAY_SIZE(opts->internal.unpack_rejects); i++) - opts->internal.unpack_rejects[i].strdup_strings = 1; + string_list_init_dup(&opts->internal.unpack_rejects[i]); } void clear_unpack_trees_porcelain(struct unpack_trees_options *opts) From 252c36785ff4765fc2036aea98e0eba24f7d483d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 18:33:24 -0500 Subject: [PATCH 119/156] fill_textconv: always check DIFF_FILE_VALID() When we have an invalid diff_filespec (e.g., "/dev/null" for an addition or removal) that is not getting a textconv conversion, we "fill" the buffer by pointing to the empty string. But we don't do a similar check when we are doing a conversion. Instead we let prepare_temp_file() handle it, which just sets up /dev/null as a tempfile. This makes the function hard to follow, since it seems like we're not handling this case (and if we did, that it would be done inefficiently, running a textconv filter against /dev/null). In fact this code path is not triggered at all, since get_textconv() will not ever return a non-NULL value for an invalid filespec. But let's hoist the check for DIFF_FILE_VALID() to the top of the function to make it clear that we treat invalid files consistently. Signed-off-by: Jeff King --- diff.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/diff.c b/diff.c index 414532d09f5ba3..5966ed7aacfbba 100644 --- a/diff.c +++ b/diff.c @@ -7777,11 +7777,12 @@ size_t fill_textconv(struct repository *r, { size_t size; + if (!DIFF_FILE_VALID(df)) { + *outbuf = (char *)""; + return 0; + } + if (!driver) { - if (!DIFF_FILE_VALID(df)) { - *outbuf = (char *) ""; - return 0; - } if (diff_populate_filespec(r, df, NULL)) die("unable to read files to diff"); *outbuf = df->data; From 30d271500bbc90f84162384c59fd7a3086db08f1 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 01:47:52 -0400 Subject: [PATCH 120/156] mailmap: drop manual setting of strdup_strings I think this is redundant, because we initialize the namemap with strdup_strings when we allocate it in add_mapping(). --- mailmap.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mailmap.c b/mailmap.c index 1386ab208cc57c..f966f2f029fc24 100644 --- a/mailmap.c +++ b/mailmap.c @@ -36,7 +36,6 @@ static void free_mailmap_entry(void *p, const char *s UNUSED) free(me->name); free(me->email); - me->namemap.strdup_strings = 1; string_list_clear_func(&me->namemap, free_mailmap_info); free(me); } From 814cbed45e15b3e278c14f7268fbf1f0eeb1f5d6 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 17:47:04 -0500 Subject: [PATCH 121/156] userdiff: give textconv its own struct The interface for userdiff_get_textconv() is a little funny. It takes a "struct userdiff_driver" as input, and returns that same driver as output, but only if textconv is configured. This makes it easy to misuse, since you can just pass the initial userdiff_driver to any function which expected one to have been passed through the accessor function. The type system doesn't help us at all. The reason for this funny design is that the textconv data is spread across several struct members, and functions like fill_textconv() need to see all of them. Let's instead push all of those fields into a "struct userdiff_textconv" that's embedded inside the userdiff_driver. Then the get_textconv() accessor can return just the textconv struct, making it clear that the result came from the accessor function. Signed-off-by: Jeff King --- combine-diff.c | 6 +++--- diff.c | 36 ++++++++++++++++++------------------ diff.h | 13 ++++++------- diffcore-pickaxe.c | 4 ++-- grep.c | 8 ++++---- userdiff.c | 24 +++++++++++++----------- userdiff.h | 16 ++++++++++------ 7 files changed, 56 insertions(+), 51 deletions(-) diff --git a/combine-diff.c b/combine-diff.c index 717d5376888f8e..9bf91cff333a1f 100644 --- a/combine-diff.c +++ b/combine-diff.c @@ -304,7 +304,7 @@ static struct lline *coalesce_lines(struct lline *base, int *lenbase, static char *grab_blob(struct repository *r, const struct object_id *oid, unsigned int mode, - unsigned long *size, struct userdiff_driver *textconv, + unsigned long *size, struct userdiff_textconv *textconv, const char *path) { char *blob; @@ -422,7 +422,7 @@ static void combine_diff(struct repository *r, mmfile_t *result_file, struct sline *sline, unsigned int cnt, int n, int num_parent, int result_deleted, - struct userdiff_driver *textconv, + struct userdiff_textconv *textconv, const char *path, long flags) { unsigned int p_lno, lno; @@ -1023,7 +1023,7 @@ static void show_patch_diff(struct combine_diff_path *elem, int num_parent, int i, show_hunks; mmfile_t result_file; struct userdiff_driver *userdiff; - struct userdiff_driver *textconv = NULL; + struct userdiff_textconv *textconv = NULL; int is_binary; const char *line_prefix = diff_line_prefix(opt); diff --git a/diff.c b/diff.c index 5966ed7aacfbba..8a1c8e341ae06f 100644 --- a/diff.c +++ b/diff.c @@ -1875,8 +1875,8 @@ static void emit_rewrite_diff(const char *name_a, const char *name_b, struct diff_filespec *one, struct diff_filespec *two, - struct userdiff_driver *textconv_one, - struct userdiff_driver *textconv_two, + struct userdiff_textconv *textconv_one, + struct userdiff_textconv *textconv_two, struct diff_options *o) { int lc_a, lc_b; @@ -3738,8 +3738,8 @@ void diff_set_default_prefix(struct diff_options *options) options->b_prefix = diff_dst_prefix ? diff_dst_prefix : "b/"; } -struct userdiff_driver *get_textconv(struct repository *r, - struct diff_filespec *one) +struct userdiff_textconv *get_textconv(struct repository *r, + struct diff_filespec *one) { if (!DIFF_FILE_VALID(one)) return NULL; @@ -3832,8 +3832,8 @@ static void builtin_diff(const char *name_a, const char *meta = diff_get_color_opt(o, DIFF_METAINFO); const char *reset = diff_get_color_opt(o, DIFF_RESET); const char *a_prefix, *b_prefix; - struct userdiff_driver *textconv_one = NULL; - struct userdiff_driver *textconv_two = NULL; + struct userdiff_textconv *textconv_one = NULL; + struct userdiff_textconv *textconv_two = NULL; struct strbuf header = STRBUF_INIT; const char *line_prefix = diff_line_prefix(o); @@ -7771,7 +7771,7 @@ static char *run_textconv(struct repository *r, } size_t fill_textconv(struct repository *r, - struct userdiff_driver *driver, + struct userdiff_textconv *textconv, struct diff_filespec *df, char **outbuf) { @@ -7782,39 +7782,39 @@ size_t fill_textconv(struct repository *r, return 0; } - if (!driver) { + if (!textconv) { if (diff_populate_filespec(r, df, NULL)) die("unable to read files to diff"); *outbuf = df->data; return df->size; } - if (!driver->textconv) - BUG("fill_textconv called with non-textconv driver"); + if (!textconv->program) + BUG("fill_textconv called with empty textconv program"); - if (driver->textconv_cache && df->oid_valid) { - *outbuf = notes_cache_get(driver->textconv_cache, + if (textconv->cache && df->oid_valid) { + *outbuf = notes_cache_get(textconv->cache, &df->oid, &size); if (*outbuf) return size; } - *outbuf = run_textconv(r, driver->textconv, df, &size); + *outbuf = run_textconv(r, textconv->program, df, &size); if (!*outbuf) die("unable to read files to diff"); - if (driver->textconv_cache && df->oid_valid) { + if (textconv->cache && df->oid_valid) { /* ignore errors, as we might be in a readonly repository */ - notes_cache_put(driver->textconv_cache, &df->oid, *outbuf, - size); + notes_cache_put(textconv->cache, &df->oid, *outbuf, size); + /* * we could save up changes and flush them all at the end, * but we would need an extra call after all diffing is done. * Since generating a cache entry is the slow path anyway, * this extra overhead probably isn't a big deal. */ - notes_cache_write(driver->textconv_cache); + notes_cache_write(textconv->cache); } return size; @@ -7829,7 +7829,7 @@ int textconv_object(struct repository *r, unsigned long *buf_size) { struct diff_filespec *df; - struct userdiff_driver *textconv; + struct userdiff_textconv *textconv; df = alloc_filespec(path); fill_filespec(df, oid, oid_valid, mode); diff --git a/diff.h b/diff.h index bb5cddaf3499e9..5380e0748b65a7 100644 --- a/diff.h +++ b/diff.h @@ -74,6 +74,7 @@ struct option; struct repository; struct rev_info; struct userdiff_driver; +struct userdiff_textconv; typedef int (*pathchange_fn_t)(struct diff_options *options, struct combine_diff_path *path); @@ -725,18 +726,16 @@ void show_interdiff(const struct object_id *, const struct object_id *, int indent, struct diff_options *); /* - * Fill the contents of the filespec "df", respecting any textconv defined by - * its userdiff driver. The "driver" parameter must come from a - * previous call to get_textconv(), and therefore should either be NULL or have - * textconv enabled. + * Fill the contents of the filespec "df", respecting respecting the textconv + * driver (if non-NULL). * * Note that the memory ownership of the resulting buffer depends on whether - * the driver field is NULL. If it is, then the memory belongs to the filespec + * the textconv field is NULL. If it is, then the memory belongs to the filespec * struct. If it is non-NULL, then "outbuf" points to a newly allocated buffer * that should be freed by the caller. */ size_t fill_textconv(struct repository *r, - struct userdiff_driver *driver, + struct userdiff_textconv *textconv, struct diff_filespec *df, char **outbuf); @@ -745,7 +744,7 @@ size_t fill_textconv(struct repository *r, * and only if it has textconv enabled (otherwise return NULL). The result * can be passed to fill_textconv(). */ -struct userdiff_driver *get_textconv(struct repository *r, +struct userdiff_textconv *get_textconv(struct repository *r, struct diff_filespec *one); /* diff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c index b0915be86fc475..4ccd15506828f4 100644 --- a/diffcore-pickaxe.c +++ b/diffcore-pickaxe.c @@ -129,8 +129,8 @@ static int has_changes(mmfile_t *one, mmfile_t *two, static int pickaxe_match(struct diff_filepair *p, struct diff_options *o, regex_t *regexp, kwset_t kws, pickaxe_fn fn) { - struct userdiff_driver *textconv_one = NULL; - struct userdiff_driver *textconv_two = NULL; + struct userdiff_textconv *textconv_one = NULL; + struct userdiff_textconv *textconv_two = NULL; mmfile_t mf1, mf2; int ret; diff --git a/grep.c b/grep.c index 1d75d314211aa8..dcd5321d8a4db3 100644 --- a/grep.c +++ b/grep.c @@ -1497,14 +1497,14 @@ static int look_ahead(struct grep_opt *opt, } static int fill_textconv_grep(struct repository *r, - struct userdiff_driver *driver, + struct userdiff_textconv *textconv, struct grep_source *gs) { struct diff_filespec *df; char *buf; size_t size; - if (!driver || !driver->textconv) + if (!textconv) return grep_source_load(gs); /* @@ -1537,7 +1537,7 @@ static int fill_textconv_grep(struct repository *r, * non-worktreee git-grep with --textconv. */ obj_read_lock(); - size = fill_textconv(r, driver, df, &buf); + size = fill_textconv(r, textconv, df, &buf); obj_read_unlock(); free_filespec(df); @@ -1574,7 +1574,7 @@ static int grep_source_1(struct grep_opt *opt, struct grep_source *gs, int colle unsigned count = 0; int try_lookahead = 0; int show_function = 0; - struct userdiff_driver *textconv = NULL; + struct userdiff_textconv *textconv = NULL; enum grep_context ctx = GREP_CONTEXT_HEAD; xdemitconf_t xecfg; diff --git a/userdiff.c b/userdiff.c index 7129bf148266b7..34f04ff94cb73d 100644 --- a/userdiff.c +++ b/userdiff.c @@ -498,13 +498,13 @@ int userdiff_config(const char *k, const char *v) } if (!strcmp(type, "textconv")) { int ret; - FREE_AND_NULL(drv->textconv_owned); - ret = git_config_string(&drv->textconv_owned, k, v); - drv->textconv = drv->textconv_owned; + FREE_AND_NULL(drv->textconv.program_owned); + ret = git_config_string(&drv->textconv.program_owned, k, v); + drv->textconv.program = drv->textconv.program_owned; return ret; } if (!strcmp(type, "cachetextconv")) - return parse_bool(&drv->textconv_want_cache, k, v); + return parse_bool(&drv->textconv.want_cache, k, v); if (!strcmp(type, "wordregex")) { int ret; FREE_AND_NULL(drv->word_regex_owned); @@ -555,24 +555,26 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate, return userdiff_find_by_name(check->items[0].value); } -struct userdiff_driver *userdiff_get_textconv(struct repository *r, - struct userdiff_driver *driver) +struct userdiff_textconv *userdiff_get_textconv(struct repository *r, + struct userdiff_driver *driver) { - if (!driver->textconv) + struct userdiff_textconv *textconv = &driver->textconv; + + if (!textconv->program) return NULL; - if (driver->textconv_want_cache && !driver->textconv_cache && + if (textconv->want_cache && !textconv->cache && have_git_dir()) { struct notes_cache *c = xmalloc(sizeof(*c)); struct strbuf name = STRBUF_INIT; strbuf_addf(&name, "textconv/%s", driver->name); - notes_cache_init(r, c, name.buf, driver->textconv); - driver->textconv_cache = c; + notes_cache_init(r, c, name.buf, textconv->program); + textconv->cache = c; strbuf_release(&name); } - return driver; + return textconv; } static int for_each_userdiff_driver_list(each_userdiff_driver_fn fn, diff --git a/userdiff.h b/userdiff.h index 827361b0bc9569..5f4699e12f0ddd 100644 --- a/userdiff.h +++ b/userdiff.h @@ -17,6 +17,13 @@ struct external_diff { unsigned trust_exit_code:1; }; +struct userdiff_textconv { + const char *program; + char *program_owned; + struct notes_cache *cache; + int want_cache; +}; + struct userdiff_driver { const char *name; struct external_diff external; @@ -27,10 +34,7 @@ struct userdiff_driver { const char *word_regex; char *word_regex_owned; const char *word_regex_multi_byte; - const char *textconv; - char *textconv_owned; - struct notes_cache *textconv_cache; - int textconv_want_cache; + struct userdiff_textconv textconv; }; enum userdiff_driver_type { USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0, @@ -48,8 +52,8 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate, * Initialize any textconv-related fields in the driver and return it, or NULL * if it does not have textconv enabled at all. */ -struct userdiff_driver *userdiff_get_textconv(struct repository *r, - struct userdiff_driver *driver); +struct userdiff_textconv *userdiff_get_textconv(struct repository *r, + struct userdiff_driver *driver); /* * Iterate over all userdiff drivers. The userdiff_driver_type From ad1ebd9af3e34d44e221e9a0e91660ee6356295e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 01:48:44 -0400 Subject: [PATCH 122/156] mailmap: prefer initializer to manual strdup_string set --- mailmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mailmap.c b/mailmap.c index f966f2f029fc24..8fa57f05225e32 100644 --- a/mailmap.c +++ b/mailmap.c @@ -71,7 +71,7 @@ static void add_mapping(struct string_list *map, me = (struct mailmap_entry *)item->util; } else { CALLOC_ARRAY(me, 1); - me->namemap.strdup_strings = 1; + string_list_init_dup(&me->namemap); me->namemap.cmp = namemap_cmp; item->util = me; } From 1e1632b004e2660d7e2a7b8ab93880c30ecb28c2 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 18:23:05 -0500 Subject: [PATCH 123/156] diff: respect diff_options inside get_textconv() The get_textconv() function just returns the textconv config for a given filespec. It's up to the caller to decide if the textconv feature is actually enabled for this run. Let's instead move that check inside get_textconv(). It will save a tiny bit of effort now at each caller, but that will become more important as the logic gets more complicated in future patches. Note that there's one funny case we need to handle: textconv_object() abuses the diff_filespec system to convert arbitrary buffers. So its callers don't necessarily have a diff_options struct at all. As a convenience, we'll let callers of that function pass NULL to get a fallback diff_options that enables the feature. Since we're already touching each callsite, let's also rename get_textconv() to the more verbose diff_get_textconv(). That probably should have happened when it became globally available long ago. Signed-off-by: Jeff King --- blame.c | 12 ++++++------ builtin/cat-file.c | 4 ++-- builtin/log.c | 2 +- diff.c | 25 ++++++++++++++++++------- diff.h | 12 +++++++----- diffcore-pickaxe.c | 6 ++---- 6 files changed, 36 insertions(+), 25 deletions(-) diff --git a/blame.c b/blame.c index 126e2324162353..2368de7a705d02 100644 --- a/blame.c +++ b/blame.c @@ -254,8 +254,9 @@ static struct commit *fake_working_tree_commit(struct repository *r, switch (st.st_mode & S_IFMT) { case S_IFREG: - if (opt->flags.allow_textconv && - textconv_object(r, read_from, mode, null_oid(the_hash_algo), 0, &buf_ptr, &buf_len)) + if (textconv_object(r, opt, read_from, mode, + null_oid(the_hash_algo), 0, + &buf_ptr, &buf_len)) strbuf_attach(&buf, buf_ptr, buf_len, buf_len + 1); else if (strbuf_read_file(&buf, read_from, st.st_size) != st.st_size) die_errno("cannot open or read '%s'", read_from); @@ -1037,8 +1038,7 @@ static void fill_origin_blob(struct diff_options *opt, unsigned long file_size; (*num_read_blob)++; - if (opt->flags.allow_textconv && - textconv_object(opt->repo, o->path, o->mode, + if (textconv_object(opt->repo, opt, o->path, o->mode, &o->blob_oid, 1, &file->ptr, &file_size)) ; else { @@ -2868,8 +2868,8 @@ void setup_scoreboard(struct blame_scoreboard *sb, if (fill_blob_sha1_and_mode(sb->repo, o)) die(_("no such path %s in %s"), sb->path, final_commit_name); - if (sb->revs->diffopt.flags.allow_textconv && - textconv_object(sb->repo, sb->path, o->mode, &o->blob_oid, 1, (char **) &sb->final_buf, + if (textconv_object(sb->repo, &sb->revs->diffopt, + sb->path, o->mode, &o->blob_oid, 1, (char **) &sb->final_buf, &sb->final_buf_size)) ; else { diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 8870a210ec2e94..965973eabb8765 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -203,7 +203,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name) case 'c': { unsigned long size_ul = 0; - int textconv_ret = textconv_object(the_repository, path, + int textconv_ret = textconv_object(the_repository, NULL, path, obj_context.mode, &oid, 1, &buf, &size_ul); size = size_ul; @@ -458,7 +458,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d } else if (opt->transform_mode == 'c') { enum object_type type; unsigned long size_ul = 0; - if (textconv_object(the_repository, + if (textconv_object(the_repository, NULL, data->rest, 0100644, oid, 1, &contents, &size_ul)) size = size_ul; diff --git a/builtin/log.c b/builtin/log.c index 350b35c556362d..9d41509985cf0d 100644 --- a/builtin/log.c +++ b/builtin/log.c @@ -596,7 +596,7 @@ static int show_blob_object(const struct object_id *oid, struct rev_info *rev, c &oidc, &obj_context)) die(_("not a valid object name %s"), obj_name); if (!obj_context.path || - !textconv_object(the_repository, obj_context.path, + !textconv_object(the_repository, NULL, obj_context.path, obj_context.mode, &oidc, 1, &buf, &size)) { object_context_release(&obj_context); return odb_stream_blob_to_fd(the_repository->objects, 1, oid, NULL, 0); diff --git a/diff.c b/diff.c index 8a1c8e341ae06f..974b900802c78e 100644 --- a/diff.c +++ b/diff.c @@ -3738,9 +3738,13 @@ void diff_set_default_prefix(struct diff_options *options) options->b_prefix = diff_dst_prefix ? diff_dst_prefix : "b/"; } -struct userdiff_textconv *get_textconv(struct repository *r, - struct diff_filespec *one) +struct userdiff_textconv *diff_get_textconv(struct repository *r, + struct diff_options *opt, + struct diff_filespec *one) { + if (!opt->flags.allow_textconv) + return NULL; + if (!DIFF_FILE_VALID(one)) return NULL; @@ -3866,10 +3870,8 @@ static void builtin_diff(const char *name_a, return; } - if (o->flags.allow_textconv) { - textconv_one = get_textconv(o->repo, one); - textconv_two = get_textconv(o->repo, two); - } + textconv_one = diff_get_textconv(o->repo, o, one); + textconv_two = diff_get_textconv(o->repo, o, two); /* Never use a non-valid filename anywhere if at all possible */ name_a = DIFF_FILE_VALID(one) ? name_a : name_b; @@ -7821,6 +7823,7 @@ size_t fill_textconv(struct repository *r, } int textconv_object(struct repository *r, + struct diff_options *opt, const char *path, unsigned mode, const struct object_id *oid, @@ -7828,12 +7831,20 @@ int textconv_object(struct repository *r, char **buf, unsigned long *buf_size) { + struct diff_options fallback_opt; struct diff_filespec *df; struct userdiff_textconv *textconv; + if (!opt) { + repo_diff_setup(r, &fallback_opt); + fallback_opt.flags.allow_textconv = 1; + diff_setup_done(&fallback_opt); + opt = &fallback_opt; + } + df = alloc_filespec(path); fill_filespec(df, oid, oid_valid, mode); - textconv = get_textconv(r, df); + textconv = diff_get_textconv(r, opt, df); if (!textconv) { free_filespec(df); return 0; diff --git a/diff.h b/diff.h index 5380e0748b65a7..ad962560ebff5e 100644 --- a/diff.h +++ b/diff.h @@ -740,12 +740,13 @@ size_t fill_textconv(struct repository *r, char **outbuf); /* - * Look up the userdiff driver for the given filespec, and return it if - * and only if it has textconv enabled (otherwise return NULL). The result - * can be passed to fill_textconv(). + * Return the userdiff textconv driver for the given filespec if textconv is + * both enabled and configured for the filespec. Otherwise return NULL. + * The result can be passed to fill_textconv(). */ -struct userdiff_textconv *get_textconv(struct repository *r, - struct diff_filespec *one); +struct userdiff_textconv *diff_get_textconv(struct repository *r, + struct diff_options *opt, + struct diff_filespec *one); /* * Prepare diff_filespec and convert it using diff textconv API @@ -753,6 +754,7 @@ struct userdiff_textconv *get_textconv(struct repository *r, * Return 1 if the conversion succeeds, 0 otherwise. */ int textconv_object(struct repository *repo, + struct diff_options *opt, const char *path, unsigned mode, const struct object_id *oid, int oid_valid, diff --git a/diffcore-pickaxe.c b/diffcore-pickaxe.c index 4ccd15506828f4..c0ef8b1ff08637 100644 --- a/diffcore-pickaxe.c +++ b/diffcore-pickaxe.c @@ -145,10 +145,8 @@ static int pickaxe_match(struct diff_filepair *p, struct diff_options *o, oidset_contains(o->objfind, &p->two->oid)); } - if (o->flags.allow_textconv) { - textconv_one = get_textconv(o->repo, p->one); - textconv_two = get_textconv(o->repo, p->two); - } + textconv_one = diff_get_textconv(o->repo, o, p->one); + textconv_two = diff_get_textconv(o->repo, o, p->two); /* * If we have an unmodified pair, we know that the count will be the From abba287bd5da72a2785e7d736c559c1b5d964413 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 01:49:50 -0400 Subject: [PATCH 124/156] mailmap: another case of pointless strdup_strings setting --- mailmap.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mailmap.c b/mailmap.c index 8fa57f05225e32..235d935cb29ad1 100644 --- a/mailmap.c +++ b/mailmap.c @@ -236,7 +236,6 @@ int read_mailmap(struct repository *repo, struct string_list *map) void clear_mailmap(struct string_list *map) { - map->strdup_strings = 1; string_list_clear_func(map, free_mailmap_entry); } From 07e31507eac867783cc4646f5fc7efe978e30d84 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 18:55:01 -0500 Subject: [PATCH 125/156] diff: convert filespec is_binary flag to enum We have a tri-state is_binary flag that tells us whether the file is binary, text, or if we haven't figured it out yet. In preparation for determining (and caching) other characteristics of the file, let's switch this to an enum. There should be no functional changes here, and the enum values were chosen to be compatible with the original numeric values. Signed-off-by: Jeff King --- diff.c | 38 +++++++++++++++++++++++--------------- diffcore.h | 8 ++++++-- merge-ort.c | 2 +- 3 files changed, 30 insertions(+), 18 deletions(-) diff --git a/diff.c b/diff.c index 974b900802c78e..bc5e50bcd6d1bb 100644 --- a/diff.c +++ b/diff.c @@ -3688,28 +3688,36 @@ static void emit_binary_diff(struct diff_options *o, emit_binary_diff_body(o, two, one); } -int diff_filespec_is_binary(struct repository *r, - struct diff_filespec *one) +enum diff_content diff_filespec_content_type(struct repository *r, + struct diff_filespec *one) { struct diff_populate_filespec_options dpf_options = { .check_binary = 1, }; - if (one->is_binary == -1) { + if (one->content_type == DIFF_CONTENT_UNKNOWN) { diff_filespec_load_driver(one, r->index); if (one->driver->binary != -1) - one->is_binary = one->driver->binary; + one->content_type = one->driver->binary; else { if (!one->data && DIFF_FILE_VALID(one)) diff_populate_filespec(r, one, &dpf_options); - if (one->is_binary == -1 && one->data) - one->is_binary = buffer_is_binary(one->data, - one->size); - if (one->is_binary == -1) - one->is_binary = 0; + if (one->content_type == DIFF_CONTENT_UNKNOWN && one->data) { + if (!buffer_is_binary(one->data, one->size)) + one->content_type = DIFF_CONTENT_TEXT; + else + one->content_type = DIFF_CONTENT_BINARY; + } + if (one->content_type == DIFF_CONTENT_UNKNOWN) + one->content_type = DIFF_CONTENT_TEXT; } } - return one->is_binary; + return one->content_type; +} + +int diff_filespec_is_binary(struct repository *r, struct diff_filespec *one) +{ + return diff_filespec_content_type(r, one) != DIFF_CONTENT_TEXT; } static const struct userdiff_funcname * @@ -4339,7 +4347,7 @@ struct diff_filespec *alloc_filespec(const char *path) FLEXPTR_ALLOC_STR(spec, path, path); spec->count = 1; - spec->is_binary = -1; + spec->content_type = DIFF_CONTENT_UNKNOWN; return spec; } @@ -4552,8 +4560,8 @@ int diff_populate_filespec(struct repository *r, */ if (check_binary && s->size > repo_settings_get_big_file_threshold(the_repository) && - s->is_binary == -1) { - s->is_binary = 1; + s->content_type == DIFF_CONTENT_UNKNOWN) { + s->content_type = DIFF_CONTENT_BINARY; return 0; } fd = open(s->path, O_RDONLY); @@ -4605,8 +4613,8 @@ int diff_populate_filespec(struct repository *r, if (size_only) return 0; if (s->size > repo_settings_get_big_file_threshold(the_repository) && - s->is_binary == -1) { - s->is_binary = 1; + s->content_type == DIFF_CONTENT_UNKNOWN) { + s->content_type = DIFF_CONTENT_BINARY; return 0; } } diff --git a/diffcore.h b/diffcore.h index d75038d1b3a1b2..2b105da934b5c9 100644 --- a/diffcore.h +++ b/diffcore.h @@ -70,8 +70,11 @@ struct diff_filespec { #define DIRTY_SUBMODULE_MODIFIED 2 unsigned is_stdin : 1; unsigned has_more_entries : 1; /* only appear in combined diff */ - /* data should be considered "binary"; -1 means "don't know yet" */ - signed int is_binary : 2; + enum diff_content { + DIFF_CONTENT_UNKNOWN = -1, + DIFF_CONTENT_TEXT = 0, + DIFF_CONTENT_BINARY = 1 + } content_type; struct userdiff_driver *driver; }; @@ -102,6 +105,7 @@ int diff_populate_filespec(struct repository *, struct diff_filespec *, void diff_free_filespec_data(struct diff_filespec *); void diff_free_filespec_blob(struct diff_filespec *); int diff_filespec_is_binary(struct repository *, struct diff_filespec *); +enum diff_content diff_filespec_content_type(struct repository *, struct diff_filespec *); /** * This records a pair of `struct diff_filespec`; the filespec for a file in diff --git a/merge-ort.c b/merge-ort.c index c410a5d353234c..cdd866c9569f8f 100644 --- a/merge-ort.c +++ b/merge-ort.c @@ -890,7 +890,7 @@ static struct diff_filespec *pool_alloc_filespec(struct mem_pool *pool, spec->path = (char*)path; /* spec won't modify it */ spec->count = 1; - spec->is_binary = -1; + spec->content_type = DIFF_CONTENT_UNKNOWN; return spec; } From dcce1b8fef835e06e218c63a145507e14b810993 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 16 Jun 2015 14:03:02 -0400 Subject: [PATCH 126/156] trace: implement %p placeholder for filenames This lets you trace output on a per-process basis by naming the trace file after the PID. This is probably not all that useful for GIT_TRACE itself (which is about a global view of the process hierarchy anyway), but will be useful for traces which produce large amounts of data (e.g., whole packfiles). Signed-off-by: Jeff King --- Documentation/git.adoc | 3 ++- trace.c | 17 +++++++++++++++-- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Documentation/git.adoc b/Documentation/git.adoc index 8a5cdd3b3d22c5..372e1606c7d4ef 100644 --- a/Documentation/git.adoc +++ b/Documentation/git.adoc @@ -804,7 +804,8 @@ trace messages into this file descriptor. Alternatively, if the variable is set to an absolute path (starting with a '/' character), Git will interpret this as a file path and will try to append the trace messages -to it. +to it. If the filename contains the string `%p`, that string +will be replaced with the PID of the traced process. + Unsetting the variable, or setting it to empty, "0" or "false" (case insensitive) disables trace messages. diff --git a/trace.c b/trace.c index 515b99e7f5a105..92a257fb67edb7 100644 --- a/trace.c +++ b/trace.c @@ -53,15 +53,28 @@ static int get_trace_fd(struct trace_key *key, const char *override_envvar) else if (strlen(trace) == 1 && isdigit(*trace)) key->fd = atoi(trace); else if (is_absolute_path(trace)) { - int fd = open(trace, O_WRONLY | O_APPEND | O_CREAT, 0666); + struct strbuf name = STRBUF_INIT; + int fd; + + while (strbuf_expand_step(&name, &trace)) { + if (skip_prefix(trace, "%", &trace)) + strbuf_addch(&name, '%'); + else if (skip_prefix(trace, "p", &trace)) + strbuf_addf(&name, "%lu", (unsigned long)getpid()); + else + strbuf_addch(&name, '%'); + } + + fd = open(name.buf, O_WRONLY | O_APPEND | O_CREAT, 0666); if (fd == -1) { warning("could not open '%s' for tracing: %s", - trace, strerror(errno)); + name.buf, strerror(errno)); trace_disable(key); } else { key->fd = fd; key->need_close = 1; } + strbuf_release(&name); } else { warning("unknown trace value for '%s': %s\n" " If you want to trace into a file, then please set %s\n" From 747a876d4001a40396b249b1d61db4cb2ca63772 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 01:54:01 -0400 Subject: [PATCH 127/156] mailmap: another case of raw initialization This one is weird. We are initializing with "dup", even though every caller has initialized already with nodup! Probably it is a mistake to expose a raw string_list as the mailmap element, and it should be inside a "struct mailmap" or similar. --- mailmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mailmap.c b/mailmap.c index 235d935cb29ad1..af2a219bb527ec 100644 --- a/mailmap.c +++ b/mailmap.c @@ -213,7 +213,7 @@ int read_mailmap(struct repository *repo, struct string_list *map) repo_config_get_pathname(repo, "mailmap.file", &mailmap_file); repo_config_get_string(repo, "mailmap.blob", &mailmap_blob); - map->strdup_strings = 1; + string_list_init_dup(map); map->cmp = namemap_cmp; if (!mailmap_blob && is_bare_repository(repo)) From 37542c0d1196f045f51265aaca4ce94402c797b5 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 19:07:27 -0500 Subject: [PATCH 128/156] diff: add utf-{16,32} binary detection When we examine a diff_filespec for binary-ness, we can now also determine whether it's utf-16 or utf-32 by looking for a BOM. Those encodings are still considered binary, but future patches will add callers which handle them specially. Note that even though we know the endian-ness of the encoding from the BOM, we do not include it in the returned name. This is because "iconv" (which we'll ultimately feed these names to) does not expect to see a BOM if we tell it the source is "UTF-16LE". We should feed it "UTF-16" with the BOM, which it will interpret itself. Signed-off-by: Jeff King --- diff.c | 23 +++++++++++++++++++++++ diffcore.h | 3 ++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/diff.c b/diff.c index bc5e50bcd6d1bb..f5a5d634b562db 100644 --- a/diff.c +++ b/diff.c @@ -3688,6 +3688,27 @@ static void emit_binary_diff(struct diff_options *o, emit_binary_diff_body(o, two, one); } +static const char *buffer_has_utf_bom(const void *vdata, size_t size) +{ + const unsigned char *data = vdata; + + if (size >= 4) { + if (data[0] == 0x00 && data[1] == 0x00 && + data[2] == 0xfe && data[3] == 0xff) + return "UTF-32"; + if (data[0] == 0xff && data[1] == 0xfe && + data[2] == 0x00 && data[3] == 0x00) + return "UTF-32"; + } + if (size >= 2) { + if (data[0] == 0xfe && data[1] == 0xff) + return "UTF-16"; + if (data[0] == 0xff && data[1] == 0xfe) + return "UTF-16"; + } + return NULL; +} + enum diff_content diff_filespec_content_type(struct repository *r, struct diff_filespec *one) { @@ -3705,6 +3726,8 @@ enum diff_content diff_filespec_content_type(struct repository *r, if (one->content_type == DIFF_CONTENT_UNKNOWN && one->data) { if (!buffer_is_binary(one->data, one->size)) one->content_type = DIFF_CONTENT_TEXT; + else if (buffer_has_utf_bom(one->data, one->size)) + one->content_type = DIFF_CONTENT_UTF; else one->content_type = DIFF_CONTENT_BINARY; } diff --git a/diffcore.h b/diffcore.h index 2b105da934b5c9..3b5620bafd5cd5 100644 --- a/diffcore.h +++ b/diffcore.h @@ -73,7 +73,8 @@ struct diff_filespec { enum diff_content { DIFF_CONTENT_UNKNOWN = -1, DIFF_CONTENT_TEXT = 0, - DIFF_CONTENT_BINARY = 1 + DIFF_CONTENT_BINARY = 1, + DIFF_CONTENT_UTF } content_type; struct userdiff_driver *driver; }; From 677270560de9ebefb95ad275f4e134666a4da22d Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 16 Jun 2015 14:13:01 -0400 Subject: [PATCH 129/156] trace: add pid to each output line When there are many git commands running, it can be confusing which process is producing which message. And if one is trying to correlate a specific invocation with a trace-file created using "%p", it's rather impossible. Let's print the pid alongside each line. This does extend the already-wide trace lines. We could make it optional, but it's probably not worth the complexity of extra knobs. Looking at trace output is already an exceptional thing to be doing, and verbose output is not going to bother anyone. Signed-off-by: Jeff King --- trace.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/trace.c b/trace.c index 92a257fb67edb7..3f33870353b197 100644 --- a/trace.c +++ b/trace.c @@ -129,8 +129,9 @@ static int prepare_trace_line(const char *file, int line, localtime_r(&secs, &tm); strbuf_addf(buf, "%02d:%02d:%02d.%06ld %s:%d", tm.tm_hour, tm.tm_min, tm.tm_sec, (long) tv.tv_usec, file, line); - /* align trace output (column 40 catches most files names in git) */ - while (buf->len < 40) + strbuf_addf(buf, "[pid=%lu] ", (unsigned long)getpid()); + /* align trace output (column 50 catches most files names in git) */ + while (buf->len < 50) strbuf_addch(buf, ' '); return 1; From 148f8459c6075996582429063d9789b1ca238726 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 21 May 2026 03:22:39 -0400 Subject: [PATCH 130/156] stuff mailmap into struct --- builtin/blame.c | 2 +- builtin/cat-file.c | 6 ++++-- builtin/check-mailmap.c | 4 ++-- builtin/commit.c | 4 ++-- builtin/log.c | 4 ++-- ident.c | 4 ++-- ident.h | 4 ++-- mailmap.c | 32 +++++++++++++++++++------------- mailmap.h | 21 +++++++++++++++------ pretty.c | 7 ++++--- pretty.h | 2 +- ref-filter.c | 4 ++-- revision.c | 2 +- revision.h | 2 +- shortlog.h | 3 ++- 15 files changed, 60 insertions(+), 41 deletions(-) diff --git a/builtin/blame.c b/builtin/blame.c index 48d5251c6df700..77d3d0f69571c8 100644 --- a/builtin/blame.c +++ b/builtin/blame.c @@ -74,7 +74,7 @@ static int mark_ignored_lines; static struct date_mode blame_date_mode = { DATE_ISO8601 }; static size_t blame_date_width; -static struct string_list mailmap = STRING_LIST_INIT_NODUP; +static struct mailmap mailmap = MAILMAP_INIT; #ifndef DEBUG_BLAME #define DEBUG_BLAME 0 diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 8870a210ec2e94..b19c985fc8c013 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -68,7 +68,8 @@ struct batch_options { static const char *force_path; -static struct string_list mailmap = STRING_LIST_INIT_NODUP; +static struct mailmap mailmap = MAILMAP_INIT; +static int loaded_mailmap; static int use_mailmap; static char *replace_idents_using_mailmap(char *, size_t *); @@ -81,10 +82,11 @@ static char *replace_idents_using_mailmap(char *, size_t *); */ static void load_mailmap(void) { - if (mailmap.strdup_strings) + if (loaded_mailmap) return; read_mailmap(the_repository, &mailmap); + loaded_mailmap = 1; } static char *replace_idents_using_mailmap(char *object_buf, size_t *size) diff --git a/builtin/check-mailmap.c b/builtin/check-mailmap.c index 3f2a39cae0c1d0..469c25a174098f 100644 --- a/builtin/check-mailmap.c +++ b/builtin/check-mailmap.c @@ -24,7 +24,7 @@ static const struct option check_mailmap_options[] = { OPT_END() }; -static void check_mailmap(struct string_list *mailmap, const char *contact) +static void check_mailmap(struct mailmap *mailmap, const char *contact) { const char *name, *mail; size_t namelen, maillen; @@ -55,7 +55,7 @@ int cmd_check_mailmap(int argc, struct repository *repo UNUSED) { int i; - struct string_list mailmap = STRING_LIST_INIT_NODUP; + struct mailmap mailmap = MAILMAP_INIT; repo_config(the_repository, git_default_config, NULL); argc = parse_options(argc, argv, prefix, check_mailmap_options, diff --git a/builtin/commit.c b/builtin/commit.c index 28f61745034506..4c95540f4806dc 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1153,8 +1153,8 @@ static const char *find_author_by_nickname(const char *name) av[++ac] = buf.buf; av[++ac] = NULL; setup_revisions(ac, av, &revs, NULL); - revs.mailmap = xmalloc(sizeof(struct string_list)); - string_list_init_nodup(revs.mailmap); + revs.mailmap = xmalloc(sizeof(*revs.mailmap)); + mailmap_init(revs.mailmap); read_mailmap(the_repository, revs.mailmap); if (prepare_revision_walk(&revs)) diff --git a/builtin/log.c b/builtin/log.c index 350b35c556362d..607d1383110fde 100644 --- a/builtin/log.c +++ b/builtin/log.c @@ -340,8 +340,8 @@ static void cmd_log_init_finish(int argc, const char **argv, const char *prefix, } if (mailmap) { - rev->mailmap = xmalloc(sizeof(struct string_list)); - string_list_init_nodup(rev->mailmap); + rev->mailmap = xmalloc(sizeof(*rev->mailmap)); + mailmap_init(rev->mailmap); read_mailmap(the_repository, rev->mailmap); } diff --git a/ident.c b/ident.c index 0b7aacecd7d9af..d5c4fefccc0138 100644 --- a/ident.c +++ b/ident.c @@ -354,7 +354,7 @@ int split_ident_line(struct ident_split *split, const char *line, size_t len) */ static ssize_t rewrite_ident_line(const char *person, size_t len, struct strbuf *buf, - struct string_list *mailmap) + struct mailmap *mailmap) { size_t namelen, maillen; const char *name; @@ -390,7 +390,7 @@ static ssize_t rewrite_ident_line(const char *person, size_t len, } void apply_mailmap_to_header(struct strbuf *buf, const char **header, - struct string_list *mailmap) + struct mailmap *mailmap) { size_t buf_offset = 0; diff --git a/ident.h b/ident.h index 3c034038791b78..92c8f344faadf2 100644 --- a/ident.h +++ b/ident.h @@ -1,7 +1,7 @@ #ifndef IDENT_H #define IDENT_H -#include "string-list.h" +#include "mailmap.h" struct ident_split { const char *name_begin; @@ -41,7 +41,7 @@ int split_ident_line(struct ident_split *, const char *, size_t); * Given a commit or tag object buffer and the commit or tag headers, replaces * the idents in the headers with their canonical versions using the mailmap mechanism. */ -void apply_mailmap_to_header(struct strbuf *, const char **, struct string_list *); +void apply_mailmap_to_header(struct strbuf *, const char **, struct mailmap *); /* * Compare split idents for equality or strict ordering. Note that we diff --git a/mailmap.c b/mailmap.c index af2a219bb527ec..726e292b3f883a 100644 --- a/mailmap.c +++ b/mailmap.c @@ -7,6 +7,12 @@ #include "setup.h" #include "config.h" +void mailmap_init(struct mailmap *map) +{ + struct mailmap blank = MAILMAP_INIT; + memcpy(map, &blank, sizeof(*map)); +} + struct mailmap_info { char *name; char *email; @@ -54,7 +60,7 @@ static int namemap_cmp(const char *a, const char *b) return strcasecmp(a, b); } -static void add_mapping(struct string_list *map, +static void add_mapping(struct mailmap *map, char *new_name, char *new_email, char *old_name, char *old_email) { @@ -66,7 +72,7 @@ static void add_mapping(struct string_list *map, new_email = NULL; } - item = string_list_insert(map, old_email); + item = string_list_insert(&map->map, old_email); if (item->util) { me = (struct mailmap_entry *)item->util; } else { @@ -123,7 +129,7 @@ static char *parse_name_and_email(char *buffer, char **name, return (*right == '\0' ? NULL : right); } -static void read_mailmap_line(struct string_list *map, char *buffer) +static void read_mailmap_line(struct mailmap *map, char *buffer) { char *name1 = NULL, *email1 = NULL, *name2 = NULL, *email2 = NULL; @@ -137,7 +143,7 @@ static void read_mailmap_line(struct string_list *map, char *buffer) add_mapping(map, name1, email1, name2, email2); } -int read_mailmap_file(struct string_list *map, const char *filename, +int read_mailmap_file(struct mailmap *map, const char *filename, unsigned flags) { char buffer[1024]; @@ -165,7 +171,7 @@ int read_mailmap_file(struct string_list *map, const char *filename, return 0; } -static void read_mailmap_string(struct string_list *map, char *buf) +static void read_mailmap_string(struct mailmap *map, char *buf) { while (*buf) { char *end = strchrnul(buf, '\n'); @@ -178,7 +184,7 @@ static void read_mailmap_string(struct string_list *map, char *buf) } } -int read_mailmap_blob(struct repository *repo, struct string_list *map, +int read_mailmap_blob(struct repository *repo, struct mailmap *map, const char *name) { struct object_id oid; @@ -205,7 +211,7 @@ int read_mailmap_blob(struct repository *repo, struct string_list *map, return 0; } -int read_mailmap(struct repository *repo, struct string_list *map) +int read_mailmap(struct repository *repo, struct mailmap *map) { int err = 0; char *mailmap_file = NULL, *mailmap_blob = NULL; @@ -213,8 +219,8 @@ int read_mailmap(struct repository *repo, struct string_list *map) repo_config_get_pathname(repo, "mailmap.file", &mailmap_file); repo_config_get_string(repo, "mailmap.blob", &mailmap_blob); - string_list_init_dup(map); - map->cmp = namemap_cmp; + string_list_init_dup(&map->map); + map->map.cmp = namemap_cmp; if (!mailmap_blob && is_bare_repository(repo)) mailmap_blob = xstrdup("HEAD:.mailmap"); @@ -234,9 +240,9 @@ int read_mailmap(struct repository *repo, struct string_list *map) return err; } -void clear_mailmap(struct string_list *map) +void clear_mailmap(struct mailmap *map) { - string_list_clear_func(map, free_mailmap_entry); + string_list_clear_func(&map->map, free_mailmap_entry); } /* @@ -289,14 +295,14 @@ static struct string_list_item *lookup_prefix(struct string_list *map, return NULL; } -int map_user(struct string_list *map, +int map_user(struct mailmap *map, const char **email, size_t *emaillen, const char **name, size_t *namelen) { struct string_list_item *item; struct mailmap_entry *me; - item = lookup_prefix(map, *email, *emaillen); + item = lookup_prefix(&map->map, *email, *emaillen); if (item) { me = (struct mailmap_entry *)item->util; if (me->namemap.nr) { diff --git a/mailmap.h b/mailmap.h index 6866cb6f1d6c6f..0942f6810fd71c 100644 --- a/mailmap.h +++ b/mailmap.h @@ -1,21 +1,30 @@ #ifndef MAILMAP_H #define MAILMAP_H +#include "string-list.h" + struct repository; -struct string_list; + +struct mailmap { + struct string_list map; +}; + +#define MAILMAP_INIT { STRING_LIST_INIT_DUP } + +void mailmap_init(struct mailmap *map); /* Flags for read_mailmap_file() */ #define MAILMAP_NOFOLLOW (1<<0) -int read_mailmap_file(struct string_list *map, const char *filename, +int read_mailmap_file(struct mailmap *map, const char *filename, unsigned flags); -int read_mailmap_blob(struct repository *repo, struct string_list *map, +int read_mailmap_blob(struct repository *repo, struct mailmap *map, const char *name); -int read_mailmap(struct repository *repo, struct string_list *map); -void clear_mailmap(struct string_list *map); +int read_mailmap(struct repository *repo, struct mailmap *map); +void clear_mailmap(struct mailmap *map); -int map_user(struct string_list *map, +int map_user(struct mailmap *map, const char **email, size_t *emaillen, const char **name, size_t *namelen); #endif diff --git a/pretty.c b/pretty.c index d8a9f370f6c2f5..602afadb80f790 100644 --- a/pretty.c +++ b/pretty.c @@ -777,12 +777,13 @@ const char *repo_logmsg_reencode(struct repository *r, static int mailmap_name(const char **email, size_t *email_len, const char **name, size_t *name_len) { - static struct string_list *mail_map; + static struct mailmap *mail_map; if (!mail_map) { - CALLOC_ARRAY(mail_map, 1); + ALLOC_ARRAY(mail_map, 1); + mailmap_init(mail_map); read_mailmap(the_repository, mail_map); } - return mail_map->nr && map_user(mail_map, email, email_len, name, name_len); + return mail_map->map.nr && map_user(mail_map, email, email_len, name, name_len); } static size_t format_person_part(struct strbuf *sb, char part, diff --git a/pretty.h b/pretty.h index fac699033e943c..d798fa073b2af5 100644 --- a/pretty.h +++ b/pretty.h @@ -46,7 +46,7 @@ struct pretty_print_context { struct reflog_walk_info *reflog_info; struct rev_info *rev; const char *output_encoding; - struct string_list *mailmap; + struct mailmap *mailmap; enum git_colorbool color; struct ident_split *from_ident; unsigned encode_email_headers:1; diff --git a/ref-filter.c b/ref-filter.c index bdf54f6f592499..360a1f915c4cb6 100644 --- a/ref-filter.c +++ b/ref-filter.c @@ -1719,7 +1719,7 @@ static void grab_date(const char *buf, struct atom_value *v, const char *atomnam v->value = 0; } -static struct string_list mailmap = STRING_LIST_INIT_NODUP; +static struct mailmap mailmap = MAILMAP_INIT; /* See grab_values */ static void grab_person(const char *who, struct atom_value *val, int deref, void *buf) @@ -1752,7 +1752,7 @@ static void grab_person(const char *who, struct atom_value *val, int deref, void (atom->u.name_option.option == N_MAILMAP)) || (starts_with(name + wholen, "email") && (atom->u.email_option.option & EO_MAILMAP))) { - if (!mailmap.items) + if (!mailmap.map.items) read_mailmap(the_repository, &mailmap); strbuf_addstr(&mailmap_buf, buf); apply_mailmap_to_header(&mailmap_buf, headers, &mailmap); diff --git a/revision.c b/revision.c index 35afe52208e710..a96b7fcb7f4c60 100644 --- a/revision.c +++ b/revision.c @@ -3277,7 +3277,7 @@ static void release_revisions_cmdline(struct rev_cmdline_info *cmdline) free(cmdline->rev); } -static void release_revisions_mailmap(struct string_list *mailmap) +static void release_revisions_mailmap(struct mailmap *mailmap) { if (!mailmap) return; diff --git a/revision.h b/revision.h index acf6d06b24126c..be563b64197960 100644 --- a/revision.h +++ b/revision.h @@ -306,7 +306,7 @@ struct rev_info { int patch_name_max; int no_inline; int show_log_size; - struct string_list *mailmap; + struct mailmap *mailmap; /* Filter by commit log message */ struct grep_opt grep_filter; diff --git a/shortlog.h b/shortlog.h index c7bec6495d6254..37655468ad8197 100644 --- a/shortlog.h +++ b/shortlog.h @@ -3,6 +3,7 @@ #include "string-list.h" #include "date.h" +#include "mailmap.h" struct commit; struct kh_shortlog; @@ -30,7 +31,7 @@ struct shortlog { struct string_list format; int email; - struct string_list mailmap; + struct mailmap mailmap; FILE *file; }; From c84bb7a68f188b74665f336731e5a8ebdda05cd7 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 1 Sep 2026 20:09:38 -0400 Subject: [PATCH 131/156] repository: make repo_clear() idempotent Calling repo_clear() twice in a row will segfault because the second call will invoke parse_object_pool_clear() on a NULL pointer. This is not usually a big deal, but we can make some error cleanup a little simpler if callers do not need to worry about invoking it twice. We can fix it by catching the NULL case. The rest of repo_clear() appears to be idempotent. --- repository.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/repository.c b/repository.c index 2d66cfdbd55994..b857e1c580e83e 100644 --- a/repository.c +++ b/repository.c @@ -386,7 +386,8 @@ void repo_clear(struct repository *repo) odb_free(repo->objects); repo->objects = NULL; - parsed_object_pool_clear(repo->parsed_objects); + if (repo->parsed_objects) + parsed_object_pool_clear(repo->parsed_objects); FREE_AND_NULL(repo->parsed_objects); repo_settings_clear(repo); From 79a09a2865862446057714b54dfa5f42919b411e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 27 Feb 2018 20:31:06 -0500 Subject: [PATCH 132/156] diff: add autoencode feature This feature auto-converts utf-16 and utf-32 content into utf-8 for diffing, as long as: 1. textconv is enabled 2. the autoencode feature is enabled (which defaults to off for now) 3. the file does not have a configured textconv filter TODO: - documentation - tests - address the fixme Signed-off-by: Jeff King --- diff.c | 43 ++++++++++++++++++++++++++++++++++++++++++- diff.h | 1 + 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/diff.c b/diff.c index f5a5d634b562db..86515d6a06edfd 100644 --- a/diff.c +++ b/diff.c @@ -78,6 +78,8 @@ static struct diff_options default_diff_options; static long diff_algorithm; static unsigned ws_error_highlight_default = WSEH_NEW; +static struct userdiff_textconv autoencode_textconv = { "autoencode" }; + static char diff_colors[][COLOR_MAXLEN] = { GIT_COLOR_RESET, GIT_COLOR_NORMAL, /* CONTEXT */ @@ -3773,6 +3775,8 @@ struct userdiff_textconv *diff_get_textconv(struct repository *r, struct diff_options *opt, struct diff_filespec *one) { + struct userdiff_textconv *textconv; + if (!opt->flags.allow_textconv) return NULL; @@ -3780,7 +3784,13 @@ struct userdiff_textconv *diff_get_textconv(struct repository *r, return NULL; diff_filespec_load_driver(one, r->index); - return userdiff_get_textconv(r, one->driver); + textconv = userdiff_get_textconv(r, one->driver); + + if (!textconv && opt->flags.allow_autoencode && + diff_filespec_content_type(r, one) == DIFF_CONTENT_UTF) + textconv = &autoencode_textconv; + + return textconv; } static struct string_list *additional_headers(struct diff_options *o, @@ -6274,6 +6284,8 @@ struct option *add_diff_options(const struct option *opts, OPT_CALLBACK_F(0, "textconv", options, NULL, N_("run external text conversion filters when comparing binary files"), PARSE_OPT_NOARG, diff_opt_textconv), + OPT_BOOL(0, "autoencode", &options->flags.allow_autoencode, + N_("allow automatic encoding conversion")), OPT_CALLBACK_F(0, "ignore-submodules", options, N_(""), N_("ignore changes to submodules in the diff generation"), PARSE_OPT_NONEG | PARSE_OPT_OPTARG, @@ -7815,6 +7827,35 @@ size_t fill_textconv(struct repository *r, return 0; } + if (textconv == &autoencode_textconv) { + size_t outsize; + const char *from_encoding; + + if (diff_populate_filespec(r, df, NULL)) + die("unable to read files to diff"); + + from_encoding = buffer_has_utf_bom(df->data, df->size); + if (!from_encoding) + BUG("autoencode triggered for non-utf content"); + + *outbuf = reencode_string_len(df->data, df->size, + "UTF-8", from_encoding, + &outsize); + + /* + * FIXME Our encoding guess failed. It's too late to return + * the original content, since the caller has already decided + * not to treat the contents as binary. But we could perhaps + * give some munged text form (e.g., by escaping high-bit + * characters and NULs). + */ + if (!*outbuf) + die_errno("unable to reencode from %s for path '%s'", + from_encoding, df->path); + + return outsize; + } + if (!textconv) { if (diff_populate_filespec(r, df, NULL)) die("unable to read files to diff"); diff --git a/diff.h b/diff.h index ad962560ebff5e..3d91a98346aa0c 100644 --- a/diff.h +++ b/diff.h @@ -194,6 +194,7 @@ struct diff_flags { unsigned dirstat_by_file; unsigned allow_textconv; unsigned textconv_set_via_cmdline; + unsigned allow_autoencode; unsigned diff_from_contents; unsigned dirty_submodules; unsigned ignore_untracked_in_submodules; From 996f39d33cb37a92a491664a4a752b03a202e98e Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 16 Jun 2015 14:47:18 -0400 Subject: [PATCH 133/156] trace: add GIT_TRACE_STDIN Sometimes tracing the invocation of git programs with GIT_TRACE is not quite enough to replay a situation; the interesting input to the program often comes over its standard input. For instance, if you want to replay a particular fetch (e.g., for performance analysis or debugging), you would want both the arguments and stdin sent to pack-objects. This patch lets you capture the stdin of any git process. For instance: GIT_TRACE=/tmp/processes.out \ GIT_TRACE_STDIN=/tmp/stdin.%p \ git daemon ... After a fetch, processes.out will contain a line like: 15:19:08.275493 [pid=13196] git.c:348 trace: built-in: git 'pack-objects' '--revs' '--thin' '--stdout' '--progress' '--delta-base-offset' And stdin.13196 (the pid picked from the above line) will contain its stdin. Signed-off-by: Jeff King --- git.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/git.c b/git.c index d7b5f88e52ae87..3116a38987ae86 100644 --- a/git.c +++ b/git.c @@ -919,6 +919,65 @@ static int run_argv(struct strvec *args) return done_alias; } +static int copy_stdin(int in, int out, void *data) +{ + struct trace_key *key = data; + while (1) { + char buf[8192]; + ssize_t len = xread(in, buf, sizeof(buf)); + if (!len) + break; + if (len < 0) { + warning("error reading stdin trace: %s", + strerror(errno)); + break; + } + + trace_verbatim(key, buf, len); + if (write_in_full(out, buf, len) < 0) { + warning("error writing stdin trace: %s", + strerror(errno)); + break; + } + } + close(in); + close(out); + return 0; +} + +static void trace_stdin(void) +{ + static struct trace_key key = TRACE_KEY_INIT(STDIN); + static struct async async; + + if (!trace_want(&key)) + return; + + memset(&async, 0, sizeof(async)); + async.proc = copy_stdin; + async.data = &key; + async.in = dup(0); + async.out = -1; + + if (async.in < 0 || start_async(&async) < 0) { + warning("unable to trace stdin: %s", strerror(errno)); + return ; + } + + /* + * At this point we've handed stdin off to the async process, + * so there we are past the point of no return. + */ + if (dup2(async.out, 0)) + die_errno("unable to redirect stdin from async process"); + close(async.out); + + /* + * leak async; we would know to finish_async() only when we are + * exiting, and there is no point then + */ +} + int cmd_main(int argc, const char **argv) { struct strvec args = STRVEC_INIT; @@ -935,6 +994,7 @@ int cmd_main(int argc, const char **argv) } trace_command_performance(argv); + trace_stdin(); /* * "git-xxxx" is the same as "git xxxx", but we obviously: From 4006f570193e6f929f0e2e2ad2dae2a4eb018cae Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 1 Sep 2026 02:29:55 -0400 Subject: [PATCH 134/156] revision: simplify mark_argv_for_free() callers You do not want to mark an argv element for freeing unless the caller has given us the free_removed_argv_elements flag. Originally we just called free() in this case, so each caller checked the flag itself. Now that we mark them via a helper function, we can push the check down into the helper. This saves a little bit of duplicated code, but also hopefully makes the result conceptually simpler. Every caller but one was already checking this flag. The exception is setup_revisions_from_strvec(), but it always sets the flag explicitly (since its whole purpose is managing argv memory). So even though it was not checking the flag, doing so is OK (it will always be set). --- revision.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/revision.c b/revision.c index f90cc00361fb82..ee1df92d1d0779 100644 --- a/revision.c +++ b/revision.c @@ -2317,8 +2317,11 @@ static timestamp_t parse_age(const char *arg) * usual const argv array, even when opt->free_removed_argv_elements is set). * We cast away the const on their behalf. */ -static void mark_argv_for_free(struct rev_info *revs, const char *str) +static void mark_argv_for_free(const struct setup_revision_opt *opt, + struct rev_info *revs, const char *str) { + if (!opt || !opt->free_removed_argv_elements) + return; if (!str) return; strvec_push_nodup(&revs->argv_to_free, (char *)str); @@ -2335,8 +2338,7 @@ static void overwrite_argv(int *argc, const char **argv, * cases around the free() and NULL operations. */ if (*value != argv[*argc]) { - if (opt && opt->free_removed_argv_elements) - mark_argv_for_free(revs, argv[*argc]); + mark_argv_for_free(opt, revs, argv[*argc]); argv[*argc] = *value; *value = NULL; } @@ -3055,8 +3057,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s const char *arg = argv[i]; if (strcmp(arg, "--")) continue; - if (opt && opt->free_removed_argv_elements) - mark_argv_for_free(revs, argv[i]); + mark_argv_for_free(opt, revs, argv[i]); argv[i] = NULL; argc = i; if (argv[i + 1]) @@ -3260,8 +3261,7 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s } if (argv) { - if (opt && opt->free_removed_argv_elements) - mark_argv_for_free(revs, argv[left]); + mark_argv_for_free(opt, revs, argv[left]); argv[left] = NULL; } @@ -3283,7 +3283,7 @@ void setup_revisions_from_strvec(struct strvec *argv, struct rev_info *revs, ret = setup_revisions(argv->nr, argv->v, revs, opt); for (size_t i = ret; i < argv->nr; i++) - mark_argv_for_free(revs, argv->v[i]); + mark_argv_for_free(opt, revs, argv->v[i]); argv->nr = ret; } From cbda348fed24c8bd2c7096d1ff95e4be2c4e0ebc Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 24 May 2026 01:11:39 -0400 Subject: [PATCH 135/156] shortlog: initialize mailmap With this done, we can drop the extra initialization from read_mailmap(). --- builtin/shortlog.c | 1 + mailmap.c | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/builtin/shortlog.c b/builtin/shortlog.c index dd46c4e109258c..717f784386c288 100644 --- a/builtin/shortlog.c +++ b/builtin/shortlog.c @@ -373,6 +373,7 @@ void shortlog_init(struct shortlog *log) { memset(log, 0, sizeof(*log)); + mailmap_init(&log->mailmap); read_mailmap(the_repository, &log->mailmap); log->entries = kh_init_shortlog(); diff --git a/mailmap.c b/mailmap.c index 726e292b3f883a..7fe31b6bca694a 100644 --- a/mailmap.c +++ b/mailmap.c @@ -219,7 +219,6 @@ int read_mailmap(struct repository *repo, struct mailmap *map) repo_config_get_pathname(repo, "mailmap.file", &mailmap_file); repo_config_get_string(repo, "mailmap.blob", &mailmap_blob); - string_list_init_dup(&map->map); map->map.cmp = namemap_cmp; if (!mailmap_blob && is_bare_repository(repo)) From f9f503196df192fee4e32f606047c480a4258e23 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 1 Sep 2026 20:09:41 -0400 Subject: [PATCH 136/156] submodule--helper: free URL when repository setup fails If repo setup fails, we'll return an error without freeing the allocated url string, leaking the memory. We do trigger this error in the test suite, but never with the leak. We only allocate a url if submodule_from_path() returned something, but out our error tests use things like totally non-existent submodules. We can cover this case in the tests by asking about a submodule that exists but which has not been initialized. The new test fails with SANITIZE=leak. The smallest fix would just be a call to free(url), but I think it's a little nicer to set up a dedicated out-path for cleanup here. The previous commit made it safe to call repo_clear() even if repo_submodule_init() fails. --- builtin/submodule--helper.c | 10 +++++++--- t/t7426-submodule-get-default-remote.sh | 17 +++++++++++++++++ 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index e7cd3225fa84c4..469e3dbcc9750e 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -80,6 +80,7 @@ static int get_default_remote_submodule(const char *module_path, char **default_ struct repository subrepo; const char *remote_name = NULL; char *url = NULL; + int ret = 0; sub = submodule_from_path(the_repository, null_oid(the_hash_algo), module_path); if (sub && sub->url) { @@ -96,9 +97,11 @@ static int get_default_remote_submodule(const char *module_path, char **default_ } if (repo_submodule_init(&subrepo, the_repository, module_path, - null_oid(the_hash_algo)) < 0) - return die_message(_("could not get a repository handle for submodule '%s'"), + null_oid(the_hash_algo)) < 0) { + ret = die_message(_("could not get a repository handle for submodule '%s'"), module_path); + goto out; + } /* Look up by URL first */ if (url) @@ -108,10 +111,11 @@ static int get_default_remote_submodule(const char *module_path, char **default_ *default_remote = xstrdup(remote_name); +out: repo_clear(&subrepo); free(url); - return 0; + return ret; } static int module_get_default_remote(int argc, const char **argv, const char *prefix, diff --git a/t/t7426-submodule-get-default-remote.sh b/t/t7426-submodule-get-default-remote.sh index b842af9a2d26ff..0379c9f04489f3 100755 --- a/t/t7426-submodule-get-default-remote.sh +++ b/t/t7426-submodule-get-default-remote.sh @@ -60,6 +60,23 @@ test_expect_success 'get-default-remote fails with non-submodule path' ' ) ' +test_expect_success 'get-default-remote fails with uninitialized submodule' ' + test_when_finished " + git -C super config -f .gitmodules --remove-section submodule.uninitialized && + git -C super update-index --force-remove uninitialized + " && + ( + cd super && + git config -f .gitmodules submodule.uninitialized.path uninitialized && + git config -f .gitmodules submodule.uninitialized.url ../sub && + head=$(git -C ../sub rev-parse HEAD) && + git update-index --add --cacheinfo 160000,$head,uninitialized && + test_must_fail git submodule--helper get-default-remote \ + uninitialized 2>err && + test_grep "could not get a repository handle" err + ) +' + test_expect_success 'get-default-remote fails without path argument' ' ( cd super && From 8666057da9873b28d35bb41e37361700e6322152 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Wed, 9 Nov 2016 23:24:09 -0500 Subject: [PATCH 137/156] optionally block out-of-repo symlinks Signed-off-by: Jeff King --- apply.c | 6 +- entry.c | 3 +- path.c | 73 ++++++++++++++++++ path.h | 2 + repo-settings.c | 2 + repo-settings.h | 2 + t/meson.build | 1 + t/t2031-checkout-symlink-external.sh | 109 +++++++++++++++++++++++++++ 8 files changed, 195 insertions(+), 3 deletions(-) create mode 100755 t/t2031-checkout-symlink-external.sh diff --git a/apply.c b/apply.c index f00b7ba4d3a7e6..369ed969c9d95b 100644 --- a/apply.c +++ b/apply.c @@ -4524,11 +4524,13 @@ static int try_create_file(struct apply_state *state, const char *path, return !!mkdir(path, 0777); } - if (repo_has_symlinks(state->repo) && S_ISLNK(mode)) + if (repo_has_symlinks(state->repo) && S_ISLNK(mode)) { /* Although buf:size is counted string, it also is NUL * terminated. */ - return !!symlink(buf, path); + prepare_repo_settings(state->repo); + return !!safe_symlink(state->repo, buf, path); + } fd = open(path, O_CREAT | O_EXCL | O_WRONLY, (mode & 0100) ? 0777 : 0666); if (fd < 0) diff --git a/entry.c b/entry.c index 1c4f0f44070ea3..8eaba9106f84b0 100644 --- a/entry.c +++ b/entry.c @@ -4,6 +4,7 @@ #include "odb.h" #include "odb/streaming.h" #include "dir.h" +#include "path.h" #include "environment.h" #include "gettext.h" #include "hex.h" @@ -323,7 +324,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca state->istate->repo : the_repository) || to_tempfile) goto write_file_entry; - ret = symlink(new_blob, path); + ret = safe_symlink(the_repository, new_blob, path); free(new_blob); if (ret) return error_errno("unable to create symlink %s", path); diff --git a/path.c b/path.c index c3a709a9284b7f..beea077c8b28f0 100644 --- a/path.c +++ b/path.c @@ -1542,6 +1542,79 @@ int looks_like_command_line_option(const char *str) return str && str[0] == '-'; } +static int symlink_leaves_repo(const char *target, const char *linkpath) +{ + /* + * Absolute paths are always considered to leave the repository (even + * if they happen to point to the working tree path). + */ + if (is_absolute_path(target)) + return 1; + + /* + * Allow relative paths that start with a sequence of "../", + * as long as they do not break out of the symlink's root. + * This loop will detect break-out cases and return; otherwise, at the + * end of the loop "target" will point to the first non-".." component. + * + * We count the depth of linkpath by eating up directory components left + * to right. Technically the symlink would resolve right-to-left, but + * we don't care about the actual values, only the number. + */ + while (target[0] == '.') { + if (!target[1]) { + /* trailing "." -- ignore */ + target++; + } else if (is_dir_sep(target[1])) { + /* "./" -- ignore */ + target += 2; + } else if (target[1] == '.' && + (!target[2] || is_dir_sep(target[2]))) { + /* ".." or "../" -- drop one from linkpath depth */ + while (!is_dir_sep(*linkpath)) { + /* end-of-string; target exceeded our depth */ + if (!*linkpath) + return 1; + linkpath++; + } + /* skip final "/" */ + linkpath++; + + /* skip past ".." */ + target += 2; + /* and "/" if present */ + if (is_dir_sep(*target)) + target++; + } + } + + /* + * Now we have a path in "target" that only go down into the tree. + * Disallow any interior "../", like "foo/../bar". These might be + * OK, but we cannot know unless we know whether "foo" is itself a + * symlink. So err on the side of caution. + */ + while (*target) { + const char *v; + if (skip_prefix(target, "..", &v) && (!*v || is_dir_sep(*v))) + return 1; + target++; + } + + return 0; +} + +int safe_symlink(struct repository *r, const char *target, const char *linkpath) +{ + if (!r->settings.allow_external_symlinks && + symlink_leaves_repo(target, linkpath)) { + errno = EPERM; + return -1; + } + + return symlink(target, linkpath); +} + char *xdg_config_home_for(const char *subdir, const char *filename) { const char *home, *config_home; diff --git a/path.h b/path.h index 7e7408dd05fce3..3c5d5f74d0c539 100644 --- a/path.h +++ b/path.h @@ -292,6 +292,8 @@ enum path_format { void format_path(struct strbuf *dest, const char *path, const char *prefix, enum path_format format); +int safe_symlink(struct repository *, const char *target, const char *linkpath); + # ifdef USE_THE_REPOSITORY_VARIABLE # include "strbuf.h" # include "repository.h" diff --git a/repo-settings.c b/repo-settings.c index f3be3b8c5a3d09..ee77e664b1a256 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -84,6 +84,8 @@ void prepare_repo_settings(struct repository *r) &r->settings.pack_use_bitmap_boundary_traversal, r->settings.pack_use_bitmap_boundary_traversal); repo_cfg_bool(r, "core.usereplacerefs", &r->settings.read_replace_refs, 1); + repo_cfg_bool(r, "core.allowexternalsymlinks", + &r->settings.allow_external_symlinks, 1); /* * The GIT_TEST_MULTI_PACK_INDEX variable is special in that diff --git a/repo-settings.h b/repo-settings.h index e5253ead025c83..f20b154bc18831 100644 --- a/repo-settings.h +++ b/repo-settings.h @@ -63,6 +63,8 @@ struct repo_settings { int max_allowed_tree_depth; char *hooks_path; + + int allow_external_symlinks; }; #define REPO_SETTINGS_INIT { \ .shared_repository = -1, \ diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..e31d53327fc05e 100644 --- a/t/meson.build +++ b/t/meson.build @@ -281,6 +281,7 @@ integration_tests = [ 't2026-checkout-pathspec-file.sh', 't2027-checkout-track.sh', 't2030-unresolve-info.sh', + 't2031-checkout-symlink-external.sh', 't2050-git-dir-relative.sh', 't2060-switch.sh', 't2070-restore.sh', diff --git a/t/t2031-checkout-symlink-external.sh b/t/t2031-checkout-symlink-external.sh new file mode 100755 index 00000000000000..73738da97b9a36 --- /dev/null +++ b/t/t2031-checkout-symlink-external.sh @@ -0,0 +1,109 @@ +#!/bin/sh + +test_description='detection and prevention of out-of-tree symlinks' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +. ./test-lib.sh + +if ! test_have_prereq SYMLINKS +then + skip_all='skipping external symlink tests (missing SYMLINKS)' + test_done +fi + +create_symlink() { + symlink=$1 + target=$2 + test_expect_success "create symlink ($symlink)" ' + sha1=$(printf "%s" "$target" | git hash-object -w --stdin) && + git update-index --add --cacheinfo "120000,$sha1,$symlink" + ' +} + +check_symlink () { + symlink=$1 + config=$2 + outcome=$3 + expect=$4 + + if test "$outcome" = "allow" + then + fail= + : ${expect:=test_cmp ../target} + else + fail=test_must_fail + : ${expect:=! cat} + fi + + test_expect_success " check symlink ($symlink, $config -> $outcome)" " + rm -f $symlink && + $fail git -c core.allowExternalSymlinks=$config \\ + checkout-index -- $symlink && + $expect $symlink + " +} + +# we want to try breaking out of the repository, +# so let's work inside a sub-repository, and break +# out to the top-level trash directory +test_expect_success 'set up repository' ' + echo content >target && + git init subrepo && + cd subrepo && + test_commit base && + echo content >in-repo-target +' + +create_symlink in-repo in-repo-target +check_symlink in-repo false allow + +create_symlink subdir/in-repo ../in-repo-target +check_symlink subdir/in-repo false allow + +create_symlink absolute "$TRASH_DIRECTORY/target" +check_symlink absolute true allow +check_symlink absolute false forbid + +create_symlink relative "../target" +check_symlink relative true allow +check_symlink relative false forbid + +create_symlink curdir . +check_symlink curdir false allow test_path_is_dir +create_symlink sneaky curdir/../target +check_symlink sneaky true allow +check_symlink sneaky false forbid + +test_expect_success 'applying a patch checks symlink config' ' + git diff-index -p --cached HEAD -- relative >patch && + rm -f relative && + git -c core.allowExternalSymlinks=true apply Date: Tue, 27 Feb 2018 20:31:16 -0500 Subject: [PATCH 138/156] diff: enable autoencode by default This turns on auto-encode whenever textconv is on and the --binary option has not been used. Signed-off-by: Jeff King --- diff.c | 5 +++++ diff.h | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/diff.c b/diff.c index 86515d6a06edfd..1aa9fb32ea5e46 100644 --- a/diff.c +++ b/diff.c @@ -5199,6 +5199,8 @@ void repo_diff_setup(struct repository *r, struct diff_options *options) options->color_moved = diff_color_moved_default; options->color_moved_ws_handling = diff_color_moved_ws_default; + + options->flags.allow_autoencode = -1; } static const char diff_status_letters[] = { @@ -5397,6 +5399,9 @@ void diff_setup_done(struct diff_options *options) if (options->pathspec.has_wildcard && options->max_depth_valid) die("max-depth cannot be used with wildcard pathspecs"); + + if (options->flags.allow_autoencode == -1) + options->flags.allow_autoencode = !options->flags.binary; } int parse_long_opt(const char *opt, const char **argv, diff --git a/diff.h b/diff.h index 3d91a98346aa0c..d13df7b6c2094b 100644 --- a/diff.h +++ b/diff.h @@ -194,7 +194,7 @@ struct diff_flags { unsigned dirstat_by_file; unsigned allow_textconv; unsigned textconv_set_via_cmdline; - unsigned allow_autoencode; + int allow_autoencode; unsigned diff_from_contents; unsigned dirty_submodules; unsigned ignore_untracked_in_submodules; From 20c8dd4511e9330972807026ceb8975cac8e04ca Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 30 Nov 2017 15:14:40 -0500 Subject: [PATCH 139/156] curl: handle non-http protocols in trace Signed-off-by: Jeff King --- http.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 60 insertions(+), 7 deletions(-) diff --git a/http.c b/http.c index c8fcfd7693e897..25a108d7071e67 100644 --- a/http.c +++ b/http.c @@ -827,7 +827,7 @@ static int set_curl_proxy_type(CURL *result, const char *protocol) } /* Return 1 if redactions have been made, 0 otherwise. */ -static int redact_sensitive_header(struct strbuf *header, size_t offset) +static int redact_http_header(struct strbuf *header, size_t offset) { int ret = 0; char *sensitive_header; @@ -924,14 +924,67 @@ static void redact_sensitive_info_header(struct strbuf *header) if (trace_curl_redact && match_curl_h2_trace(header->buf, &sensitive_header)) { - if (redact_sensitive_header(header, sensitive_header - header->buf)) { + if (redact_http_header(header, sensitive_header - header->buf)) { /* redaction ate our closing bracket */ strbuf_addch(header, ']'); } } } -static void curl_dump_header(const char *text, unsigned char *ptr, size_t size, int hide_sensitive_header) +static void redact_imap_header(struct strbuf *header) +{ + const char *p; + + /* skip past the command tag */ + p = strchr(header->buf, ' '); + if (!p) + return; /* no tag */ + p++; + + if (skip_prefix(p, "AUTHENTICATE ", &p)) { + /* the first token is the auth type, which is OK to log */ + while (*p && !isspace(*p)) + p++; + /* the rest is an opaque blob; fall through to redact */ + } else if (skip_prefix(p, "LOGIN ", &p)) { + /* fall through to redact both login and password */ + } else { + /* not a sensitive header */ + return; + } + + strbuf_setlen(header, p - header->buf); + strbuf_addstr(header, " "); +} + +static void redact_sensitive_header(CURL *handle, struct strbuf *header) +{ + const char *url; + int ret; + + ret = curl_easy_getinfo(handle, CURLINFO_EFFECTIVE_URL, &url); + if (!ret && url) { + if (starts_with(url, "http")) { + redact_http_header(header, 0); + return; + } + if (starts_with(url, "imap")) { + redact_imap_header(header); + return; + } + } + + /* + * We weren't able to figure out the protocol. Err on the side of + * redacting too much. + */ + redact_http_header(header, 0); + redact_imap_header(header); +} + +static void curl_dump_header(CURL *handle, const char *text, + unsigned char *ptr, size_t size, + int hide_sensitive_header) { struct strbuf out = STRBUF_INIT; struct strbuf **headers, **header; @@ -945,7 +998,7 @@ static void curl_dump_header(const char *text, unsigned char *ptr, size_t size, for (header = headers; *header; header++) { if (hide_sensitive_header) - redact_sensitive_header(*header, 0); + redact_sensitive_header(handle, *header); strbuf_insertstr((*header), 0, text); strbuf_insertstr((*header), strlen(text), ": "); strbuf_rtrim((*header)); @@ -996,7 +1049,7 @@ static void curl_dump_info(char *data, size_t size) strbuf_release(&buf); } -static int curl_trace(CURL *handle UNUSED, curl_infotype type, +static int curl_trace(CURL *handle, curl_infotype type, char *data, size_t size, void *userp UNUSED) { @@ -1009,7 +1062,7 @@ static int curl_trace(CURL *handle UNUSED, curl_infotype type, break; case CURLINFO_HEADER_OUT: text = "=> Send header"; - curl_dump_header(text, (unsigned char *)data, size, DO_FILTER); + curl_dump_header(handle, text, (unsigned char *)data, size, DO_FILTER); break; case CURLINFO_DATA_OUT: if (trace_curl_data) { @@ -1025,7 +1078,7 @@ static int curl_trace(CURL *handle UNUSED, curl_infotype type, break; case CURLINFO_HEADER_IN: text = "<= Recv header"; - curl_dump_header(text, (unsigned char *)data, size, NO_FILTER); + curl_dump_header(handle, text, (unsigned char *)data, size, NO_FILTER); break; case CURLINFO_DATA_IN: if (trace_curl_data) { From ae11b90ba0db5a853f5a5648c19240b37ed6ec74 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Tue, 11 Aug 2020 01:29:20 -0400 Subject: [PATCH 140/156] run-command config experiment This used to take a repo option, but it gets tricky: if we are called from start_command(), how do we know which repository struct to use? Instead, this just punts and does not respect in-repo config at all. Signed-off-by: Jeff King --- run-command.c | 3 +++ trace.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ trace.h | 10 ++++++++++ 3 files changed, 59 insertions(+) diff --git a/run-command.c b/run-command.c index ce84db87824262..66ae0fc8fd8373 100644 --- a/run-command.c +++ b/run-command.c @@ -681,6 +681,9 @@ int start_command(struct child_process *cmd) int failed_errno; const char *str; + if (cmd->git_cmd) + trace_config_for(NULL, cmd->args.v[0], &cmd->env); + /* * In case of errors we must keep the promise to close FDs * that have been passed in via ->in and ->out. diff --git a/trace.c b/trace.c index 3f33870353b197..42f1543f94c5a5 100644 --- a/trace.c +++ b/trace.c @@ -29,6 +29,8 @@ #include "quote.h" #include "setup.h" #include "trace.h" +#include "strvec.h" +#include "config.h" struct trace_key trace_default_key = { "GIT_TRACE", 0, 0, 0 }; struct trace_key trace_perf_key = TRACE_KEY_INIT(PERFORMANCE); @@ -442,3 +444,47 @@ void trace_command_performance(const char **argv) sq_quote_argv_pretty(&command_line, argv); trace_performance_enter(); } + +struct trace_config_data { + const char *want_cmd; + struct strvec *out; +}; + +static int trace_config_cb(const char *var, const char *value, + const struct config_context *ctx UNUSED, void *vdata) +{ + struct trace_config_data *data = vdata; + const char *have_cmd, *key; + size_t have_len; + + if (!parse_config_key(var, "trace", &have_cmd, &have_len, &key) && + have_cmd && + !strncmp(data->want_cmd, have_cmd, have_len) && + data->want_cmd[have_len] == '\0') { + struct strbuf buf = STRBUF_INIT; + + strbuf_addstr(&buf, "GIT_TRACE_"); + while (*key) + strbuf_addch(&buf, toupper(*key++)); + + /* + * Environment always takes precedence over config, so do not + * override existing variables. We cannot rely on setenv()'s + * overwrite flag here, because we may pass the list off to + * a spawn() implementation, which always overwrites. + */ + if (!getenv(buf.buf)) + strvec_pushf(data->out, "%s=%s", buf.buf, value); + + strbuf_release(&buf); + } + return 0; +} + +void trace_config_for(struct repository *r, const char *cmd, struct strvec *out) +{ + struct trace_config_data data; + data.want_cmd = cmd; + data.out = out; + repo_config(r, trace_config_cb, &data); +} diff --git a/trace.h b/trace.h index 9152fe9b3e565e..1fdbd3956d98d0 100644 --- a/trace.h +++ b/trace.h @@ -3,6 +3,9 @@ #include "strbuf.h" +struct repository; +struct strvec; + /** * The trace API can be used to print debug messages to stderr or a file. Trace * code is inactive unless explicitly enabled by setting `GIT_TRACE*` environment @@ -127,6 +130,13 @@ void trace_command_performance(const char **argv); void trace_verbatim(struct trace_key *key, const void *buf, unsigned len); uint64_t trace_performance_enter(void); +/** + * Load any trace-related config for git command "cmd", and insert the matching + * environment variables into "out", which is suitable for use by run-command + * and friends. + */ +void trace_config_for(struct repository *r, const char *cmd, struct strvec *out); + /* * TRACE_CONTEXT may be set to __FUNCTION__ if the compiler supports it. The * default is __FILE__, as it is consistent with assert(), and static function From b5c9d6e7101cac0a896993988a11516b327efca8 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 26 May 2017 11:59:34 -0400 Subject: [PATCH 141/156] prepare_revision_walk: complain of unused objects todo: - write commit message - explain fast-export test issue (flushed out by switching to die; seems legit, but it does mean we double the warnings Signed-off-by: Jeff King --- revision.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/revision.c b/revision.c index 35afe52208e710..7fe81b46f66fde 100644 --- a/revision.c +++ b/revision.c @@ -379,6 +379,16 @@ void add_pending_oid(struct rev_info *revs, const char *name, add_pending_object(revs, object, name); } +static void warn_ignored_object(struct object *object, const char *name) +{ + if (object->flags & UNINTERESTING) + return; + + warning(_("ignoring %s object in traversal: %s"), + type_name(object->type), + (name && *name) ? name : oid_to_hex(&object->oid)); +} + static struct commit *handle_commit(struct rev_info *revs, struct object_array_entry *entry) { @@ -450,8 +460,10 @@ static struct commit *handle_commit(struct rev_info *revs, */ if (object->type == OBJ_TREE) { struct tree *tree = (struct tree *)object; - if (!revs->tree_objects) + if (!revs->tree_objects) { + warn_ignored_object(object, name); return NULL; + } if (flags & UNINTERESTING) { mark_tree_contents_uninteresting(revs->repo, tree); return NULL; @@ -464,8 +476,10 @@ static struct commit *handle_commit(struct rev_info *revs, * Blob object? You know the drill by now.. */ if (object->type == OBJ_BLOB) { - if (!revs->blob_objects) + if (!revs->blob_objects) { + warn_ignored_object(object, name); return NULL; + } if (flags & UNINTERESTING) return NULL; add_pending_object_with_path(revs, object, name, mode, path); From 8ec5957c71dc83ab662a6c196cb0be98540162c5 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Fri, 23 May 2014 15:29:34 -0400 Subject: [PATCH 142/156] teach update-ref a "--rename" option You can rename a branch with "branch -m", but there is no plumbing mechanism to rename an arbitrary ref. You can simulate it with: git update-ref newname oldname && git update-ref -d oldname but that drops the reflog from oldname. This patch instead allows: git update-ref --rename oldname newname Unlike "git branch -m", it does not munge the branch config (since this plumbing command is purely about refs), nor does it allow a missing "oldname" to mean an implicit HEAD (because it is plumbing, and we are better to make the caller be explicit than risk a surprise). Signed-off-by: Jeff King --- builtin/update-ref.c | 24 +++++++++++++++++++++++- t/t1400-update-ref.sh | 23 +++++++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/builtin/update-ref.c b/builtin/update-ref.c index 6355c3dd3e7fc5..f7d3ff84fc06c2 100644 --- a/builtin/update-ref.c +++ b/builtin/update-ref.c @@ -15,6 +15,7 @@ static const char * const git_update_ref_usage[] = { N_("git update-ref [] -d []"), N_("git update-ref [] []"), + N_("git update-ref [] --rename "), N_("git update-ref [] --stdin [-z] [--batch-updates]"), NULL }; @@ -34,6 +35,19 @@ struct command_options { bool allow_update_failures; }; +static int do_rename_ref(const char *from, const char *to) +{ + struct strbuf msg = STRBUF_INIT; + int ret; + + strbuf_addf(&msg, "update-ref: renamed %s to %s", from, to); + ret = refs_rename_ref(get_main_ref_store(the_repository), + from, to, msg.buf); + strbuf_release(&msg); + + return !!ret; +} + /* * Parse one whitespace- or NUL-terminated, possibly C-quoted argument * and append the result to arg. Return a pointer to the terminator. @@ -817,10 +831,12 @@ int cmd_update_ref(int argc, int delete = 0, no_deref = 0, read_stdin = 0, end_null = 0; int create_reflog = 0; unsigned int flags = 0; + int rename = 0; struct option options[] = { OPT_STRING( 'm', NULL, &msg, N_("reason"), N_("reason of the update")), OPT_BOOL('d', NULL, &delete, N_("delete the reference")), + OPT_BOOL( 0 , "rename", &rename, N_("rename the reference")), OPT_BOOL( 0 , "no-deref", &no_deref, N_("update not the one it points to")), OPT_BOOL('z', NULL, &end_null, N_("stdin has NUL-terminated arguments")), @@ -845,7 +861,7 @@ int cmd_update_ref(int argc, } if (read_stdin) { - if (delete || argc > 0) + if (delete || rename || argc > 0) usage_with_options(git_update_ref_usage, options); if (end_null) line_termination = '\0'; @@ -858,6 +874,12 @@ int cmd_update_ref(int argc, if (end_null) usage_with_options(git_update_ref_usage, options); + if (rename) { + if (delete || argc < 2 || argc > 2) + usage_with_options(git_update_ref_usage, options); + return do_rename_ref(argv[0], argv[1]); + } + if (delete) { if (argc < 1 || argc > 2) usage_with_options(git_update_ref_usage, options); diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index 269fdaa3ed71aa..1623e4092df7b2 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -12,6 +12,7 @@ Z=$ZERO_OID m=refs/heads/main outside=refs/foo bare=bare-repo +d=refs/heads/dst create_test_commits () { @@ -439,8 +440,30 @@ test_expect_success 'Query "main@{2005-05-28}" (past end of history)' ' test_grep -F "warning: log for ref $m unexpectedly ended on $ld" e ' +test_expect_success 'rename a ref' ' + git rev-parse --verify $m >expect && + + # set the date to match the reflog entries we created + # above, which do not follow test_tick; otherwise + # the we write an out-of-order entry into the reflog, + # which confuses the reflog parser + GIT_COMMITTER_DATE=$ld \ + git update-ref --rename $m $d && + + test_must_fail git rev-parse --verify $m && + git rev-parse --verify $d >o && + test_cmp expect o +' + +test_expect_success 'renames copy reflogs' ' + echo "$C" >expect && + git rev-parse --verify "$d@{2005-05-26 23:32:00}" >o && + test_cmp expect o +' + rm -f expect git update-ref -d $m +git update-ref -d $d test_expect_success 'query reflog with gap' ' test_when_finished "git update-ref -d $m" && From 5192d303acce52b9fe2faecd0af6638e6edfeef4 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 10 Jul 2017 04:38:01 -0400 Subject: [PATCH 143/156] fast-export: leave warnings to revision code Signed-off-by: Jeff King --- builtin/fast-export.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/builtin/fast-export.c b/builtin/fast-export.c index 50278bef7a6a4d..8d3ebb43609a6c 100644 --- a/builtin/fast-export.c +++ b/builtin/fast-export.c @@ -1082,9 +1082,6 @@ static void get_tags_and_duplicates(struct rev_cmdline_info *info) commit = get_commit(e, full_name); if (!commit) { - warning(_("%s: unexpected object of type %s, skipping."), - e->name, - type_name(e->item->type)); free(full_name); continue; } From a02c0fb2aff2bc211284ef938e4689e22caf7b23 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 3 Sep 2026 01:02:14 -0400 Subject: [PATCH 144/156] t/perf: add pack checksum performance tests Several callers rewrite a completed pack header and then recompute its checksum. This is most visible when the pack data is large but otherwise cheap to produce, but reproducing it currently requires assembling an ad-hoc repository and input for each caller. Add perf coverage for the three interesting cases: splitting an existing pack with pack-objects, completing a thin pack with index-pack, and finishing a pack written by fast-import. Use the same pair of large nearly-identical objects for the first two cases, plus an independent input for fast-import. The object size defaults to 128 MiB, which is large enough to force the pack-objects case past its 100 MiB split limit. It can be adjusted with GIT_PERF_PACK_CHECKSUM_SIZE when a larger or quicker run is desired. --- t/perf/p5305-pack-checksum.sh | 61 +++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100755 t/perf/p5305-pack-checksum.sh diff --git a/t/perf/p5305-pack-checksum.sh b/t/perf/p5305-pack-checksum.sh new file mode 100755 index 00000000000000..6e55f3c44d6e3a --- /dev/null +++ b/t/perf/p5305-pack-checksum.sh @@ -0,0 +1,61 @@ +#!/bin/sh + +test_description='Tests performance of recomputing pack checksums' +. ./perf-lib.sh + +test_perf_fresh_repo + +pack_size=${GIT_PERF_PACK_CHECKSUM_SIZE:-134217728} +pack_limit=$((pack_size / 2)) +export pack_size pack_limit + +test_expect_success 'set up large objects' ' + # The small change to "big" makes the second version a good delta + # against the first. The thin pack will contain that delta but omit + # its base, which index-pack must append before fixing the header. + test-tool genrandom base $pack_size >big && + git add big && + git commit -m base && + base=$(git rev-parse HEAD) && + + # Without another object, the thin pack would contain little besides + # the small delta. Include an unrelated large object so that fixing + # the header has to re-read and checksum a large amount of existing + # pack data. It also gives pack-objects enough data to split below. + printf foo >>big && + test-tool genrandom unrelated $pack_size >unrelated && + git add big unrelated && + git commit -m tip && + tip=$(git rev-parse HEAD) && + + printf "%s\n^%s\n" "$tip" "$base" | + git pack-objects --revs --thin --stdout >thin.pack && + + # Use distinct contents here so that fast-import cannot notice that + # the object is already in the repository and skip writing it. + { + echo blob && + echo "data $pack_size" && + test-tool genrandom fast-import $pack_size + } >fast-import.in && + + git repack -ad +' + +test_perf 'split pack with pack-objects' \ + --setup 'rm -f pack-big-*' ' + git pack-objects --all --max-pack-size=$pack_limit pack-big Date: Wed, 2 Sep 2026 04:29:06 -0400 Subject: [PATCH 145/156] pack-write: use unsafe hash for header fixup When we have to fix up the object count in a pack header, we also have to recompute the pack checksum. We currently do so using the regular object-hashing implementation, including its collision detection. But this is a checksum of pack data, not an object name. We can use the faster unsafe variant, just as the hashfile API does when computing the original pack checksum. This comes up for split packs from pack-objects, whenever fast-import finishes a pack, and when index-pack completes a thin pack. The perf tests added in the previous commit show improvements for all three: Test HEAD^ HEAD ------------------------------------------------------------------------------- 5305.2: split pack with pack-objects 12.72(12.49+0.23) 10.30(10.12+0.18) -19.0% 5305.3: fix thin pack with index-pack 5.27(5.03+0.24) 4.39(4.29+0.09) -16.7% 5305.4: fix pack header with fast-import 4.48(4.41+0.06) 3.93(3.88+0.05) -12.3% These were compiled with OPENSSL_SHA1_UNSAFE=1 and -O2. --- pack-write.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pack-write.c b/pack-write.c index 85674e4b726bab..476ff8fc1c8bf5 100644 --- a/pack-write.c +++ b/pack-write.c @@ -402,6 +402,13 @@ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo, char *buf; ssize_t read_result; + /* + * We are computing a pack checksum here, not an object hash. So we + * can use a faster "unsafe" variant that does not do collision + * detection. + */ + hash_algo = unsafe_hash_algo(hash_algo); + git_hash_init(&old_hash_ctx, hash_algo); git_hash_init(&new_hash_ctx, hash_algo); From b9d1b188522daf5907ff8ff2b47a6b716185ddce Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 20 Aug 2026 04:21:02 -0400 Subject: [PATCH 146/156] pack-objects: consistently use hashfile algorithm When we split a pack due to the --max-pack-size argument, we have to fix up the header to record the correct number of objects, and thus also re-hash the whole file and fix up the footer checksum. The previous commit taught fixup_pack_header_footer() to switch to the unsafe variant of the requested algorithm, so passing the_hash_algo here already produces the desired result. But it is more direct to pass the algorithm from the hashfile whose output we are fixing. Save the pointer before calling finalize_hashfile(), which frees the hashfile itself. Likewise, use the hashfile algorithm's raw size when accounting for the trailing checksum in the bytes-written trace. The safe and unsafe variants have the same raw size, so neither change affects behavior, but this keeps all of the bookkeeping tied to the algorithm actually used by the hashfile. Suggested-by: Patrick Steinhardt --- builtin/pack-objects.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 708b719f403b96..9f9531871d1da6 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1391,8 +1391,7 @@ static void write_pack_file(void) display_progress(progress_state, written); } - bytes_written += hashfile_total(f) + - the_repository->hash_algo->rawsz; + bytes_written += hashfile_total(f) + f->algop->rawsz; if (pack_to_stdout) { /* * We never fsync when writing to stdout since we may @@ -1411,9 +1410,9 @@ static void write_pack_file(void) * If we wrote the wrong number of entries in the * header, rewrite it like in fast-import. */ - + const struct git_hash_algo *algo = f->algop; int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); - fixup_pack_header_footer(the_hash_algo, fd, hash, + fixup_pack_header_footer(algo, fd, hash, pack_tmp_name, nr_written, hash, offset); close(fd); From 529669c1a8ecac0f08e5f42fab7e99815cb49e57 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 3 Sep 2026 00:37:47 -0400 Subject: [PATCH 147/156] pack-write: drop old checksum from header fixup When fixup_pack_header_footer() rewrites the object count in a pack header, it computes the new checksum by reading the whole pack back from disk. In parallel, it also computes the checksum of the original data and compares it to the checksum produced while writing the file. But this extra check is not buying us much. The important result of the function is the checksum of the data which is actually on disk, and we compute that independently either way. The check also means hashing the whole file twice. Drop the old checksum entirely, along with the partial-offset machinery used to compute it. This also lets callers avoid asking their hashfile for a result which they no longer use. In p5305, this improves the split pack, thin pack, and fast-import cases by 9.4%, 6.5%, and 6.9%, respectively. These results compare this commit directly to its predecessor, using OPENSSL_SHA1_UNSAFE=1 and -O2. --- builtin/fast-import.c | 6 ++--- builtin/index-pack.c | 10 ++------ builtin/pack-objects.c | 3 +-- object-file.c | 3 +-- pack-write.c | 57 +++--------------------------------------- pack.h | 3 +-- 6 files changed, 11 insertions(+), 71 deletions(-) diff --git a/builtin/fast-import.c b/builtin/fast-import.c index fbd919982c956a..77ed9b64494c40 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -897,18 +897,16 @@ static void end_packfile(void) if (object_count) { struct odb_source_files *files = odb_source_files_downcast(pack_data->repo->objects->sources); struct packed_git *new_p; - struct object_id cur_pack_oid; char *idx_name; int i; struct branch *b; struct tag *t; close_pack_windows(pack_data); - finalize_hashfile(pack_file, cur_pack_oid.hash, FSYNC_COMPONENT_PACK, 0); + finalize_hashfile(pack_file, NULL, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(the_hash_algo, pack_data->pack_fd, pack_data->hash, pack_data->pack_name, - object_count, cur_pack_oid.hash, - pack_size); + object_count); if (object_count <= unpack_limit) { if (!loosen_small_pack(pack_data)) { diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 6b2a87e2d39355..02831fba6c0e06 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1378,7 +1378,6 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha if (fix_thin_pack) { struct hashfile *f; - unsigned char read_hash[GIT_MAX_RAWSZ], tail_hash[GIT_MAX_RAWSZ]; struct strbuf msg = STRBUF_INIT; int nr_unresolved = nr_ofs_deltas + nr_ref_deltas - nr_resolved_deltas; int nr_objects_initial = nr_objects; @@ -1395,14 +1394,9 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha nr_objects - nr_objects_initial); stop_progress_msg(&progress, msg.buf); strbuf_release(&msg); - finalize_hashfile(f, tail_hash, FSYNC_COMPONENT_PACK, 0); - hashcpy(read_hash, pack_hash, the_repository->hash_algo); + finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(the_hash_algo, output_fd, pack_hash, - curr_pack, nr_objects, - read_hash, consumed_bytes-the_hash_algo->rawsz); - if (!hasheq(read_hash, tail_hash, the_repository->hash_algo)) - die(_("Unexpected tail checksum for %s " - "(disk corruption?)"), curr_pack); + curr_pack, nr_objects); } if (nr_ofs_deltas + nr_ref_deltas != nr_resolved_deltas) die(Q_("pack has %d unresolved delta", diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 9f9531871d1da6..154614500aaec3 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1413,8 +1413,7 @@ static void write_pack_file(void) const struct git_hash_algo *algo = f->algop; int fd = finalize_hashfile(f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(algo, fd, hash, - pack_tmp_name, nr_written, - hash, offset); + pack_tmp_name, nr_written); close(fd); if (write_bitmap_index) { if (write_bitmap_index != WRITE_BITMAP_QUIET) diff --git a/object-file.c b/object-file.c index a4cbf8b081df1e..1f4b0f6b370fb0 100644 --- a/object-file.c +++ b/object-file.c @@ -833,8 +833,7 @@ static void flush_packfile_transaction(struct odb_transaction_files *transaction } else { int fd = finalize_hashfile(state->f, hash, FSYNC_COMPONENT_PACK, 0); fixup_pack_header_footer(repo->hash_algo, fd, hash, state->pack_tmp_name, - state->nr_written, hash, - state->offset); + state->nr_written); close(fd); } diff --git a/pack-write.c b/pack-write.c index 476ff8fc1c8bf5..a495d336844768 100644 --- a/pack-write.c +++ b/pack-write.c @@ -377,27 +377,15 @@ off_t write_pack_header(struct hashfile *f, uint32_t nr_entries) * associated to pack_fd, and write that SHA1 at the end. That new SHA1 * is also returned in new_pack_sha1. * - * If partial_pack_sha1 is non null, then the SHA1 of the existing pack - * (without the header update) is computed and validated against the - * one provided in partial_pack_sha1. The validation is performed at - * partial_pack_offset bytes in the pack file. The SHA1 of the remaining - * data (i.e. from partial_pack_offset to the end) is then computed and - * returned in partial_pack_sha1. - * - * Note that new_pack_sha1 is updated last, so both new_pack_sha1 and - * partial_pack_sha1 can refer to the same buffer if the caller is not - * interested in the resulting SHA1 of pack data above partial_pack_offset. */ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo, int pack_fd, unsigned char *new_pack_hash, const char *pack_name, - uint32_t object_count, - unsigned char *partial_pack_hash, - off_t partial_pack_offset) + uint32_t object_count) { - int aligned_sz, buf_sz = 8 * 1024; - struct git_hash_ctx old_hash_ctx, new_hash_ctx; + int buf_sz = 8 * 1024; + struct git_hash_ctx new_hash_ctx; struct pack_header hdr; char *buf; ssize_t read_result; @@ -409,7 +397,6 @@ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo, */ hash_algo = unsafe_hash_algo(hash_algo); - git_hash_init(&old_hash_ctx, hash_algo); git_hash_init(&new_hash_ctx, hash_algo); if (lseek(pack_fd, 0, SEEK_SET) != 0) @@ -422,55 +409,19 @@ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo, pack_name); if (lseek(pack_fd, 0, SEEK_SET) != 0) die_errno("Failed seeking to start of '%s'", pack_name); - git_hash_update(&old_hash_ctx, &hdr, sizeof(hdr)); hdr.hdr_entries = htonl(object_count); git_hash_update(&new_hash_ctx, &hdr, sizeof(hdr)); write_or_die(pack_fd, &hdr, sizeof(hdr)); - partial_pack_offset -= sizeof(hdr); - buf = xmalloc(buf_sz); - aligned_sz = buf_sz - sizeof(hdr); for (;;) { - ssize_t m, n; - m = (partial_pack_hash && partial_pack_offset < aligned_sz) ? - partial_pack_offset : aligned_sz; - n = xread(pack_fd, buf, m); + ssize_t n = xread(pack_fd, buf, buf_sz); if (!n) break; if (n < 0) die_errno("Failed to checksum '%s'", pack_name); git_hash_update(&new_hash_ctx, buf, n); - - aligned_sz -= n; - if (!aligned_sz) - aligned_sz = buf_sz; - - if (!partial_pack_hash) - continue; - - git_hash_update(&old_hash_ctx, buf, n); - partial_pack_offset -= n; - if (partial_pack_offset == 0) { - unsigned char hash[GIT_MAX_RAWSZ]; - git_hash_final(hash, &old_hash_ctx); - if (!hasheq(hash, partial_pack_hash, - hash_algo)) - die("Unexpected checksum for %s " - "(disk corruption?)", pack_name); - /* - * Now let's compute the SHA1 of the remainder of the - * pack, which also means making partial_pack_offset - * big enough not to matter anymore. - */ - git_hash_init(&old_hash_ctx, hash_algo); - partial_pack_offset = ~partial_pack_offset; - partial_pack_offset -= MSB(partial_pack_offset, 1); - } } free(buf); - - if (partial_pack_hash) - git_hash_final(partial_pack_hash, &old_hash_ctx); git_hash_final(new_pack_hash, &new_hash_ctx); write_or_die(pack_fd, new_pack_hash, hash_algo->rawsz); fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name); diff --git a/pack.h b/pack.h index ada506b5c5d0db..f33a2667014f05 100644 --- a/pack.h +++ b/pack.h @@ -104,8 +104,7 @@ int verify_pack(struct repository *, struct packed_git *, verify_fn fn, void *fn struct progress *, uint32_t); off_t write_pack_header(struct hashfile *f, uint32_t); void fixup_pack_header_footer(const struct git_hash_algo *, int, - unsigned char *, const char *, uint32_t, - unsigned char *, off_t); + unsigned char *, const char *, uint32_t); char *index_pack_lockfile(struct odb_source *source, int fd, int *is_well_formed); From a316c5b30010682d3ef9369ee011cc748dce33c3 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Thu, 3 Sep 2026 00:39:03 -0400 Subject: [PATCH 148/156] index-pack, fast-import: skip redundant pack hashing Both index-pack's thin-pack fixup and fast-import always rewrite their pack headers with fixup_pack_header_footer(). That function has to read the complete pack from disk to compute the checksum of the updated data. But we also compute a checksum as we initially write the pack. Since the previous commit dropped the comparison with that old checksum, the result is not used at all. Set the hashfile's skip_hash flag in these two code paths to avoid computing it. With OPENSSL_SHA1_UNSAFE, the p5305 results are mostly within the noise: the thin-pack result is unchanged, while fast-import improves by 1.9%. But there is no reason to spend cycles computing a result we immediately throw away. --- builtin/fast-import.c | 1 + builtin/index-pack.c | 1 + 2 files changed, 2 insertions(+) diff --git a/builtin/fast-import.c b/builtin/fast-import.c index 77ed9b64494c40..a59b1af27553b3 100644 --- a/builtin/fast-import.c +++ b/builtin/fast-import.c @@ -796,6 +796,7 @@ static void start_packfile(void) p->do_not_close = 1; p->repo = the_repository; pack_file = hashfd(the_repository->hash_algo, pack_fd, p->pack_name); + pack_file->skip_hash = 1; pack_data = p; pack_size = write_pack_header(pack_file, 0); diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 02831fba6c0e06..10de95eff8f0e8 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1387,6 +1387,7 @@ static void conclude_pack(int fix_thin_pack, const char *curr_pack, unsigned cha memset(objects + nr_objects + 1, 0, nr_unresolved * sizeof(*objects)); f = hashfd(the_repository->hash_algo, output_fd, curr_pack); + f->skip_hash = 1; fix_unresolved_deltas(f); strbuf_addf(&msg, Q_("completed with %d local object", "completed with %d local objects", From d433fff3b7757f7e09c2aeda2463e0554ebb04fd Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Fri, 4 Sep 2026 07:58:00 -0700 Subject: [PATCH 149/156] ### match next From 8ff7c606d640e5bbc634439ebf1c9110b4aeb613 Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 2 Dec 2019 07:22:44 -0800 Subject: [PATCH 150/156] pack-objects: add window slot limit The delta window size of pack-objects is a somewhat crude tool for limiting CPU. For each window slot, we'll try an object against a candidate source via try_delta(). But that function may do a very small amount of work (if heuristics tell it that the two objects aren't likely to delta well) or a large amount (if we actually do a rolling hash over the candidate bytes to look for similarities). The difference can be particularly large when delta islands are in effect, since they will often trigger the "small work" case by immediately forbidding a delta. To compensate for this, it's useful to bump the window size when using delta islands, to get better results while (hopefully) still doing a similar amount of "large work". But this introduces a new problem: if the island restrictions _don't_ prevent a lot of deltas, then we may end up doing quite a lot of "large work" cases, spending a lot of CPU but not finding a lot of new deltas. Let's introduce a new knob for adjusting the window on the fly: we'll have a maximum window size as usual, but we'll stop looking through the window early if we hit more than a certain number of large-window cases. The new perf test simulates a delta-island situation. Here are the results on a clone of linux.git: Test origin HEAD ----------------------------------------------------------------------------------- 5305.2: window=10 504.82(1317.13+24.38) 491.19(1314.10+25.06) -2.7% 5305.3: pack size 1.5G 1.5G -0.0% 5305.4: window=100, slots=10 589.73(3789.62+17.86) 499.10(1415.41+25.53) -15.4% 5305.5: pack size 1.3G 1.5G +14.4% Not amazing. Without this feature (the "origin" column), bumping the window from 10 to 100 reduces the resulting pack from 1.5G to 1.3G, but increases the CPU time by almost a factor of 3 (these are multithreaded repacks, so look at the user CPU, the first number in the parentheses). The "slots" setting does nothing there, because "origin" doesn't yet support the feature. In the second column, we really do set slots=10, and we reclaim most of that CPU. But we also get a pack that's about the same size as we would have just by setting window=10 in the first place. Some real-world tests show better results, though. Here are comparisons of real from-scratch repacks of the whole GitHub fork-network of torvalds/linux using delta islands: name window-200 window-20 slots-20 cpu 66450.04 40487.01 (-39%) 47504.28 (-28%) size 1937481275 2695427807 (+39%) 2056879350 (+6%) The "window-200" column is a repack using window=200, which is what we normally do. The CPU row is total user CPU, and the size is the size of a clone of torvalds/linux using the resulting pack. The third column, slots-20 uses "window=200, slots=20". The resulting clone is only 6% bigger, but we saved 28% of the CPU during the repack. Compare that to just setting window=20, we spent a bit more CPU, but our clone size is much better. So we've actually accomplished something: a slider that puts us in the middle of window=200 and window=20. Here are similar numbers for the git/git network: name window-200 window-20 slots-20 cpu 82780.5 15301.26 (-81%) 15350.64 (-81%) size 146817734 215067328 (+46%) 161205555 (+9%) Here we do even better: we reclaim almost all of the CPU we could get with window=20, but our resulting pack is quite a bit smaller. And finally, on the extreme end, here's a real-world case where the repository was full of jpegs that don't delta, and had no delta islands at all: name window-200 window-20 slots-20 cpu 12577.52 2015.18 (-83%) 1953.42 (-84%) size 6211959217 6211960234 (+0%) 6211960234 (+0%) The pack size doesn't change no matter what we do, but we reclaim the CPU by using a smaller window. Now obviously you'd do about as well to just use window=20 here. But the general idea is that we don't know what the repository looks like ahead of time; by using a larger window=200 but limiting the work with slots=20, we perform reasonably well on repositories that _do_ have a lot of islands, without paying a penalty on the ones that don't. Signed-off-by: Jeff King --- Documentation/config/pack.adoc | 6 +++ Documentation/git-pack-objects.adoc | 13 ++++++ builtin/pack-objects.c | 17 +++++++- t/meson.build | 1 + t/perf/p5306-pack-limits.sh | 66 +++++++++++++++++++++++++++++ t/t5300-pack-object.sh | 8 ++++ 6 files changed, 110 insertions(+), 1 deletion(-) create mode 100755 t/perf/p5306-pack-limits.sh diff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc index 22384c2d2f0773..6f537762966915 100644 --- a/Documentation/config/pack.adoc +++ b/Documentation/config/pack.adoc @@ -14,6 +14,12 @@ pack.windowMemory:: suffixed with "k", "m", or "g". When left unconfigured (or set explicitly to 0), there will be no limit. +pack.windowSlotLimit:: + The maximum number of "expensive" delta candidate comparisons + that will be performed for each object. See the description of + `--window-slot-limit` in linkgit:git-pack-objects[1]. If + unconfigured, no limit is applied. + pack.compression:: An integer -1..9, indicating the compression level for objects in a pack file. -1 is the zlib default. 0 means no diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc index 65cd00c152f495..6f29e95f70fb8f 100644 --- a/Documentation/git-pack-objects.adoc +++ b/Documentation/git-pack-objects.adoc @@ -169,6 +169,19 @@ depth is 4095. `--window-memory=0` makes memory usage unlimited. The default is taken from the `pack.windowMemory` configuration variable. +--window-slot-limit=:: + This option provides an additional limit on top of `--window`. + Some delta candidates within the window can be rejected quickly + by heuristics, and some require comparing the bytes of the two + objects, which is expensive in CPU. This option limits us to + `` "expensive" comparisons for each object. When using delta + islands, we may cheaply reject many delta candidates due to + island restrictions. In that case it is useful to increase the + `--window` parameter higher (to look further for delta + candidates) but use `--window-slot-limit` to avoid spending too + much CPU. The default is taken from the `pack.windowSlotLimit` + configuration variable. + --max-pack-size=:: In unusual scenarios, you may not be able to create files larger than a certain size on your filesystem, and this option diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 31069830e48581..9b5f71eb22dad5 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -266,6 +266,7 @@ static size_t max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE; static unsigned long cache_max_small_delta_size = 1000; static unsigned long window_memory_limit = 0; +static int window_slot_limit; static struct string_list uri_protocols = STRING_LIST_INIT_NODUP; @@ -3015,6 +3016,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, struct object_entry *entry; struct unpacked *n = array + idx; int j, max_depth, best_base = -1; + int slow_count = 0; progress_lock(); if (!*list_size) { @@ -3071,8 +3073,15 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, ret = try_delta(n, m, max_depth, &mem_usage); if (ret == TRY_DELTA_STOP) break; - else if (ret == TRY_DELTA_FOUND) + else if (ret == TRY_DELTA_FOUND) { best_base = other_idx; + slow_count++; + } else if (ret == TRY_DELTA_NONE_SLOW) { + slow_count++; + } + if (window_slot_limit && + window_slot_limit <= slow_count) + break; } /* @@ -3712,6 +3721,10 @@ static int git_pack_config(const char *k, const char *v, window_memory_limit = git_config_ulong(k, v, ctx->kvi); return 0; } + if (!strcmp(k, "pack.windowslotlimit")) { + window_slot_limit = git_config_int(k, v, ctx->kvi); + return 0; + } if (!strcmp(k, "pack.depth")) { depth = git_config_int(k, v, ctx->kvi); return 0; @@ -5173,6 +5186,8 @@ int cmd_pack_objects(int argc, N_("limit pack window by objects")), OPT_UNSIGNED(0, "window-memory", &window_memory_limit, N_("limit pack window by memory in addition to object limit")), + OPT_INTEGER(0, "window-slot-limit", &window_slot_limit, + N_("limit delta attempts within window by number of objects")), OPT_INTEGER(0, "depth", &depth, N_("maximum length of delta chain allowed in the resulting pack")), OPT_BOOL(0, "reuse-delta", &reuse_delta, diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..e796cb0050b15b 100644 --- a/t/meson.build +++ b/t/meson.build @@ -1166,6 +1166,7 @@ benchmarks = [ 'perf/p4221-log-grep-engines-fixed.sh', 'perf/p5302-pack-index.sh', 'perf/p5303-many-packs.sh', + 'perf/p5305-pack-limits.sh', 'perf/p5304-prune.sh', 'perf/p5310-pack-bitmaps.sh', 'perf/p5311-pack-bitmaps-fetch.sh', diff --git a/t/perf/p5306-pack-limits.sh b/t/perf/p5306-pack-limits.sh new file mode 100755 index 00000000000000..bb5d73fc3885f5 --- /dev/null +++ b/t/perf/p5306-pack-limits.sh @@ -0,0 +1,66 @@ +#!/bin/sh + +test_description='packing with adjustable in-window limits' +. ./perf-lib.sh + +test_perf_default_repo + +# The most interesting case for adjustable windows is when our heuristics often +# tell us we can skip a delta candidate without even trying to compute it. And +# a likely reason for that is that delta islands forbid the pairing. +# +# We can simulate an island setup where each island was "forked" at some time +# in the past from the main history (by just taking every 100th first-parent +# head). We'll clear out any extraneous refs and just give each fork a single +# copy of HEAD (at the time of forking) to keep things simple. +test_expect_success 'set up delta islands' ' + head=$(git rev-parse HEAD) && + git for-each-ref --format="delete %(refname)" | + git update-ref --no-deref --stdin && + + n=0 && + fork=0 && + git rev-list --first-parent $head | + while read commit; do + n=$((n+1)) && + if test "$n" = 100; then + echo "create refs/forks/$fork/master $commit" && + fork=$((fork+1)) && + n=0 || + return 1 + fi + done | + git update-ref --stdin && + + git config pack.island "refs/forks/([0-9]*)/" +' + +# simulate a "repack -aif" without actually changing the on-disk state; we'll +# use no-reuse-delta because we want to see the effect on the delta search +pack_opts='--all --no-reuse-delta --delta-base-offset --delta-islands' +export pack_opts + +test_perf 'window=10' ' + rm -f pack-*.pack && + git pack-objects $pack_opts --window=10 pack Date: Mon, 2 Dec 2019 07:26:05 -0800 Subject: [PATCH 151/156] pack-objects: add window byte limit The previous commit introduced a "window slot limit" that would try to avoid doing too many "expensive" delta attempts within the window. But "expensive" is relative: looking for deltas in a 1-kilobyte file is much cheaper than looking for ones in a 1-megabyte file. Instead of adjusting the delta search based on the number of slots, let's instead provide the ability to limit based on the number of bytes examined in each attempt. We can base this on the number of "slow" slots, since we're really just measuring the effort to walk over the target object in each slot. One could argue that we should be counting the sum of the base and the target for each candidate, but the base effort is all in creating the index, and is amortized over many targets. Here are numbers for p5305 on a clone of linux.git: Test origin HEAD ----------------------------------------------------------------------------------- 5305.2: window=10 504.82(1317.13+24.38) 491.19(1314.10+25.06) -2.7% 5305.3: pack size 1.5G 1.5G -0.0% 5305.4: window=100, slots=10 589.73(3789.62+17.86) 499.10(1415.41+25.53) -15.4% 5305.5: pack size 1.3G 1.5G +14.4% 5305.6: window=100, bytes=1m 589.73(3789.62+17.86) 572.52(3321.29+19.38) -2.9% 5305.7: pack size 1.3G 1.4G +3.6% Compared to reducing the slot size, using a 1-megabyte limit means we still end up doing quite a few comparisons (note that we save a much smaller amount of CPU compared to a true 100-window repack, but our increase in pack size is much better). So it's arguable which of the two you'd want to use in this simulated case; they're really just moving the slider different amounts between the window=10 and window=100 cases. But let's look at some real-world instances. Here's the GitHub fork network of torvalds/linux again, this time with some byte limits: name window-200 window-20 slots-20 bytes-10m cpu 66450.04 40487.01 (-39%) 47504.28 (-28%) 45344.89 (-31%) size 1937481275 2695427807 (+39%) 2056879350 (+6%) 1939474673 (+0%) We managed to reduce the CPU for the repack by 31% without making any significant increase in the resulting clone size! It's better in every metric than the slots-20 run. Here are benchmarks with a few more byte values: name window-200 bytes-10m bytes-1m bytes-500k cpu 66450.04 45344.89 (-31%) 46210.42 (-30%) 36448.09 (-45%) size 1937481275 1939474673 (+0%) 1998607712 (+3%) 2091254047 (+7%) We can see the cpu versus size tradeoff as we use smaller limits. Somewhere between 1m and 10m seems pretty optimal for this particular repository. Let's look at the git/git network: name window-200 window-20 slots-20 bytes-1m cpu 82780.5 15301.26 (-81%) 15350.64 (-81%) 25172.65 (-69%) size 146817734 215067328 (+46%) 161205555 (+9%) 156297226 (+6%) Here again we spend a bit more CPU than the slots-20 strategy, but we yield a slightly better packfile. Notice this used a 1-megabyte limit. Here's the same repository with a few different byte limits: name window-200 bytes-10m bytes-1m bytes-500k cpu 82780.5 87759.18 (+6%) 25172.65 (-69%) 25435.35 (-69%) size 146817734 146848537 (+0%) 156297226 (+6%) 173301406 (+18%) The 10m limit is clearly too much. The packfile is good, but we spent even more CPU than without the feature (there's a fair bit of run-to-run noise, so in reality it's probably about the same). And 500k doesn't buy us much CPU, but gives us a much worse pack. And finally, here's that same repository full of jpegs that don't delta: name window-200 window-20 slots-20 bytes-1m bytes-500k cpu 12577.52 2015.18 (-83%) 1953.42 (-84%) 294.03 (-97%) 211.55 (-98%) size 6211959217 6211960234 (+0%) 6211960234 (+0%) 6211961945 (+0%) 6211959206 (+0%) Wow! We get the same pack each time, but now we're using a tenth of the CPU we'd use even for the small window size. What's going on here? This is an extreme case: the files are big but don't delta well. So we're hitting our byte limit almost immediately, and this is behaving more like a window size of 1 for the blobs, but still finding decent deltas for smaller blobs and trees. So while the results are impressive, this repository is atypical. And there's a mirror-image pathological case: a repository full of large blobs that _do_ delta well, where we spend very little time looking for deltas and the pack size ends up much worse. So this new knob can't magically solve all problems, but it does seem to improve things for most cases. Signed-off-by: Jeff King --- Documentation/config/pack.adoc | 6 ++++++ Documentation/git-pack-objects.adoc | 10 ++++++++++ builtin/pack-objects.c | 10 ++++++++++ t/perf/p5306-pack-limits.sh | 11 +++++++++++ t/t5300-pack-object.sh | 4 ++++ 5 files changed, 41 insertions(+) diff --git a/Documentation/config/pack.adoc b/Documentation/config/pack.adoc index 6f537762966915..1da696234564cc 100644 --- a/Documentation/config/pack.adoc +++ b/Documentation/config/pack.adoc @@ -20,6 +20,12 @@ pack.windowSlotLimit:: `--window-slot-limit` in linkgit:git-pack-objects[1]. If unconfigured, no limit is applied. +pack.windowByteLimit:: + Same as `pack.windowSlotLimit`, but scaled by the number of + bytes in each candidate object. See the description of + `--window-byte-limit` in linkgit:git-pack-objects[1]. If + unconfigured, no limit is applied. + pack.compression:: An integer -1..9, indicating the compression level for objects in a pack file. -1 is the zlib default. 0 means no diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc index 6f29e95f70fb8f..3b3511e0a55989 100644 --- a/Documentation/git-pack-objects.adoc +++ b/Documentation/git-pack-objects.adoc @@ -182,6 +182,16 @@ depth is 4095. much CPU. The default is taken from the `pack.windowSlotLimit` configuration variable. +--window-byte-limit=:: + Provides a similar limit to `--window-slot-limit`, but scaled by + the number of bytes in each candidate object. For example, a + window byte limit of one megabyte would be equivalent to a slot + limit of 10 for an object of 100 kilobytes, or 100 for an object + of 10 kilobytes. This can provide better results (less CPU + and/or smaller packs) than a slot limit. The size can be + suffixed with "k", "m", or "g" for kilobytes, etc. The default + is taken from the `pack.windowByteLimit` configuration variable. + --max-pack-size=:: In unusual scenarios, you may not be able to create files larger than a certain size on your filesystem, and this option diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 9b5f71eb22dad5..1f8914c375e0e9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -267,6 +267,7 @@ static unsigned long cache_max_small_delta_size = 1000; static unsigned long window_memory_limit = 0; static int window_slot_limit; +static unsigned long window_byte_limit; static struct string_list uri_protocols = STRING_LIST_INIT_NODUP; @@ -3082,6 +3083,9 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, if (window_slot_limit && window_slot_limit <= slow_count) break; + if (window_byte_limit && + window_byte_limit <= SIZE(n->entry) * slow_count) + break; } /* @@ -3725,6 +3729,10 @@ static int git_pack_config(const char *k, const char *v, window_slot_limit = git_config_int(k, v, ctx->kvi); return 0; } + if (!strcmp(k, "pack.windowbytelimit")) { + window_byte_limit = git_config_ulong(k, v, ctx->kvi); + return 0; + } if (!strcmp(k, "pack.depth")) { depth = git_config_int(k, v, ctx->kvi); return 0; @@ -5188,6 +5196,8 @@ int cmd_pack_objects(int argc, N_("limit pack window by memory in addition to object limit")), OPT_INTEGER(0, "window-slot-limit", &window_slot_limit, N_("limit delta attempts within window by number of objects")), + OPT_UNSIGNED(0, "window-byte-limit", &window_byte_limit, + N_("limit delta attempts within window by bytes")), OPT_INTEGER(0, "depth", &depth, N_("maximum length of delta chain allowed in the resulting pack")), OPT_BOOL(0, "reuse-delta", &reuse_delta, diff --git a/t/perf/p5306-pack-limits.sh b/t/perf/p5306-pack-limits.sh index bb5d73fc3885f5..4c5c04b8d80ab2 100755 --- a/t/perf/p5306-pack-limits.sh +++ b/t/perf/p5306-pack-limits.sh @@ -63,4 +63,15 @@ test_size 'pack size' ' wc -c <$pack ' +test_perf 'window=100, bytes=1m' ' + rm -f pack-*.pack && + git -c pack.windowbytelimit=1m \ + pack-objects $pack_opts --window=100 pack Date: Mon, 2 Dec 2019 07:00:16 -0800 Subject: [PATCH 152/156] pack-objects: trace adjustable window stats for each object This trace outputs the slot and byte numbers at which we found the delta candidate that we end up using in the packfile output. If run without any pack.windowSlotLimit or pack.windowByteLimit, that can give a sense of what good values are for those config options (at least for this repository). Running a full window=200 repack of the torvalds/linux fork network with GIT_TRACE_DELTA yields a 3GB trace file, from which we can gather some percentile statistics on the slot numbers: p50: 1 p75: 2 p90: 8 p98: 62 p99: 99 max: 200 So we usually find our best candidate quite early on. Half the time it's the first one we try, and 90% of the time it's in the first 8. Of course there's a tail of cases that require going to 100 or further, but those are the minority. That explains why limiting to slots=20 worked reasonably well. And here's the same run for byte statistics: p50: 3462 p75: 16464 p90: 97644 p98: 708408 p99: 1592577 max: 265848594 Setting the byte limit at 1m means we'll find the same candidates in 98-99% of cases. But of course the more bytes we look through to find candidates, the bigger the object, and thus the more we have to gain by actually finding a delta. So even though our limit excludes less than 2% of the deltas we'd otherwise found, we still saw our resulting pack size increase by several percent. Signed-off-by: Jeff King --- builtin/pack-objects.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 1f8914c375e0e9..b693946e53a7ca 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -38,6 +38,7 @@ #include "replace-object.h" #include "dir.h" #include "midx.h" +#include "trace.h" #include "trace2.h" #include "shallow.h" #include "promisor-remote.h" @@ -189,6 +190,8 @@ static inline void oe_set_delta_size(struct packing_data *pack, #define SET_DELTA_CHILD(obj, val) oe_set_delta_child(&to_pack, obj, val) #define SET_DELTA_SIBLING(obj, val) oe_set_delta_sibling(&to_pack, obj, val) +static struct trace_key trace_delta = TRACE_KEY_INIT(DELTA); + static const char *const pack_usage[] = { N_("git pack-objects [-q | --progress | --all-progress] [--all-progress-implied]\n" " [--no-reuse-delta] [--delta-base-offset] [--non-empty]\n" @@ -3018,6 +3021,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, struct unpacked *n = array + idx; int j, max_depth, best_base = -1; int slow_count = 0; + int slow_count_at_best_base; progress_lock(); if (!*list_size) { @@ -3077,6 +3081,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, else if (ret == TRY_DELTA_FOUND) { best_base = other_idx; slow_count++; + slow_count_at_best_base = slow_count; } else if (ret == TRY_DELTA_NONE_SLOW) { slow_count++; } @@ -3088,6 +3093,14 @@ static void find_deltas(struct object_entry **list, unsigned *list_size, break; } + if (best_base >= 0) { + trace_printf_key(&trace_delta, + "best base for %s found at slot %d, byte %"PRIuMAX, + oid_to_hex(&entry->idx.oid), + slow_count_at_best_base, + (uintmax_t)(SIZE(n->entry) * slow_count_at_best_base)); + } + /* * If we decided to cache the delta data, then it is best * to compress it right away. First because we have to do From 15a803259fcd5cfde939c4db64ce8ad9a75ba3fd Mon Sep 17 00:00:00 2001 From: Jeff King Date: Mon, 24 Aug 2026 22:58:39 -0400 Subject: [PATCH 153/156] merge-fixup jk/object-size-hooks Code moved to object-file.c (and we need to pass size via opts). --- builtin/receive-pack.c | 1 + object-file.c | 3 +++ odb/transaction.h | 6 ++++++ 3 files changed, 10 insertions(+) diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index 9b9891332f6ce0..ab69a9df5ae11f 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -2308,6 +2308,7 @@ static int unpack_with_sideband(struct odb_transaction *transaction, : 0), .fsck_msg_types = fsck_msg_types.buf, .max_input_size = max_input_size, + .warn_object_size = warn_object_size, .shallow_file = shallow_file, .reject_thin = reject_thin, .quiet = quiet, diff --git a/object-file.c b/object-file.c index 694f3e9bc82b94..e3d501031c71a1 100644 --- a/object-file.c +++ b/object-file.c @@ -1339,6 +1339,9 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base, if (opts->max_input_size) strvec_pushf(&child.args, "--max-input-size=%"PRIuMAX, (uintmax_t)opts->max_input_size); + if (opts->warn_object_size) + strvec_pushf(&child.args, "--warn-object-size=%"PRIuMAX, + (uintmax_t)opts->warn_object_size); child.out = -1; child.in = pack_fd; child.err = err_fd; diff --git a/odb/transaction.h b/odb/transaction.h index 2a40c85bc819c8..0b55e472837bdf 100644 --- a/odb/transaction.h +++ b/odb/transaction.h @@ -26,6 +26,12 @@ struct odb_transaction_write_pack_opts { */ off_t max_input_size; + /* + * Like max_input_size, but do not abort; just write the names into a + * file which can be interpreted by hooks. + */ + off_t warn_object_size; + /* * Whether the validity of incoming objects should be verified. */ From 22ac833d233b5f6b90d4f6c5ef6a3311143164ba Mon Sep 17 00:00:00 2001 From: Jeff King Date: Sun, 24 May 2026 00:48:43 -0400 Subject: [PATCH 154/156] merge-fixup jk/string-list-cleanups This is due to an interaction with my jk/mailmap-trailers topic. --- pretty.c | 3 +-- trailer.c | 2 +- trailer.h | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/pretty.c b/pretty.c index c9e5b790f19c96..5ee120b7fe58ef 100644 --- a/pretty.c +++ b/pretty.c @@ -1341,8 +1341,7 @@ int format_set_trailers_options(struct process_trailer_options *opts, opts->key_value_separator = expand_string_arg(kvsepbuf, argval, arglen); } else if (match_placeholder_bool_arg(*arg, "mailmap", arg, &bool_arg)) { if (bool_arg) { - /* yuck but this is how mailmap_name() above does it */ - static struct string_list mailmap = STRING_LIST_INIT_DUP; + static struct mailmap mailmap = MAILMAP_INIT; read_mailmap(the_repository, &mailmap); opts->mailmap = &mailmap; } else { diff --git a/trailer.c b/trailer.c index 66c0199b10b6cc..46d3af19d097ec 100644 --- a/trailer.c +++ b/trailer.c @@ -1140,7 +1140,7 @@ void trailer_block_release(struct trailer_block *trailer_block) free(trailer_block); } -static int mailmap_value(struct string_list *mailmap, +static int mailmap_value(struct mailmap *mailmap, struct strbuf *out, const struct strbuf *in) { const char *mailbuf, *namebuf; diff --git a/trailer.h b/trailer.h index b76254630cbd6f..99b28103af1378 100644 --- a/trailer.h +++ b/trailer.h @@ -55,7 +55,7 @@ struct process_trailer_options { int no_divider; int key_only; int value_only; - struct string_list *mailmap; + struct mailmap *mailmap; const struct strbuf *separator; const struct strbuf *key_value_separator; int (*filter)(const struct strbuf *, void *); From d0aac127adbe5b583d19a4725dba672fe13de5aa Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 5 Sep 2026 13:14:10 +0000 Subject: [PATCH 155/156] Initial plan From 719882e6881416bc784f68616ae30ed4bb12c295 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 5 Sep 2026 13:15:40 +0000 Subject: [PATCH 156/156] Remove expired Debian 11 CI job Co-authored-by: peff <45925+peff@users.noreply.github.com> --- .github/workflows/main.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 6c08d58706998f..828207c951c16e 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -421,9 +421,6 @@ jobs: # A RHEL 8 compatible distro. Supported until 2029-05-31. - jobname: almalinux-8 image: almalinux:8 - # Supported until 2026-08-31. - - jobname: debian-11 - image: debian:11 env: jobname: ${{matrix.vector.jobname}} CC: ${{matrix.vector.cc}}