"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> The fill_packs_from_midx() method was refactored in fcb2205b77 (midx:
> implement support for writing incremental MIDX chains, 2024-08-06) to
> allow for preferred packfiles and incremental multi-pack-indexes.
> However, this led to some conditions that can cause improperly
> initialized memory in the context's list of packfiles.
>
> The conditions caring about the preferred pack name or the incremental
> flag are currently necessary to load a packfile. But the context is
> still being populated with pack_info structs based on the packfile array
> for the existing multi-pack-index even if prepare_midx_pack() isn't
> called.
>
> Add a new test that breaks under --stress when compiled with
> SANITIZE=address. The chosen number of 100 packfiles was selected to get
> the --stress output to fail about 50% of the time, while 50 packfiles
> could not get a failure in most --stress runs. This test has a very
> minor check at the end confirming only one packfile remaining. The
> failing nature of this test actually relies on auto-GC cleaning up some
> packfiles during the creation of the commits, as tests setting gc.auto
> to zero make the packfile count match the number of added commits but
> also avoids hitting the memory issue.
>
> The test case is marked as EXPENSIVE not only because of the number of
> packfiles it creates, but because some CI environments were reporting
> errors during the test that I could not reproduce, specifically around
> being unable to open the packfiles or their pack-indexes.
>
> When it fails under SANITIZE=address, it provides the following error:
>
> AddressSanitizer:DEADLYSIGNAL
> =================================================================
> ==3263517==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000027
> ==3263517==The signal is caused by a READ memory access.
> ==3263517==Hint: address points to the zero page.
>     #0 0x562d5d82d1fb in close_pack_windows packfile.c:299
>     #1 0x562d5d82d3ab in close_pack packfile.c:354
>     #2 0x562d5d7bfdb4 in write_midx_internal midx-write.c:1490
>     #3 0x562d5d7c7aec in midx_repack midx-write.c:1795
>     #4 0x562d5d46fff6 in cmd_multi_pack_index builtin/multi-pack-index.c:305
>     ...
>
> This failure stack trace is disconnected from the real fix because it
> the bad pointers are accessed later when closing the packfiles from the
> context.

"because it the bad pointers" -> ???  Perhaps just drop "it"?

> There are a few different aspects to this fix that are worth noting:
>
>  1. We return to the previous behavior of fill_packs_from_midx to not
>     rely on the incremental flag or existence of a preferred pack.
>
>  2. The behavior to scan all layers of an incremental midx is kept, so
>     this is not a full revert of the change.
>
>  3. We skip allocating more room in the pack_info array if the pack
>     fails prepare_midx_pack().
>
>  4. The method has always returned 0 for success and 1 for failure, but
>     the condition checking for error added a check for a negative result
>     for failure, so that is now updated.

So did the callee think it is signalling an error, but the sole
caller was not taking that as an error, and that was one of the bugs
this change fixes?

Even if that is the case, I still do not understand why we want to
say in the callee

	error("message");
	return 1;

and adjust the caller to it, when we can just say

	return error("message");

in the callee(), especially if the only caller is expecting to be
signalled by a negative return value for an error already.

>  5. The call to open_pack_index() is removed, but this is needed later
>     in the case of a preferred pack. That call is moved to immediately
>     before its result is needed (checking for the object count).
>
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c                | 38 ++++++++++++-------------------------
>  t/t5319-multi-pack-index.sh | 17 +++++++++++++++++
>  2 files changed, 29 insertions(+), 26 deletions(-)

Thanks.

> diff --git a/midx-write.c b/midx-write.c
> index a0aceab5e0..d8f9679868 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -920,8 +920,7 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,
>  	return get_multi_pack_index(source);
>  }
>  
> -static int fill_packs_from_midx(struct write_midx_context *ctx,
> -				const char *preferred_pack_name, uint32_t flags)
> +static int fill_packs_from_midx(struct write_midx_context *ctx)
>  {
>  	struct multi_pack_index *m;
>  
> @@ -929,30 +928,13 @@ static int fill_packs_from_midx(struct write_midx_context *ctx,
>  		uint32_t i;
>  
>  		for (i = 0; i < m->num_packs; i++) {
> -			ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);
> -
> -			/*
> -			 * If generating a reverse index, need to have
> -			 * packed_git's loaded to compare their
> -			 * mtimes and object count.
> -			 *
> -			 * If a preferred pack is specified, need to
> -			 * have packed_git's loaded to ensure the chosen
> -			 * preferred pack has a non-zero object count.
> -			 */
> -			if (flags & MIDX_WRITE_REV_INDEX ||
> -			    preferred_pack_name) {
> -				if (prepare_midx_pack(ctx->repo, m,
> -						      m->num_packs_in_base + i)) {
> -					error(_("could not load pack"));
> -					return 1;
> -				}
> -
> -				if (open_pack_index(m->packs[i]))
> -					die(_("could not open index for %s"),
> -					    m->packs[i]->pack_name);
> +			if (prepare_midx_pack(ctx->repo, m,
> +					      m->num_packs_in_base + i)) {
> +				error(_("could not load pack"));
> +				return 1;
>  			}
>  
> +			ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);
>  			fill_pack_info(&ctx->info[ctx->nr++], m->packs[i],
>  				       m->pack_names[i],
>  				       m->num_packs_in_base + i);
> @@ -1123,8 +1105,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  			ctx.num_multi_pack_indexes_before++;
>  			m = m->base_midx;
>  		}
> -	} else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,
> -						 flags) < 0) {
> +	} else if (ctx.m && fill_packs_from_midx(&ctx)) {
>  		goto cleanup;
>  	}
>  
> @@ -1223,6 +1204,11 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  
>  	if (ctx.preferred_pack_idx > -1) {
>  		struct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;
> +
> +		if (open_pack_index(preferred))
> +			die(_("failed to open preferred pack %s"),
> +			    ctx.info[ctx.preferred_pack_idx].pack_name);
> +
>  		if (!preferred->num_objects) {
>  			error(_("cannot select preferred pack %s with no objects"),
>  			      preferred->pack_name);
> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
> index bd75dea950..49705c62a2 100755
> --- a/t/t5319-multi-pack-index.sh
> +++ b/t/t5319-multi-pack-index.sh
> @@ -989,6 +989,23 @@ test_expect_success 'repack --batch-size=0 repacks everything' '
>  	)
>  '
>  
> +test_expect_success EXPENSIVE 'repack/expire with many packs' '
> +	cp -r dup many &&
> +	(
> +		cd many &&
> +
> +		for i in $(test_seq 1 100)
> +		do
> +			test_commit extra$i &&
> +			git maintenance run --task=loose-objects || return 1
> +		done &&
> +
> +		git multi-pack-index write &&
> +		git multi-pack-index repack &&
> +		git multi-pack-index expire
> +	)
> +'
> +
>  test_expect_success 'repack --batch-size=<large> repacks everything' '
>  	(
>  		cd dup2 &&

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <stolee@gmail.com>
>
> This instance of setting the result to 1 before going to cleanup was
> accidentally removed in fcb2205b77 (midx: implement support for writing
> incremental MIDX chains, 2024-08-06).
>
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c | 1 +
>  1 file changed, 1 insertion(+)

The cover letter made it sound as if [1/5] was the only fix and the
rest was clean-up, but unless all callers of write_midx_internal()
ignores the return value from it, this surely would change the
behaviour of the program, no?

And the results from write-midx_file_only(), write_midx_file(), and
expire_midx_packs(), the three callers of this _internal() function,
all seem to be used in builtin/multi-pack-index.c so wouldn't this
also be a fix?

Not that I endorse "0 is success and any non-zero value is an error",
but this does not look like a mere clean-up to me.

> diff --git a/midx-write.c b/midx-write.c
> index d8f9679868..85b2d471ef 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1106,6 +1106,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  			m = m->base_midx;
>  		}
>  	} else if (ctx.m && fill_packs_from_midx(&ctx)) {
> +		result = 1;
>  		goto cleanup;
>  	}

On Thu, Aug 28, 2025 at 01:45:36PM -0700, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > From: Derrick Stolee <stolee@gmail.com>
> >
> > This instance of setting the result to 1 before going to cleanup was
> > accidentally removed in fcb2205b77 (midx: implement support for writing
> > incremental MIDX chains, 2024-08-06).
> >
> > Signed-off-by: Derrick Stolee <stolee@gmail.com>
> > ---
> >  midx-write.c | 1 +
> >  1 file changed, 1 insertion(+)
>
> The cover letter made it sound as if [1/5] was the only fix and the
> rest was clean-up, but unless all callers of write_midx_internal()
> ignores the return value from it, this surely would change the
> behaviour of the program, no?

I think the cover letter is saying that only the first patch is required
to fix the crash that was reported, and the rest are clean-ups.

Not that this isn't a bug in its own right, but it's definitely distinct
from the one that is addressed in the previous patch.

> And the results from write-midx_file_only(), write_midx_file(), and
> expire_midx_packs(), the three callers of this _internal() function,
> all seem to be used in builtin/multi-pack-index.c so wouldn't this
> also be a fix?
>
> Not that I endorse "0 is success and any non-zero value is an error",
> but this does not look like a mere clean-up to me.
>
> > diff --git a/midx-write.c b/midx-write.c
> > index d8f9679868..85b2d471ef 100644
> > --- a/midx-write.c
> > +++ b/midx-write.c
> > @@ -1106,6 +1106,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
> >  			m = m->base_midx;
> >  		}
> >  	} else if (ctx.m && fill_packs_from_midx(&ctx)) {
> > +		result = 1;
> >  		goto cleanup;

It might be nice to have a test that confirms this behavior (though the
changes look obviously correct to me). It should suffice to write a
MIDX, drop one of its packs, and then try to write it again with a new
pack.

Thanks,
Taylor

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Derrick Stolee <stolee@gmail.com>
>
> The incremental mode of writing a multi-pack-index has a few extra
> conditions that could lead to failure, but these are currently
> short-ciruiting with 'return -1' instead of setting the method's
> 'result' variable and going to the cleanup tag.
>
> Replace these returns with gotos to avoid memory issues when exiting
> early due to error conditions.
>
> Unfortunately, these error conditions are difficult to reproduce with
> test cases, which is perhaps one reason why the memory loss was not
> caught by existing test cases in memory tracking modes.
>
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c | 18 ++++++++++++------
>  1 file changed, 12 insertions(+), 6 deletions(-)

Good thinking, but may I suggest us to go one more step to adopt
even better convention if we were to do this?

Pessimistically initialize the "result" to -1 and let many "goto
cleanup" just jump there.  And have "result = 0" just before the
cleanup label where the success code path joins the final cleanup
part of the function.

This is often the right way to make the flow easier to see, because
often the success code path is straight forward, and these error
conditions are what employ the "goto cleanup" from many places.  By
starting pessimistic, and declaring the success at the very end of
the straight-forward success case code path, all other flows to the
clean-up labels do not have to set the "ah I failed" flag.  It would
eliminate the need for patches like the previous step if the
original were following that pattern.

> diff --git a/midx-write.c b/midx-write.c
> index 85b2d471ef..f2d9a990e6 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1321,13 +1321,15 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  		incr = mks_tempfile_m(midx_name.buf, 0444);
>  		if (!incr) {
>  			error(_("unable to create temporary MIDX layer"));
> -			return -1;
> +			result = -1;
> +			goto cleanup;
>  		}
>  
>  		if (adjust_shared_perm(r, get_tempfile_path(incr))) {
>  			error(_("unable to adjust shared permissions for '%s'"),
>  			      get_tempfile_path(incr));
> -			return -1;
> +			result = -1;
> +			goto cleanup;
>  		}
>  
>  		f = hashfd(r->hash_algo, get_tempfile_fd(incr),
> @@ -1427,18 +1429,22 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  
>  		if (!chainf) {
>  			error_errno(_("unable to open multi-pack-index chain file"));
> -			return -1;
> +			result = -1;
> +			goto cleanup;
>  		}
>  
> -		if (link_midx_to_chain(ctx.base_midx) < 0)
> -			return -1;
> +		if (link_midx_to_chain(ctx.base_midx) < 0) {
> +			result = -1;
> +			goto cleanup;
> +		}
>  
>  		get_split_midx_filename_ext(r->hash_algo, &final_midx_name,
>  					    object_dir, midx_hash, MIDX_EXT_MIDX);
>  
>  		if (rename_tempfile(&incr, final_midx_name.buf) < 0) {
>  			error_errno(_("unable to rename new multi-pack-index layer"));
> -			return -1;
> +			result = -1;
> +			goto cleanup;
>  		}
>  
>  		strbuf_release(&final_midx_name);

On Thu, Aug 28, 2025 at 01:51:18PM -0700, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > From: Derrick Stolee <stolee@gmail.com>
> >
> > The incremental mode of writing a multi-pack-index has a few extra
> > conditions that could lead to failure, but these are currently
> > short-ciruiting with 'return -1' instead of setting the method's
> > 'result' variable and going to the cleanup tag.
> >
> > Replace these returns with gotos to avoid memory issues when exiting
> > early due to error conditions.
> >
> > Unfortunately, these error conditions are difficult to reproduce with
> > test cases, which is perhaps one reason why the memory loss was not
> > caught by existing test cases in memory tracking modes.
> >
> > Signed-off-by: Derrick Stolee <stolee@gmail.com>
> > ---
> >  midx-write.c | 18 ++++++++++++------
> >  1 file changed, 12 insertions(+), 6 deletions(-)
>
> Good thinking, but may I suggest us to go one more step to adopt
> even better convention if we were to do this?
>
> Pessimistically initialize the "result" to -1 and let many "goto
> cleanup" just jump there.  And have "result = 0" just before the
> cleanup label where the success code path joins the final cleanup
> part of the function.
>
> This is often the right way to make the flow easier to see, because
> often the success code path is straight forward, and these error
> conditions are what employ the "goto cleanup" from many places.  By
> starting pessimistic, and declaring the success at the very end of
> the straight-forward success case code path, all other flows to the
> clean-up labels do not have to set the "ah I failed" flag.  It would
> eliminate the need for patches like the previous step if the
> original were following that pattern.

Alternatively replacing something like:

    error(...);
    result = -1;
    goto cleanup;

with just

    result = error(...);
    goto cleanup;

would IMHO make the code easier to read, though I agree that nothing is
forcing us to remember to assign result in the first place ;-). I am not
sure the pessimistic initialization is better in all cases either, since
we have to remember to place it before any "cleanup" label, and make
sure that that does not regress.

So, I dunno. I'm OK with what is written here, and I think we could
certainly have a separate discussion to perhaps have CodingGuidelines
take a stronger stance here.

Thanks,
Taylor

On 8/28/25 9:29 PM, Taylor Blau wrote:
> On Thu, Aug 28, 2025 at 01:51:18PM -0700, Junio C Hamano wrote:
>> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>>
>>> From: Derrick Stolee <stolee@gmail.com>
>>>
>>> The incremental mode of writing a multi-pack-index has a few extra
>>> conditions that could lead to failure, but these are currently
>>> short-ciruiting with 'return -1' instead of setting the method's
>>> 'result' variable and going to the cleanup tag.
>>>
>>> Replace these returns with gotos to avoid memory issues when exiting
>>> early due to error conditions.
>>>
>>> Unfortunately, these error conditions are difficult to reproduce with
>>> test cases, which is perhaps one reason why the memory loss was not
>>> caught by existing test cases in memory tracking modes.
>>>
>>> Signed-off-by: Derrick Stolee <stolee@gmail.com>
>>> ---
>>>   midx-write.c | 18 ++++++++++++------
>>>   1 file changed, 12 insertions(+), 6 deletions(-)
>>
>> Good thinking, but may I suggest us to go one more step to adopt
>> even better convention if we were to do this?
>>
>> Pessimistically initialize the "result" to -1 and let many "goto
>> cleanup" just jump there.  And have "result = 0" just before the
>> cleanup label where the success code path joins the final cleanup
>> part of the function.
>>
>> This is often the right way to make the flow easier to see, because
>> often the success code path is straight forward, and these error
>> conditions are what employ the "goto cleanup" from many places.  By
>> starting pessimistic, and declaring the success at the very end of
>> the straight-forward success case code path, all other flows to the
>> clean-up labels do not have to set the "ah I failed" flag.  It would
>> eliminate the need for patches like the previous step if the
>> original were following that pattern.
> > Alternatively replacing something like:
> >      error(...);
>      result = -1;
>      goto cleanup;
> > with just
> >      result = error(...);
>      goto cleanup;
> > would IMHO make the code easier to read, though I agree that nothing is
> forcing us to remember to assign result in the first place ;-). I am not
> sure the pessimistic initialization is better in all cases either, since
> we have to remember to place it before any "cleanup" label, and make
> sure that that does not regress.
> > So, I dunno. I'm OK with what is written here, and I think we could
> certainly have a separate discussion to perhaps have CodingGuidelines
> take a stronger stance here.

I'll look into adding this as a cleanup. This specific patch is
about adding the 'goto's where they were missing. I can make a new
patch that unifies the result initialization to -1 (and thus making
the method unified in returning -1 on error). There are many more
"result = 1" lines that need to change.

Thanks,
-Stolee

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> For now, replace the use of -1 with a 'NO_PREFERRED_PACK' macro and an
> equality check. The macro stores the max value of a uint32_t, so we
> cannot store a preferred pack that appears last in a list of 2^32 total
> packs, but that's expected to be unreasonable already. This improves the
> range from 2^31 already.

;-)  

I very much like this change.  An obvious alternative may be to use
int instead of uint32_t to number and index into in-core packs, as
we all know that not just 2^32 but 2^31 is still unreasonably too
many anyway.

> There are some careful things to worry about with initializing the
> preferred pack in the struct and using that value when searching for a
> preferred pack that was already incorrect but accidentally working when
> the index was initialized to zero.

True.

> -	struct write_midx_context ctx = { 0 };
> +	struct write_midx_context ctx = {
> +		.preferred_pack_idx = NO_PREFERRED_PACK,
> +	 };

Good.

>  	if (preferred_pack_name) {
> -		ctx.preferred_pack_idx = -1;
> +		ctx.preferred_pack_idx = NO_PREFERRED_PACK;

This too.

Thanks.

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> The strategy here involves defining the variable within the for loop
> syntax to make sure we use the appropriate bitness for the loop
> sentinel. This matters in at least one method where the variable was
> compared to uint32_t in some loops and size_t in others.

Sounds like a fine way to avoid accidentally breaking these loops.

On Thu, Aug 28, 2025 at 02:01:42PM -0700, Junio C Hamano wrote:
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > The strategy here involves defining the variable within the for loop
> > syntax to make sure we use the appropriate bitness for the loop
> > sentinel. This matters in at least one method where the variable was
> > compared to uint32_t in some loops and size_t in others.
>
> Sounds like a fine way to avoid accidentally breaking these loops.

Yup, all looks good to me.

Thanks,
Taylor

On Thu, Aug 28, 2025 at 05:39:51PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <stolee@gmail.com>
>
> The fill_packs_from_midx() method was refactored in fcb2205b77 (midx:
> implement support for writing incremental MIDX chains, 2024-08-06) to
> allow for preferred packfiles and incremental multi-pack-indexes.
> However, this led to some conditions that can cause improperly
> initialized memory in the context's list of packfiles.
>
> The conditions caring about the preferred pack name or the incremental
> flag are currently necessary to load a packfile. But the context is
> still being populated with pack_info structs based on the packfile array
> for the existing multi-pack-index even if prepare_midx_pack() isn't
> called.

Thanks for looking at this one. On the surface this looks not great, but
I am having a hard time coming up with a smaller test case that
exercises this behavior.

I can get what you wrote below to fail on my machine pretty reliable
when building with SANITIZE=address (even without --stress). All of the
spots that read from the pack_info array and access the actual
packed_git structs are guarded by either writing a MIDX bitmap or having
a non-empty preferred pack.

So I can see that we're definitely trying to close_pack() on a NULL
pointer, but I can't find a way to trigger that more directly.

(I think the fact that we don't appear to be accessing the packed_git
struct because the spots that want to are guarded by the same conditions
that cause us to call prepare_midx_pack() in the first place is pure
luck and not something that I am comfortable with us relying on. So we
should fix this up for sure, but I wish I understood the existing bug a
little more clearly first.)

> Add a new test that breaks under --stress when compiled with
> SANITIZE=address. The chosen number of 100 packfiles was selected to get
> the --stress output to fail about 50% of the time, while 50 packfiles
> could not get a failure in most --stress runs. This test has a very
> minor check at the end confirming only one packfile remaining. The
> failing nature of this test actually relies on auto-GC cleaning up some
> packfiles during the creation of the commits, as tests setting gc.auto
> to zero make the packfile count match the number of added commits but
> also avoids hitting the memory issue.

Hmm. Is this portion of the commit message out-of-date? I can't see the
check you're referring to that ensures there is only one pack remaining,
nor can I see the spot where we disable gc.auto.

> The test case is marked as EXPENSIVE not only because of the number of
> packfiles it creates, but because some CI environments were reporting
> errors during the test that I could not reproduce, specifically around
> being unable to open the packfiles or their pack-indexes.
>
> When it fails under SANITIZE=address, it provides the following error:
>
> AddressSanitizer:DEADLYSIGNAL
> =================================================================
> ==3263517==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000027
> ==3263517==The signal is caused by a READ memory access.
> ==3263517==Hint: address points to the zero page.
>     #0 0x562d5d82d1fb in close_pack_windows packfile.c:299
>     #1 0x562d5d82d3ab in close_pack packfile.c:354
>     #2 0x562d5d7bfdb4 in write_midx_internal midx-write.c:1490
>     #3 0x562d5d7c7aec in midx_repack midx-write.c:1795
>     #4 0x562d5d46fff6 in cmd_multi_pack_index builtin/multi-pack-index.c:305
>     ...
>
> This failure stack trace is disconnected from the real fix because it

s/it// ?

> the bad pointers are accessed later when closing the packfiles from the
> context.
>
> There are a few different aspects to this fix that are worth noting:
>
>  1. We return to the previous behavior of fill_packs_from_midx to not
>     rely on the incremental flag or existence of a preferred pack.
>
>  2. The behavior to scan all layers of an incremental midx is kept, so
>     this is not a full revert of the change.
>
>  3. We skip allocating more room in the pack_info array if the pack
>     fails prepare_midx_pack().
>
>  4. The method has always returned 0 for success and 1 for failure, but
>     the condition checking for error added a check for a negative result
>     for failure, so that is now updated.

Oops ;-).

>  5. The call to open_pack_index() is removed, but this is needed later
>     in the case of a preferred pack. That call is moved to immediately
>     before its result is needed (checking for the object count).

I think we need to do this in at least one other spot, but see below.

> +			if (prepare_midx_pack(ctx->repo, m,
> +					      m->num_packs_in_base + i)) {
> +				error(_("could not load pack"));
> +				return 1;

Looks good, though I agree with Junio's comment in his separate reply
that we could probably just turn this into "return error(...)" while
we're at it.

> @@ -1223,6 +1204,11 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>
>  	if (ctx.preferred_pack_idx > -1) {
>  		struct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;
> +
> +		if (open_pack_index(preferred))
> +			die(_("failed to open preferred pack %s"),
> +			    ctx.info[ctx.preferred_pack_idx].pack_name);

This makes sense, but I think we need to apply similar treatment in the
"else if" arm of the if-statement immediately above this one too. That
portion of the code handles the case where we're writing a MIDX bitmap
but didn't provide a preferred pack.

When that's the case, we loop through to try and find the oldest pack
that contains at least one object. If we don't call open_pack_index()
all of those ->num_objects fields will still be zero'd, so we'll only
find the oldest pack.

That may actually produce wrong behavior if we have duplicate objects
that aren't uniformly resolved in favor of the earliest pack in lex
order. I'd have to think about it a little more to be sure, though.

Thanks,
Taylor

On 8/28/25 9:20 PM, Taylor Blau wrote:
> On Thu, Aug 28, 2025 at 05:39:51PM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <stolee@gmail.com>
>>
>> The fill_packs_from_midx() method was refactored in fcb2205b77 (midx:
>> implement support for writing incremental MIDX chains, 2024-08-06) to
>> allow for preferred packfiles and incremental multi-pack-indexes.
>> However, this led to some conditions that can cause improperly
>> initialized memory in the context's list of packfiles.
>>
>> The conditions caring about the preferred pack name or the incremental
>> flag are currently necessary to load a packfile. But the context is
>> still being populated with pack_info structs based on the packfile array
>> for the existing multi-pack-index even if prepare_midx_pack() isn't
>> called.
> > Thanks for looking at this one. On the surface this looks not great, but
> I am having a hard time coming up with a smaller test case that
> exercises this behavior.
> > I can get what you wrote below to fail on my machine pretty reliable
> when building with SANITIZE=address (even without --stress). All of the
> spots that read from the pack_info array and access the actual
> packed_git structs are guarded by either writing a MIDX bitmap or having
> a non-empty preferred pack.

I'm glad you're able to reproduce it. My --stress runs had about a
50% hit rate.

>> Add a new test that breaks under --stress when compiled with
>> SANITIZE=address. The chosen number of 100 packfiles was selected to get
>> the --stress output to fail about 50% of the time, while 50 packfiles
>> could not get a failure in most --stress runs. This test has a very
>> minor check at the end confirming only one packfile remaining. The
>> failing nature of this test actually relies on auto-GC cleaning up some
>> packfiles during the creation of the commits, as tests setting gc.auto
>> to zero make the packfile count match the number of added commits but
>> also avoids hitting the memory issue.
> > Hmm. Is this portion of the commit message out-of-date? I can't see the
> check you're referring to that ensures there is only one pack remaining,
> nor can I see the spot where we disable gc.auto.

You're right. When I added more robustness around the packfile count
by removing gc.auto, the test stopped failing pre-fix. Then, I forgot
to remove mention of those test updates.

>> The test case is marked as EXPENSIVE not only because of the number of
>> packfiles it creates, but because some CI environments were reporting
>> errors during the test that I could not reproduce, specifically around
>> being unable to open the packfiles or their pack-indexes.
>>
>> When it fails under SANITIZE=address, it provides the following error:
>>
>> AddressSanitizer:DEADLYSIGNAL
>> =================================================================
>> ==3263517==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000027
>> ==3263517==The signal is caused by a READ memory access.
>> ==3263517==Hint: address points to the zero page.
>>      #0 0x562d5d82d1fb in close_pack_windows packfile.c:299
>>      #1 0x562d5d82d3ab in close_pack packfile.c:354
>>      #2 0x562d5d7bfdb4 in write_midx_internal midx-write.c:1490
>>      #3 0x562d5d7c7aec in midx_repack midx-write.c:1795
>>      #4 0x562d5d46fff6 in cmd_multi_pack_index builtin/multi-pack-index.c:305
>>      ...
>>
>> This failure stack trace is disconnected from the real fix because it
> > s/it// ?

Thanks.

>> the bad pointers are accessed later when closing the packfiles from the
>> context.
>>
>> There are a few different aspects to this fix that are worth noting:
>>
>>   1. We return to the previous behavior of fill_packs_from_midx to not
>>      rely on the incremental flag or existence of a preferred pack.
>>
>>   2. The behavior to scan all layers of an incremental midx is kept, so
>>      this is not a full revert of the change.
>>
>>   3. We skip allocating more room in the pack_info array if the pack
>>      fails prepare_midx_pack().
>>
>>   4. The method has always returned 0 for success and 1 for failure, but
>>      the condition checking for error added a check for a negative result
>>      for failure, so that is now updated.
> > Oops ;-).
> >>   5. The call to open_pack_index() is removed, but this is needed later
>>      in the case of a preferred pack. That call is moved to immediately
>>      before its result is needed (checking for the object count).
> > I think we need to do this in at least one other spot, but see below.

Interesting!

>> +			if (prepare_midx_pack(ctx->repo, m,
>> +					      m->num_packs_in_base + i)) {
>> +				error(_("could not load pack"));
>> +				return 1;
> > Looks good, though I agree with Junio's comment in his separate reply
> that we could probably just turn this into "return error(...)" while
> we're at it.

Can do.

>> @@ -1223,6 +1204,11 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>>
>>   	if (ctx.preferred_pack_idx > -1) {
>>   		struct packed_git *preferred = ctx.info[ctx.preferred_pack_idx].p;
>> +
>> +		if (open_pack_index(preferred))
>> +			die(_("failed to open preferred pack %s"),
>> +			    ctx.info[ctx.preferred_pack_idx].pack_name);
> > This makes sense, but I think we need to apply similar treatment in the
> "else if" arm of the if-statement immediately above this one too. That
> portion of the code handles the case where we're writing a MIDX bitmap
> but didn't provide a preferred pack.
> > When that's the case, we loop through to try and find the oldest pack
> that contains at least one object. If we don't call open_pack_index()
> all of those ->num_objects fields will still be zero'd, so we'll only
> find the oldest pack.
> > That may actually produce wrong behavior if we have duplicate objects
> that aren't uniformly resolved in favor of the earliest pack in lex
> order. I'd have to think about it a little more to be sure, though.

I see. In this case, we need to open_pack_index() before relying on
oldest->num_objects, which only needs to happen for the first pack
and any packfile that wins via mtime preference. It also seems like
we can _warn_ on failures to open packfiles in those cases, since it
isn't fatal if some packfiles fail to open.

Thanks,
-Stolee

On Thu, Aug 28, 2025 at 05:39:54PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <stolee@gmail.com>
>
> midx-write.c has the DISABLE_SIGN_COMPARE_WARNINGS macro defined for a
> few reasons, but the biggest one is the use of a signed
> preferred_pack_idx member inside the write_midx_context struct. The code
> currently uses -1 to indicate an unset preferred pack but pack int ids
> are normally handled as uint32_t. There are also a few loops that search
> for the preferred pack by name and those iterators will need updates to
> uint32_t in the next change.
>
> For now, replace the use of -1 with a 'NO_PREFERRED_PACK' macro and an
> equality check. The macro stores the max value of a uint32_t, so we
> cannot store a preferred pack that appears last in a list of 2^32 total
> packs, but that's expected to be unreasonable already. This improves the
> range from 2^31 already.

Nice ;-).

> There are some careful things to worry about with initializing the
> preferred pack in the struct and using that value when searching for a
> preferred pack that was already incorrect but accidentally working when
> the index was initialized to zero.
>
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c | 26 +++++++++++++++-----------
>  1 file changed, 15 insertions(+), 11 deletions(-)
>
> diff --git a/midx-write.c b/midx-write.c
> index f2d9a990e6..ea1b3a199c 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -24,6 +24,7 @@
>  #define BITMAP_POS_UNKNOWN (~((uint32_t)0))
>  #define MIDX_CHUNK_FANOUT_SIZE (sizeof(uint32_t) * 256)
>  #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))
> +#define NO_PREFERRED_PACK (~((uint32_t)0))

I think that just (~(uint32_t)0) is necessary, but I don't mind the
extra clarity.

> @@ -274,7 +275,7 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,
>  	end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);
>
>  	for (cur_object = start; cur_object < end; cur_object++) {
> -		if ((preferred_pack > -1) &&
> +		if ((preferred_pack != NO_PREFERRED_PACK) &&
>  		    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {

OK, here we should make sure that the preferred pack was provided.

Like you mentioned above, I guess that means we can't have the 2^32-1'st
pack, but before we couldn't have any pack greater than 2^31-1
preferred, so this is a strict improvement ;-).

>  			/*
>  			 * Objects from preferred packs are added
> @@ -364,7 +365,8 @@ static void compute_sorted_entries(struct write_midx_context *ctx,
>  						    preferred, cur_fanout);
>  		}
>
> -		if (-1 < ctx->preferred_pack_idx && ctx->preferred_pack_idx < start_pack)
> +		if (ctx->preferred_pack_idx != NO_PREFERRED_PACK &&
> +		    ctx->preferred_pack_idx < start_pack)

Looks good. It's tempting to say that any value of preferred_pack_idx
lower than start_pack is going to be OK, since start_pack is a uint32_t
as well, but we need to make sure that the preferred pack was specified
in the first place.

The rest all looks good as well, thanks.

Thanks,
Taylor

On Sat, Aug 30, 2025 at 09:23:22PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <stolee@gmail.com>
> 
> The fill_packs_from_midx() method was refactored in fcb2205b77 (midx:
> implement support for writing incremental MIDX chains, 2024-08-06) to
> allow for preferred packfiles and incremental multi-pack-indexes.
> However, this led to some conditions that can cause improperly
> initialized memory in the context's list of packfiles.
> 
> The conditions caring about the preferred pack name or the incremental
> flag are currently necessary to load a packfile. But the context is
> still being populated with pack_info structs based on the packfile array
> for the existing multi-pack-index even if prepare_midx_pack() isn't
> called.

I honestly don't quite understand why the conditions are necessary here.
In other words, why do we need to be careful _not_ to open the
packfiles?

> Add a new test that breaks under --stress when compiled with
> SANITIZE=address. The chosen number of 100 packfiles was selected to get
> the --stress output to fail about 50% of the time, while 50 packfiles
> could not get a failure in most --stress runs.
> 
> The test case is marked as EXPENSIVE not only because of the number of
> packfiles it creates, but because some CI environments were reporting
> errors during the test that I could not reproduce, specifically around
> being unable to open the packfiles or their pack-indexes.
> 
> When it fails under SANITIZE=address, it provides the following error:
> 
> AddressSanitizer:DEADLYSIGNAL
> =================================================================
> ==3263517==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000027
> ==3263517==The signal is caused by a READ memory access.
> ==3263517==Hint: address points to the zero page.
>     #0 0x562d5d82d1fb in close_pack_windows packfile.c:299
>     #1 0x562d5d82d3ab in close_pack packfile.c:354
>     #2 0x562d5d7bfdb4 in write_midx_internal midx-write.c:1490
>     #3 0x562d5d7c7aec in midx_repack midx-write.c:1795
>     #4 0x562d5d46fff6 in cmd_multi_pack_index builtin/multi-pack-index.c:305
>     ...
> 
> This failure stack trace is disconnected from the real fix because the bad
> pointers are accessed later when closing the packfiles from the context.

Okay. So in other words we need to make sure to always prepare the
MIDX'd packfiles, but we may not want to open them?

> There are a few different aspects to this fix that are worth noting:
> 
>  1. We return to the previous behavior of fill_packs_from_midx to not
>     rely on the incremental flag or existence of a preferred pack.
> 
>  2. The behavior to scan all layers of an incremental midx is kept, so
>     this is not a full revert of the change.
> 
>  3. We skip allocating more room in the pack_info array if the pack
>     fails prepare_midx_pack().
> 
>  4. The method has always returned 0 for success and 1 for failure, but
>     the condition checking for error added a check for a negative result
>     for failure, so that is now updated.

Nit, feel free to ignore: this change feels like it would make for a
nice separate commit.

Patrick

On 9/3/2025 6:14 AM, Patrick Steinhardt wrote:
> On Sat, Aug 30, 2025 at 09:23:22PM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <stolee@gmail.com>
>>
>> The fill_packs_from_midx() method was refactored in fcb2205b77 (midx:
>> implement support for writing incremental MIDX chains, 2024-08-06) to
>> allow for preferred packfiles and incremental multi-pack-indexes.
>> However, this led to some conditions that can cause improperly
>> initialized memory in the context's list of packfiles.
>>
>> The conditions caring about the preferred pack name or the incremental
>> flag are currently necessary to load a packfile. But the context is
>> still being populated with pack_info structs based on the packfile array
>> for the existing multi-pack-index even if prepare_midx_pack() isn't
>> called.
> 
> I honestly don't quite understand why the conditions are necessary here.
> In other words, why do we need to be careful _not_ to open the
> packfiles?

My wording is poor. "We don't load packfiles unless one of these
conditions holds" is more appropriate.

There are some test cases that want to keep things working even
when a .idx file disappears, I think. This is a reason to be
careful about open_pack_index(), but prepare_midx_pack() is
something we want to call always.
 
>> Add a new test that breaks under --stress when compiled with
>> SANITIZE=address. The chosen number of 100 packfiles was selected to get
>> the --stress output to fail about 50% of the time, while 50 packfiles
>> could not get a failure in most --stress runs.
>>
>> The test case is marked as EXPENSIVE not only because of the number of
>> packfiles it creates, but because some CI environments were reporting
>> errors during the test that I could not reproduce, specifically around
>> being unable to open the packfiles or their pack-indexes.
>>
>> When it fails under SANITIZE=address, it provides the following error:
>>
>> AddressSanitizer:DEADLYSIGNAL
>> =================================================================
>> ==3263517==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000027
>> ==3263517==The signal is caused by a READ memory access.
>> ==3263517==Hint: address points to the zero page.
>>     #0 0x562d5d82d1fb in close_pack_windows packfile.c:299
>>     #1 0x562d5d82d3ab in close_pack packfile.c:354
>>     #2 0x562d5d7bfdb4 in write_midx_internal midx-write.c:1490
>>     #3 0x562d5d7c7aec in midx_repack midx-write.c:1795
>>     #4 0x562d5d46fff6 in cmd_multi_pack_index builtin/multi-pack-index.c:305
>>     ...
>>
>> This failure stack trace is disconnected from the real fix because the bad
>> pointers are accessed later when closing the packfiles from the context.
> 
> Okay. So in other words we need to make sure to always prepare the
> MIDX'd packfiles, but we may not want to open them?

Yes. Always prepare. Don't always open (since that loads the .idx).

>> There are a few different aspects to this fix that are worth noting:
>>
>>  1. We return to the previous behavior of fill_packs_from_midx to not
>>     rely on the incremental flag or existence of a preferred pack.
>>
>>  2. The behavior to scan all layers of an incremental midx is kept, so
>>     this is not a full revert of the change.
>>
>>  3. We skip allocating more room in the pack_info array if the pack
>>     fails prepare_midx_pack().
>>
>>  4. The method has always returned 0 for success and 1 for failure, but
>>     the condition checking for error added a check for a negative result
>>     for failure, so that is now updated.
> 
> Nit, feel free to ignore: this change feels like it would make for a
> nice separate commit.

True. I only included it since I was modifying the call anyway due to
the changing parameters.

Thanks,
-Stolee

On Sat, Aug 30, 2025 at 09:23:23PM +0000, Derrick Stolee via GitGitGadget wrote:
> diff --git a/midx-write.c b/midx-write.c
> index 070a7f61f4..0f1d5653ab 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1104,6 +1104,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  			m = m->base_midx;
>  		}
>  	} else if (ctx.m && fill_packs_from_midx(&ctx)) {
> +		result = 1;
>  		goto cleanup;
>  	}

Would it make sense to also convert this command to return negative
error codes?

> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
> index 49705c62a2..008e65c22e 100755
> --- a/t/t5319-multi-pack-index.sh
> +++ b/t/t5319-multi-pack-index.sh
> @@ -1100,7 +1100,10 @@ test_expect_success 'load reverse index when missing .idx, .pack' '
>  		mv $idx.bak $idx &&
>  
>  		mv $pack $pack.bak &&
> -		git cat-file --batch-check="%(objectsize:disk)" <tip
> +		git cat-file --batch-check="%(objectsize:disk)" <tip &&
> +
> +		test_must_fail git multi-pack-index write 2>err &&
> +		grep "could not load pack" err

Nit: this should probably use `test_grep`.

Patrick

On 9/3/2025 6:15 AM, Patrick Steinhardt wrote:
> On Sat, Aug 30, 2025 at 09:23:23PM +0000, Derrick Stolee via GitGitGadget wrote:
>> diff --git a/midx-write.c b/midx-write.c
>> index 070a7f61f4..0f1d5653ab 100644
>> --- a/midx-write.c
>> +++ b/midx-write.c
>> @@ -1104,6 +1104,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>>  			m = m->base_midx;
>>  		}
>>  	} else if (ctx.m && fill_packs_from_midx(&ctx)) {
>> +		result = 1;
>>  		goto cleanup;
>>  	}
> 
> Would it make sense to also convert this command to return negative
> error codes?

(Yes, in patch 6) 
>> diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
>> index 49705c62a2..008e65c22e 100755
>> --- a/t/t5319-multi-pack-index.sh
>> +++ b/t/t5319-multi-pack-index.sh
>> @@ -1100,7 +1100,10 @@ test_expect_success 'load reverse index when missing .idx, .pack' '
>>  		mv $idx.bak $idx &&
>>  
>>  		mv $pack $pack.bak &&
>> -		git cat-file --batch-check="%(objectsize:disk)" <tip
>> +		git cat-file --batch-check="%(objectsize:disk)" <tip &&
>> +
>> +		test_must_fail git multi-pack-index write 2>err &&
>> +		grep "could not load pack" err
> 
> Nit: this should probably use `test_grep`.

You're right. I think I misremembered the preferred use back when
test_i18ngrep was deprecated.

Thanks,
-Stolee

On Sat, Aug 30, 2025 at 09:23:24PM +0000, Derrick Stolee via GitGitGadget wrote:
> diff --git a/midx-write.c b/midx-write.c
> index 0f1d5653ab..cb0211289d 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1327,13 +1327,15 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  		incr = mks_tempfile_m(midx_name.buf, 0444);
>  		if (!incr) {
>  			error(_("unable to create temporary MIDX layer"));
> -			return -1;
> +			result = -1;
> +			goto cleanup;

We can instead use `result = error(...); goto cleanup;` here and in most
other cases.

Patrick

On Sat, Aug 30, 2025 at 09:23:25PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <stolee@gmail.com>
> 
> midx-write.c has the DISABLE_SIGN_COMPARE_WARNINGS macro defined for a
> few reasons, but the biggest one is the use of a signed
> preferred_pack_idx member inside the write_midx_context struct. The code
> currently uses -1 to indicate an unset preferred pack but pack int ids
> are normally handled as uint32_t. There are also a few loops that search
> for the preferred pack by name and those iterators will need updates to
> uint32_t in the next change.
> 
> For now, replace the use of -1 with a 'NO_PREFERRED_PACK' macro and an
> equality check. The macro stores the max value of a uint32_t, so we
> cannot store a preferred pack that appears last in a list of 2^32 total
> packs, but that's expected to be unreasonable already. This improves the
> range from 2^31 already.

Tiny nit: the last sentence reads a bit funny. Maybe something like
this?

    Furthermore, with this change we end up extending the range from
    2^31 possible packs to 2^32-1.

> There are some careful things to worry about with initializing the
> preferred pack in the struct and using that value when searching for a
> preferred pack that was already incorrect but accidentally working when
> the index was initialized to zero.
> 
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c | 26 +++++++++++++++-----------
>  1 file changed, 15 insertions(+), 11 deletions(-)
> 
> diff --git a/midx-write.c b/midx-write.c
> index cb0211289d..1822268ce2 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -274,7 +275,7 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,
>  	end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);
>  
>  	for (cur_object = start; cur_object < end; cur_object++) {
> -		if ((preferred_pack > -1) &&
> +		if ((preferred_pack != NO_PREFERRED_PACK) &&
>  		    (preferred_pack == nth_midxed_pack_int_id(m, cur_object))) {
>  			/*
>  			 * Objects from preferred packs are added

Neither of these braces around comparisons are really needed, but feel
free to ignore as you simply piggy-back on existing style.

> @@ -1040,7 +1042,9 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  	struct hashfile *f = NULL;
>  	struct lock_file lk;
>  	struct tempfile *incr;
> -	struct write_midx_context ctx = { 0 };
> +	struct write_midx_context ctx = {
> +		.preferred_pack_idx = NO_PREFERRED_PACK,
> +	 };
>  	int bitmapped_packs_concat_len = 0;
>  	int pack_name_concat_len = 0;
>  	int dropped_packs = 0;

Why is this change needed? We didn't previously initialize
`.preferred_pack_idx = -1` either.

Patrick

On 9/3/2025 6:15 AM, Patrick Steinhardt wrote:
> On Sat, Aug 30, 2025 at 09:23:25PM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <stolee@gmail.com>
>>
>> midx-write.c has the DISABLE_SIGN_COMPARE_WARNINGS macro defined for a
>> few reasons, but the biggest one is the use of a signed
>> preferred_pack_idx member inside the write_midx_context struct. The code
>> currently uses -1 to indicate an unset preferred pack but pack int ids
>> are normally handled as uint32_t. There are also a few loops that search
>> for the preferred pack by name and those iterators will need updates to
>> uint32_t in the next change.
>>
>> For now, replace the use of -1 with a 'NO_PREFERRED_PACK' macro and an
>> equality check. The macro stores the max value of a uint32_t, so we
>> cannot store a preferred pack that appears last in a list of 2^32 total
>> packs, but that's expected to be unreasonable already. This improves the
>> range from 2^31 already.
> 
> Tiny nit: the last sentence reads a bit funny. Maybe something like
> this?
> 
>     Furthermore, with this change we end up extending the range from
>     2^31 possible packs to 2^32-1.

That is better.

>> @@ -1040,7 +1042,9 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>>  	struct hashfile *f = NULL;
>>  	struct lock_file lk;
>>  	struct tempfile *incr;
>> -	struct write_midx_context ctx = { 0 };
>> +	struct write_midx_context ctx = {
>> +		.preferred_pack_idx = NO_PREFERRED_PACK,
>> +	 };
>>  	int bitmapped_packs_concat_len = 0;
>>  	int pack_name_concat_len = 0;
>>  	int dropped_packs = 0;
> 
> Why is this change needed? We didn't previously initialize
> `.preferred_pack_idx = -1` either.

I think the previous lack of initialization was incorrect. It happened
to work because it became initialized to -1 later _or_ its value of
zero was implicitly used when searching for a preferred pack.

I thought it prudent to set this value as the default instead of
implying that the 0th packfile was preferred.

Thanks,
-Stolee

On Sat, Aug 30, 2025 at 09:23:26PM +0000, Derrick Stolee via GitGitGadget wrote:
> diff --git a/midx-write.c b/midx-write.c
> index 1822268ce2..14a0947c46 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1430,6 +1428,9 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  	 * have been freed in the previous if block.
>  	 */
>  
> +	if (ctx.num_multi_pack_indexes_before == UINT32_MAX)
> +		die("too many multi-pack-indexes");
> +
>  	CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);
>  
>  	if (ctx.incremental) {

Should this error message be translated?

Everything else in this commit looks good to me. Thanks for cleaning
these up.

Patrick

On Sat, Aug 30, 2025 at 09:23:27PM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <stolee@gmail.com>
> 
> The write_midx_internal() method uses gotos to jump to a cleanup section to
> clear memory before returning 'result'. Since these jumps are more common
> for error conditions, initialize 'result' to -1 and then only set it to 0
> before returning with success. There are a couple places where we return
> with success via a jump.
> 
> This has the added benefit that the method now returns -1 on error instead
> of an inconsistent 1 or -1.
> 
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  midx-write.c | 26 +++++++++-----------------
>  1 file changed, 9 insertions(+), 17 deletions(-)
> 
> diff --git a/midx-write.c b/midx-write.c
> index 14a0947c46..047ffdcdbf 100644
> --- a/midx-write.c
> +++ b/midx-write.c
> @@ -1046,7 +1046,7 @@ static int write_midx_internal(struct repository *r, const char *object_dir,
>  	int bitmapped_packs_concat_len = 0;
>  	int pack_name_concat_len = 0;
>  	int dropped_packs = 0;
> -	int result = 0;
> +	int result = -1;
>  	const char **keep_hashes = NULL;
>  	struct chunkfile *cf;

I personally prefer to keep the result uninitialized and then assign the
result of `error()` to it. It's almost the same lines of code as we have
right now, but it has the advantage that the compiler will complain
about `result` being uninitialized if we ever forget to set it. So it's
overall way more explicit, and the compiler protects us.

But seeing that Junio previously recommended to go into the direction of
setting it to `-1` I won't insist on such a refactoring. So please feel
free to ignore this comment.

Patrick

Patrick Steinhardt <ps@pks.im> writes:

>> -	int result = 0;
>> +	int result = -1;
>>  	const char **keep_hashes = NULL;
>>  	struct chunkfile *cf;
>
> I personally prefer to keep the result uninitialized and then assign the
> result of `error()` to it. It's almost the same lines of code as we have
> right now, but it has the advantage that the compiler will complain
> about `result` being uninitialized if we ever forget to set it. So it's
> overall way more explicit, and the compiler protects us.
>
> But seeing that Junio previously recommended to go into the direction of
> setting it to `-1` I won't insist on such a refactoring. So please feel
> free to ignore this comment.

I am equally fine with uninitialized one, as long as compilers are
trustworthy in all cases.  But the code path I made a comment IIRC
did not necessarily have calls to error(), so the same number of
code argument does not apply.  And initializing it to zero is worse
than leaving it uninitialized.

Thanks.