Revision b16a8277644e4b1d21c08d97de757105039dc7ae authored by Derrick Stolee on 16 September 2020, 18:07:52 UTC, committed by Junio C Hamano on 17 September 2020, 16:31:25 UTC
Commit e3696980 (diff: halt tree-diff early after max_changes, 2020-03-30) intended to create a mechanism to short-circuit a diff calculation after a certain number of paths were modified. By incrementing a "num_changes" counter throughout the recursive ll_diff_tree_paths(), this was supposed to match the number of changes that would be written into the changed-path Bloom filters. Unfortunately, this was not implemented correctly and instead misses simple cases like file modifications. This then does not stop very large changed-path filters from being written (unless they add or remove many files). To start, change the implementation in ll_diff_tree_paths() to instead use the global diff_queue_diff struct's 'nr' member as the count. This is a way to simplify the logic instead of making more mistakes in the complicated diff code. This has a drawback: the diff_queue_diff struct only lists the paths corresponding to blob changes, not their leading directories. Thus, get_or_compute_bloom_filter() needs an additional check to see if the hashmap with the leading directories becomes too large. One reason why this was not caught by test cases was that the test in t4216-log-bloom.sh that was supposed to check this "too many changes" condition only checked this on the initial commit of a repository. The old logic counted these values correctly. Update this test in a few ways: 1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit, allowing smaller commits to engage with this logic. 2. Create several interesting cases of edits, adds, removes, and mode changes (in the second commit). By testing both sides of the inequality with the *_MAX_CHANGED_PATHS variable, we can see that the count is exactly correct, so none of these changes are missed or over-counted. 3. Use the trace2 data value filter_found_large to verify that these commits are on the correct side of the limit. Another way to verify the behavior is correct is through performance tests. By testing on my local copies of the Git repository and the Linux kernel repository, I could measure the effect of these short-circuits when computing a fresh commit-graph file with changed-path Bloom filters using the command GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \ git commit-graph write --reachable --changed-paths and reporting the wall time and resulting commit-graph size. For Git, the results are | | N=1 | N=10 | N=512 | |--------|----------------|----------------|----------------| | HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB | | HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB | For Linux, the results are | | N=1 | N=20 | N=512 | |--------|----------------|---------------|---------------| | HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB | | HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB | Naturally, the improvement becomes much less as the limit grows, as fewer commits satisfy the short-circuit. Reported-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
1 parent 9a7a9ed
gettext.h
/*
* Copyright (c) 2010-2011 Ævar Arnfjörð Bjarmason
*
* This is a skeleton no-op implementation of gettext for Git.
* You can replace it with something that uses libintl.h and wraps
* gettext() to try out the translations.
*/
#ifndef GETTEXT_H
#define GETTEXT_H
#if defined(_) || defined(Q_)
#error "namespace conflict: '_' or 'Q_' is pre-defined?"
#endif
#ifndef NO_GETTEXT
# include <libintl.h>
#else
# ifdef gettext
# undef gettext
# endif
# define gettext(s) (s)
# ifdef ngettext
# undef ngettext
# endif
# define ngettext(s, p, n) ((n == 1) ? (s) : (p))
#endif
#define FORMAT_PRESERVING(n) __attribute__((format_arg(n)))
int use_gettext_poison(void);
#ifndef NO_GETTEXT
void git_setup_gettext(void);
int gettext_width(const char *s);
#else
static inline void git_setup_gettext(void)
{
use_gettext_poison(); /* getenv() reentrancy paranoia */
}
static inline int gettext_width(const char *s)
{
return strlen(s);
}
#endif
static inline FORMAT_PRESERVING(1) const char *_(const char *msgid)
{
if (!*msgid)
return "";
return use_gettext_poison() ? "# GETTEXT POISON #" : gettext(msgid);
}
static inline FORMAT_PRESERVING(1) FORMAT_PRESERVING(2)
const char *Q_(const char *msgid, const char *plu, unsigned long n)
{
if (use_gettext_poison())
return "# GETTEXT POISON #";
return ngettext(msgid, plu, n);
}
/* Mark msgid for translation but do not translate it. */
#if !USE_PARENS_AROUND_GETTEXT_N
#define N_(msgid) msgid
#else
/*
* Strictly speaking, this will lead to invalid C when
* used this way:
* static const char s[] = N_("FOO");
* which will expand to
* static const char s[] = ("FOO");
* and in valid C, the initializer on the right hand side must
* be without the parentheses. But many compilers do accept it
* as a language extension and it will allow us to catch mistakes
* like:
* static const char *msgs[] = {
* N_("one")
* N_("two"),
* N_("three"),
* NULL
* };
* (notice the missing comma on one of the lines) by forcing
* a compilation error, because parenthesised ("one") ("two")
* will not get silently turned into ("onetwo").
*/
#define N_(msgid) (msgid)
#endif
const char *get_preferred_languages(void);
int is_utf8_locale(void);
#endif
![swh spinner](/static/img/swh-spinner.gif)
Computing file changes ...