How should we be using dependabot in this repo?

[mdboom]

I noticed a few dependabot PRs in this repo, such as this one that upgrade dependencies of specific benchmarks. While in general, I think this is good practice, for a benchmark suite, I think we'd want to upgrade these dependencies as infrequently as possible to keep benchmarking results comparable with one another (and not have to always rerun baselines). Occasionally we are forced to upgrade, for example to get compatibility with a new version of CPython, but that should be deliberate.

(It's possible there is a security counterargument to be made, but I'm not a security expert and I don't know specifically whether that matters or not).

Would it make sense to update the dependabot config to only look at the top-level dependencies of pyperformance itself rather than the dependencies of specific benchmarks?

[ericsnowcurrently]

for a benchmark suite, I think we'd want to upgrade these dependencies as infrequently as possible to keep benchmarking results comparable with one another

+1

It's possible there is a security counterargument to be made

I don't consider the benchmarks to be a security concern, though I haven't done a thorough analysis of all the benchmark code. IIRC, none of the benchmarks access anything other than files they already own or temporary network services they start themselves.

Would it make sense to update the dependabot config to only look at the top-level dependencies of pyperformance itself

+1

[mdboom]

It looks like this may not be easily possible: dependabot/dependabot-core#4364

[maurycy]

@mdboom @hugovk

I believe this can be closed now, since #407 is merged.

[AA-Turner]

Seems this hasn't worked, #412 was just opened. cc @maurycy.

[maurycy]

@AA-Turner

Thank you for spotting this.

I reached out to the Dependabot team directly:

• Ignore manifests in specific subdirectories dependabot/dependabot-core#4364 (comment)

since this seems to be still a very active area of development:

• Revert exclude paths refactoring logic dependabot/dependabot-core#13035
• Add exclusion logic in ecosystem file fetcher classes for exclude-paths dependabot/dependabot-core#13047

despite not being marked as so in the docs.

I worry that even if I clone the repository and figure out a solution, our approach won't be stable because of ongoing changes.

[hugovk]

Thanks for following up with the Dependabot team. We're not in a rush to get this sorted, so can wait for them to stabilise the feature.

[maurycy]

@hugovk @AA-Turner

The fix should be deployed:

• Ignore manifests in specific subdirectories dependabot/dependabot-core#4364 (comment)

Let's see how it goes before closing this issue.

🤞

[maurycy]

@hugovk @AA-Turner It seems to be that we're good now?

[hugovk]

I think so, we've not had any PRs for pyperformance/data-files/ since June.

https://github.com/python/pyperformance/pulls?q=sort%3Aupdated-desc+is%3Apr+author%3Aapp%2Fdependabot+is%3Aclosed

Thank you!