pin versions/tags for reproducibility#5
Conversation
There was a problem hiding this comment.
In this case, how would you benchmark a new version which does not have yet a tag. This will most likely be a common scenario in the development process.
This would require installing haddock3 via a clone first and then a pip install .
There was a problem hiding this comment.
The situation you are describing is not yet covered in this repository, in it's current state its already taking from pypi.
This PR simply makes it so it can recover a specific version rather than the latest, in another PR I can add logic to pull a specific commit from the haddock3 repository instead of pypi.
There was a problem hiding this comment.
and also here -> haddocking/haddock-runner#196
this comment made me realize haddock-runner was not catching version changes mid-run, so I also updated it there.
2 participants
I noticed this repository was not tracking any versions of the datasets and not even of haddock3 - this means that there is no way to guarantee that running it today and running it next month will produce the same results. This means in its current state there is almost no reproducibility.
This PR adds a mechanism to control which (commit) versions of the datasets are fetched and also which version of haddock3 and of haddock-runner are being used. I did this by adding a
clone_pinned()helper function in_common.sh.Considering the changes I've done in #3 (adding
run.sh/setup.sh/analyse.sh) I wanted to keep the same lean interface for users also in regards to the versions -so to facilitate the switching of versions I added aversions.envfile that will define which version/commit of each it should use.I added this file so that it's not necessary to dig into scripts to identify which is pulling which version of what, thus giving a singular simple source of truth in regards to the versioning.
Additionally, for the
protein_shapethe retrieval of the dataset was being done via a python script instead of the samesetup.shpattern of other systems - so I also updated it to be in line with the others.