Skip to content

Prevent users from initiating export of unfeasibly large AnVIL datasets #4953

Description

@hannes-ucsc

In AnVIL, disable the export to Terra functionality for the four datasets listed here with an explanation as to why. That comment also links to the Slack thread with stakeholders.

To summarize, the only problematic datasets are

* `ANVIL_eMERGE_GWAS_HMB_MDS`

* `ANVIL_eMERGE_GWAS_GRU`

* `ANVIL_eMERGE_GWAS_HMB_GSO`

* `ANVIL_eMERGE_GWAS_HMB_PUB_GSO`

As we established above, these are the same datasets that also cause aggregation issues (#8017).

And as I surmise above, a permanent solution would likely require a substantial revamp of the verbatim PFB manifest generation. We were told that Terra is unable to import three of the four datasets, so there wouldn't be any point in spending time on that.

We should, however, prevent users from even attempting to export these four datasets to Terra. This would reduce user frustration (each attempt takes 15min before it eventually fails) and save internal team resources (each failed attempt raises an alarm that the operator has to spend 5-10 min investigating). The best place to do implement these preventative measures is the Data Browser. While Azul could be modified to fail early, refusing to even start the futile 15min attempt at manifest generation, the Data Browser could disable the respective button and display an explanation as to why it is disabled.

I proposed this on Slack to stakeholders.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

escalate[process] Needs discussion with stakeholders / PI

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions