Repository navigation
Default output_format to "dataframe" #1115
Description
I propose that the several functions that take the ouput_format parameter and produce lists change their defaults to return dataframes instead of dictionaries (e.g. list_datasets).
I think originally we had loose integration with pandas and it was an optional dependency, but since it's a required dependency now I don't see a reason to provide dicts as default over dataframes. I'd argue most people prefer dataframes in interactive sessions, and it makes sense to minimize the amount of writing people have to do in those (also plenty of people might prefer dataframes in scripts too).
Activity
mfeurer commentedon Oct 27, 2021
I fully agree to this suggestion. Indeed, dataframes were added later and have matured substantially since then and I guess we can consider handling them standard knowledge for machine learners.
amastruserio commentedon Apr 18, 2022
I took a look at this to see how the proposed changes would behave and it appeared to lead to a few issues with some functions not fully integrated with dataframes yet. Many of these errors were Attribute errors centering around flow_id, task_id, and run_id, and appear to be the result of some functions expecting objects other than dataframes.
It also lead to repeated unit test failures involving Key_Errors, namely with flow_id, task_id, and uploader.
There were also some assertion errors that wanted to assert some object was a dict, but I imagine these assertions would be irrelevant with dataframes as the default.
I have a complete list of unit tests that failed after implementing these changes that I can provide as well, but I think there is still more integration with dataframes needed from the codebase before being able to totally implement this change.
PGijsbers commentedon Apr 19, 2022
In that case it might be best to do this incrementally. First we'd get a PR that changes the default, but makes internal calls use the explicit
output_format="dict"so you can change the external behavior without making many internal changes. We could then further refactor the internal handling of the data in separate PRs where it makes sense.varshneydevansh commentedon Aug 9, 2023
Hi!
I am not sure how I ended up here, but I looked at what you guys are doing and It's pretty remarkable. I find this challenge to be my starter in this project and would love to contribute more. =)
PGijsbers commentedon Aug 10, 2023
Hi @varshneydevansh, thank you! We already started working on this, as can be seen in #1258. It's already assigned to me since it's pretty involved. However, if you would like to do this instead, here is basically what needs to be done:
The next step is to actually make the change and make
output_format='dataframe'the default as well as removingoutput_format='dict'support wherever the message indicated it. You should be able to find all instances of this in# TODO: [0.15]. Then, make sure to update any unit tests necessary and update the documentation.If you'd like something smaller, I could recommend you to work on #1142 or #1213 (the latter requires some Github Actions knowledge, but no knowledge of our code-base).
PGijsbers commentedon Oct 14, 2024
We missed this in the 0.15 release, adding it to 0.16
defaults are set to dataframe as part of first step towards issue ope…
Roshangoli commentedon Apr 29, 2025
Hi! I’m new to open source and was excited to contribute to this issue as my first task, but I see it’s already being handled in PR #1390. Could I please be recommended a similar beginner-friendly issue to get started with? Thank you! 🙌
Omswastik-11 commentedon Nov 16, 2025
Hi @PGijsbers !!
is this issue open ? Should I work ?
anisha-04 commentedon Jul 23, 2026
I plan to work on a narrowly scoped solution for this issue and add deterministic tests covering the reported behavior.
AtharvaManale commentedon Oct 11, 2026
Hi @PGijsbers
I investigated this issue against the current codebase and traced the implementation in your pr #1258.
In PR #1258 I can see the conversion of
output_format="dict"tooutput_format="dataframes"for return values has been implemented and also handled the domino effect of that change which affected multiple unit tests and functions related to datasets, tasks, flows, evalutions, runs and studies.I suggest closing this issue if there are no further requirements.
PGijsbers commentedon Oct 12, 2026
Thanks for spotting this issue was already (mostly) done. I had a look myself and it looks like
list_setupscurrently still defaults to dictionary output. It was probably missed because it's referenced as "object" rather than "dict".arcwiser commentedon Oct 12, 2026
I’ll work on the remaining list_setups gap identified above: make its public default DataFrame output, keep internal callers explicit where needed, and update focused tests and documentation.
AtharvaManale commentedon Oct 12, 2026
Hi @PGijsbers
I have reviewed this pr #1769 by @arcwiser regarding the missing list_setups default_output = ''object" to "dataframe".
Also tested and ran the following test cases locally, it is a great pr to take a look.