[MLopez-Ibanez]

Hi @Dvermetten @olafmersmann @ttusar @gaplopes @light-weaver @adbjesus @apguerreiro

Since our meeting in Coimbra, @fieldsend and myself have been slowly working on implementing the archiver API that we discussed in the meeting.

We have an incomplete WIP implementation that does the most basic operations.

We are looking for feedback on the API, testcases, examples, suggestions for fixes and help in improving / completing it.

API and examples

Benchmarks

[github-actions]

📚 Python docs preview for this PR

[codecov]

Codecov Report

❌ Patch coverage is 79.84645% with 420 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.66%. Comparing base (a123dc9) to head (35ede98).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #82      +/-   ##
==========================================
+ Coverage   77.87%   78.66%   +0.79%     
==========================================
  Files          70       82      +12     
  Lines        7594     9306    +1712     
  Branches     1478     1844     +366     
==========================================
+ Hits         5914     7321    +1407     
- Misses       1314     1458     +144     
- Partials      366      527     +161     

Flag	Coverage Δ
R	71.23% <65.24%> (-1.02%)	⬇️
python	77.60% <80.92%> (+2.68%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
C	74.35% <74.20%> (+0.32%)	⬆️
R	81.59% <ø> (ø)
Python tests	99.77% <99.35%> (-0.23%)	⬇️
Python src	82.90% <85.92%> (+1.09%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📚 R docs preview for this PR

[light-weaver]

Thanks for the work @MLopez-Ibanez and @fieldsend!

A few comments:

• Based on BaseUnboundedArchive it looks like both add() and add_many() default to adding one solution to the archive at a time, looping in Python. Based on our experience with NonDominatedArchive in DESDEO (impl), this process is significantly slower compared to adding in batches as already noted in your comment in add_many(). We did some tests with IBEA and SMS-EMOA, and adding an archive makes using SMS-EMOA (which generates one solution at a time) makes using that algorithm borderline impossible.
• Naive batch processing would be simple enough to do by just implementing what's stated in the comment in add_many(). Something similar could be done in add() by making that method lazy instead. Instead of eagerly checking whether new points should be added to the archive, new points can be put into a separate buffer instead until it reaches a predetermined size or if the user accesses the archive, in which case we can basically run add_many() with this buffer. A more advanced version of this could change the buffer size dynamically based on the size/quality of the current archive, but that would probably require a study. Could make for a nice conference paper.
• Considering that the archive object is also an iterator, I wonder what would happen if one tries to add a solution to the archive while inside a loop that iterates over that archive. There should probably be guards against that just in case.

[MLopez-Ibanez]

• Based on BaseUnboundedArchive it looks like both add() and add_many() default to adding one solution to the archive at a time, looping in Python. Based on our experience with NonDominatedArchive in DESDEO (impl), this process is significantly slower compared to adding in batches as already noted in your comment in add_many(). We did some tests with IBEA and SMS-EMOA, and adding an archive makes using SMS-EMOA (which generates one solution at a time) makes using that algorithm borderline impossible.

Yes, this can definitely be improved and we have some ideas. However, it will also be interesting to understand what you mean by "impossible". It takes less than $10^{-5}$ seconds to add one point to the archive when you have $2^{16}$ points already, as shown in the Benchmarks.

Could you suggest how to extend https://github.com/multi-objective/moocore/blob/18d204abb1eccb09c8b448488e473d4112d51817/python/benchmarks/bench_archive_unbounded.py to also benchmark NonDominatedArchive in DESDEO?

If you tell me how to benchmark it, I can run the benchmark and add the results to this PR.

• Naive batch processing would be simple enough to do by just implementing what's stated in the comment in add_many(). Something similar could be done in add() by making that method lazy instead. Instead of eagerly checking whether new points should be added to the archive, new points can be put into a separate buffer instead until it reaches a predetermined size or if the user accesses the archive, in which case we can basically run add_many() with this buffer. A more advanced version of this could change the buffer size dynamically based on the size/quality of the current archive, but that would probably require a study. Could make for a nice conference paper.

Sure, there are many improvements that can be done to make the code faster. I will add a TODO-archiving file. I am also happy for people to fork the code and experiment with it. As mentioned above, there is a benchmark file that should be very easy to extend and run.

However, it is pretty fast already, so before making it even faster, we would like to know if it is fast enough for your purposes and if the API is OK or should be renamed/extended.

• Considering that the archive object is also an iterator, I wonder what would happen if one tries to add a solution to the archive while inside a loop that iterates over that archive. There should probably be guards against that just in case.

Good point! I think I know how to handle this by adding a "revision" field to the tree and iterator and if there is a mismatch, the iterator is invalid. I'll implement this next.

[MLopez-Ibanez]

• Considering that the archive object is also an iterator, I wonder what would happen if one tries to add a solution to the archive while inside a loop that iterates over that archive. There should probably be guards against that just in case.

Good point! I think I know how to handle this by adding a "revision" field to the tree and iterator and if there is a mismatch, the iterator is invalid. I'll implement this next.

This is now implemented and tested. Modifying the archive while iterating raises an exception.

[light-weaver]

Yes, this can definitely be improved and we have some ideas. However, it will also be interesting to understand what you mean by "impossible". It takes less than 10 − 5 seconds to add one point to the archive when you have 2 16 points already, as shown in the Benchmarks.

It was a bad decision on my part regarding datastructures. We were using polars dataframes in other parts of the framework so I decided to keep things consistent and used them for archivng purposes too. After doing some basic profiling, most of the time is actually being spent on constructing these.

I also tried comparing our implementation against yours using the benchmarking script you pointed out. I had to lower the budget because I didn't want to wait too long for DESDEO. Based on the result, I don't think it is necessary to add this comparison to the official docs of moocore. We will just switch our implementation to moocore's as soon as it's merged to main.

Minor suggestion for the plots/docs. Change the line and marker style for filter_dominated so that it is clearer that it is different compared to the other two.

[MLopez-Ibanez]

I also tried comparing our implementation against yours using the benchmarking script you pointed out. I had to lower the budget because I didn't want to wait too long for DESDEO. Based on the result, I don't think it is necessary to add this comparison to the official docs of moocore. We will just switch our implementation to moocore's as soon as it's merged to main.

Did you normalise the time by the number of points like it is done for the other archives, so that the time shown is the average time to add a single point?