feat(nl): add dynamic graph fallback for un-cached topic and peer group members - #6590
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a dynamic fallback mechanism for the Data Commons Platform to handle custom and UN topics that are not present in the static Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces fallback mechanisms in topic.py to dynamically fetch member and parent properties when they are missing from the TOPIC_CACHE. It also updates _prop_val_ordered to properly split and deduplicate ordered property values, and adds comprehensive unit tests to verify these fallback behaviors. The review feedback suggests deduplicating the list of missing nodes before making API calls in _members_raw and _parents_raw to prevent redundant network requests.
ac43e29 to
bc990f5
Compare
…od following the flag for other env setup (datacommonsorg#6548) This PR add the feature flag in prod environment and default to false When false, we will read default value 0.7 from the code, When true, we will set config value to be 0.6 based on test result https://docs.google.com/spreadsheets/d/12d9BPJagWS5z2WA8En9JY71TyGrCYpIBQ4LqI0JJ2v4/edit?gid=1132426848#gid=1132426848
The gemini 3 feature flag was use when we roll up from 2.5 to 3, it's no longer needed, all non prod env has removed this feature flag
…6593) ## Description The README.md in `server/routes/redirects/README.md` that describes how to update redirect mapping is incorrect/out-of-date. While the description of how to update the mapping (via the `gcloud storage cp`) is correct, the description of when those changes are picked up by production is not. The file states that production reads this redirect mapping on each redirection call, resulting in the immediate pickup of these changes. This is incorrect. The bucket is read when the Flask server starts (in `__init__.py`). This PR updates the README.md to correct the description.
…etect-and-fulfill
…_topic_resolution
gmechali
left a comment
gmechali
left a comment
There was a problem hiding this comment.
Thanks Christie for threading this needle, can't imagine how delicate and annoying this must have been, and to do it on such a short timeline 👏
Adding this code now will bring in some more urgency on cleaning this up for Base DC website longer term. Hopefully post dcp public preview we can take this back to attention because it looks really really brittle throughout. :(
4 participants
Summary
In Data Commons Platform (DCP), custom datasets and custom topics are ingested dynamically into Spanner without generating static
topic_cache.jsonfiles.Previously:
_members()inserver/lib/nl/common/topic.pyonly checkedTOPIC_CACHE. When a dynamic custom topic was queried,get_members()returned empty, terminating topic traversal.is_topic,is_svg,is_svpg,is_sv) relied on hardcodeddc/topic/orc/topic/string prefixes. Custom topic DCIDs (e.g.custom/topic/DisplacedPersonsor UN topics) were misclassified as raw Statistical Variables, bypassing_topic_chart_vars()and failing observation existence checks.Changes
topic.py:TOPIC_CACHE(or returns empty), falls back to calling_prop_val_ordered(node, prop + 'List')to retrieverelevantVariableList/memberListfrom Cloud Spanner / Mixer._members_raw()and_parents_raw()viafetch.raw_property_values().sv_listacross provenances with set deduplication in_prop_val_ordered().VarCandidatesinshared/lib/detected_variables.pywithsv2typesto capturetypeOfandtypesmetadata returned from/v2/resolve.sv2typesthroughUtteranceto explore fulfillment.enable_schema_driven_topic_resolutionfeature flag (defaults tofalseinproduction.jsonfor Base DC safety; auto-enabled on DCP container startup viaupdate_dcp_flags.py).is_topic,is_svg,is_svpg, andis_svto checksv2typesand generalized/topic/,/svg/, and/svpg/namespace patterns when the flag is enabled.explore/topic.pyto route custom topics to_topic_chart_vars().get_sv_name()to strip namespace prefixes (re.sub(r'^[^/]+/(topic|svpg|g)/', '', sv)) for uncurated custom topics.server/tests/lib/nl/topic_test.pycovering fallback traversal, schema-driven classification, flag toggling, and serialization roundtrips.Testing
uv run --project server --group test python3 -m pytest server/tests/lib/nl/(all 349 passed).calinc-devwith imagecalinc-topics-03.