Add new Commerce EDA USASpending import

statvar_imports/commerce_eda_usaspending/Investmentmetadata.csv

@@ -0,0 +1,12 @@
+parameter,value
+url,https://www.usaspending.gov/
+description,EDA Investments by Applicant State and Program (sourced from USASpending.gov)
+#place_type,AdministrativeArea1
+#places_within,country/USA
+#start_date,2012
+#end_date,2026
+release_frequency,Yearly
+process,
+comments,
+output_columns,"observationAbout,observationDate,variableMeasured,value,unit,scalingFactor,measurementMethod"
+header_rows,1

statvar_imports/commerce_eda_usaspending/Investmentpvmap.csv

@@ -0,0 +1,72 @@
+key,prop,value,p1,v1,p2,v2,p3,v3,,
+Place,,,observationAbout,geoId/{Place},,,,,
+Year,observationDate,{Number},,,,,,,,
+Value,value,{Number},populationType,Investment,measuredProperty,amount,unit,USDollar
+Alabama,observationAbout,geoId/01
+Alaska,observationAbout,geoId/02
+Arizona,observationAbout,geoId/04
+Arkansas,observationAbout,geoId/05
+California,observationAbout,geoId/06
+Colorado,observationAbout,geoId/08
+Connecticut,observationAbout,geoId/09
+Delaware,observationAbout,geoId/10
+District of Columbia,observationAbout,geoId/11
+Florida,observationAbout,geoId/12
+Georgia,observationAbout,geoId/13
+Hawaii,observationAbout,geoId/15
+Idaho,observationAbout,geoId/16
+Illinois,observationAbout,geoId/17
+Indiana,observationAbout,geoId/18
+Iowa,observationAbout,geoId/19
+Kansas,observationAbout,geoId/20
+Kentucky,observationAbout,geoId/21
+Louisiana,observationAbout,geoId/22
+Maine,observationAbout,geoId/23
+Maryland,observationAbout,geoId/24
+Massachusetts,observationAbout,geoId/25
+Michigan,observationAbout,geoId/26
+Minnesota,observationAbout,geoId/27
+Mississippi,observationAbout,geoId/28
+Missouri,observationAbout,geoId/29
+Montana,observationAbout,geoId/30
+Nebraska,observationAbout,geoId/31
+Nevada,observationAbout,geoId/32
+New Hampshire,observationAbout,geoId/33
+New Jersey,observationAbout,geoId/34
+New Mexico,observationAbout,geoId/35
+New York,observationAbout,geoId/36
+North Carolina,observationAbout,geoId/37
+North Dakota,observationAbout,geoId/38
+Ohio,observationAbout,geoId/39
+Oklahoma,observationAbout,geoId/40
+Oregon,observationAbout,geoId/41
+Pennsylvania,observationAbout,geoId/42
+Rhode Island,observationAbout,geoId/44
+South Carolina,observationAbout,geoId/45
+South Dakota,observationAbout,geoId/46
+Tennessee,observationAbout,geoId/47
+Texas,observationAbout,geoId/48
+Utah,observationAbout,geoId/49
+Vermont,observationAbout,geoId/50
+Virginia,observationAbout,geoId/51
+Washington,observationAbout,geoId/53
+West Virginia,observationAbout,geoId/54
+Wisconsin,observationAbout,geoId/55
+Wyoming,observationAbout,geoId/56
+American Samoa,observationAbout,geoId/60
+Federated States of Micronesia,observationAbout,geoId/64
+Guam,observationAbout,geoId/66
+Northern Mariana Islands,observationAbout,geoId/69
+Palau,observationAbout,geoId/70
+Puerto Rico,observationAbout,geoId/72
+Marshall Islands,observationAbout,geoId/68
+U.S. Virgin Islands,observationAbout,geoId/78
+Economic Adjustment Assistance,investmentProgram,EconomicAdjustmentAssistance
+Planning,investmentProgram,ProgramPlanning
+Public Works,investmentProgram,PublicWorks
+Regional Innovation Strategies,investmentProgram,RegionalInnovationStrategies
+Technical Assistance,investmentProgram,TechnicalAssistance
+Research and National Technical Assistance,investmentProgram,ResearchNationalTechnicalAssistance
+Trade Adjustment Assistance for Firms,investmentProgram,TradeAdjustmentAssistance
+Research,investmentProgram,Research
+Total,investmentProgram,

statvar_imports/commerce_eda_usaspending/README.md

@@ -0,0 +1,36 @@
+# Commerce_EDA (USASpending Source)
+
+This importer fetches and processes U.S. Economic Development Administration (EDA) investment datasets dynamically from USASpending.gov.
+
+- **Source**: [USASpending.gov API](https://api.usaspending.gov/)
+- **Place Type**: U.S. States and Territories (`AdministrativeArea1`)
+- **Time Coverage**: FY 2012 to Present (dynamically fetched up to current year + 1)
+- **Import Type**: Automated API-based import
+- **Release Frequency**: Continuous / On-demand
+
+## Pipeline Steps
+
+### 1. Preprocessing (`process.py`)
+Run the python script to dynamically download all awards from USAspending.gov and pivot/structure them into a clean tall format:
+```bash
+python3 process.py
+```
+This writes the formatted data to `output/Investment_cleaned.csv` with columns:
+`Place,State or Territory / EDA Program,Year,Value`
+
+### 2. Statistical Variable Processing
+Run the Data Commons `stat_var_processor.py` to generate the final MCF and CSV files for import:
+```bash
+python3 ../../tools/statvar_importer/stat_var_processor.py \
+  --input_data=output/Investment_cleaned.csv \
+  --pv_map=Investmentpvmap.csv \
+  --config_file=Investmentmetadata.csv \
+  --output_path=output/Investment_output \
+  --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
+  --output_counters=counters/Investment_counters.csv
+```
+
+## Generic Updates for Coming Years
+- The preprocessing script (`process.py`) dynamically queries the USASpending.gov API up to the current fiscal year + 1.
+- `Investmentpvmap.csv` is configured in tall format to resolve years and values generically using `{Number}` and `{Place}` wildcard templates.
+- Start and end dates are commented out in `Investmentmetadata.csv` to ensure the importer remains entirely year-agnostic and never needs manual updates as new year records become available.

statvar_imports/commerce_eda_usaspending/counters/Investment_counters.csv

@@ -0,0 +1,72 @@
+"key","value"
+"1:process_input_generated-svobs",2536
+"1:process_input_generated-svobs-Investment_cleaned.csv",2536
+"1:process_input_input-data-rows",2536
+"1:process_input_input-files-processed",1
+"1:process_input_input-header-rows",1
+"1:process_input_input-rows-processed",2537
+"1:process_input_input-sections",1
+"1:process_input_num-rows-output/Investment_cleaned.csv",2789
+"1:process_input_process-mem",4184449024
+"1:process_input_process-mem-rss",1269280768
+"1:process_input_process-time-sys-secs",1.36
+"1:process_input_process-time-user-secs",54.7
+"1:process_input_process_elapsed_time",2.145333727991581
+"1:process_input_process_remaining_time",2.5715253712346864
+"1:process_input_processed",2537
+"1:process_input_processing-input-rows-rate",1182.7543333682972
+"1:process_input_processing-input-rows-rate_output/Investment_cleaned.csv",1183.4799330170536
+"1:process_input_processing-time-seconds",2.1443347580207046
+"1:process_input_processing-time-seconds_output/Investment_cleaned.csv",2.1443347580207046
+"1:process_input_processing_rate",1182.5665941378215
+"1:process_input_start_time",163578.845437771
+"1:process_input_total",5578
+"1:process_input_warning-unresolved-value-ref",2574
+"2:prepare_output_process-mem",4184449024
+"2:prepare_output_process-mem-rss",1269280768
+"2:prepare_output_process-time-sys-secs",1.36
+"2:prepare_output_process-time-user-secs",54.71
+"2:prepare_output_process_elapsed_time",0.012940960000175982
+"2:prepare_output_processed",0
+"2:prepare_output_start_time",163580.990893019
+"3:write_statvar_mcf_process-mem",4184449024
+"3:write_statvar_mcf_process-mem-rss",1269903360
+"3:write_statvar_mcf_process-time-sys-secs",1.36
+"3:write_statvar_mcf_process-time-user-secs",54.73
+"3:write_statvar_mcf_process_elapsed_time",0.26824057200681417
+"3:write_statvar_mcf_processed",0
+"3:write_statvar_mcf_start_time",163581.004174409
+"4:write_svobs_csv_process-mem",4184449024
+"4:write_svobs_csv_process-mem-rss",1269903360
+"4:write_svobs_csv_process-time-sys-secs",1.36
+"4:write_svobs_csv_process-time-user-secs",54.78
+"4:write_svobs_csv_process_elapsed_time",0.05149816898736172
+"4:write_svobs_csv_processed",0
+"4:write_svobs_csv_start_time",163581.272647261
+"PVs-matched",55
+"dropped-output-statvars-mcf",8
+"error-input-ignore-node-diff",8
+"error-node-merge-conflict",8
+"existing-nodes-from-api",8
+"existing-statvars-from-mcf",2536
+"generated-statvars",2536
+"generated-unique-statvars",8
+"nodes-with-diff",8
+"output-svobs-csv-rows",2536
+"output-svobs-unique-observationAbout",58
+"output-svobs-unique-observationDate",15
+"output-svobs-unique-typeOf",1
+"output-svobs-unique-unit",1
+"output-svobs-unique-value",1912
+"output-svobs-unique-variableMeasured",8
+"process-mem",4183400448
+"process-mem-rss",1268830208
+"process-time-sys-secs",1.36
+"process-time-user-secs",52.56
+"process_elapsed_time",0.004311160005070269
+"processed",0
+"pvs-added",8
+"pvs-added-definition",8
+"start_time",163578.844831171
+"svobs-added",2536
+"warning-missing-property-key",106644

statvar_imports/commerce_eda_usaspending/manifest.json

@@ -0,0 +1,31 @@
+{
+    "import_specifications": [
+        {
+            "import_name": "Commerce_EDA_USASpending",
+            "curator_emails": [
+                "support@datacommons.org"
+            ],
+            "provenance_url": "https://api.usaspending.gov/",
+            "provenance_description": "EDA Investments by Applicant State and Program, dynamically sourced from USASpending.gov.",
+            "scripts": [
+                "process.py",
+                "../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=output/Investment_cleaned.csv --pv_map=Investmentpvmap.csv --config_file=Investmentmetadata.csv --output_path=output/Investment_output --output_counters=counters/Investment_counters.csv"
+            ],
+            "import_inputs": [
+                {
+                    "template_mcf": "output/Investment_output.tmcf",
+                    "cleaned_csv": "output/Investment_output.csv",
+                    "node_mcf": "output/Investment_output.mcf"
+                }
+            ],
+            "source_files": [
+                "output/Investment_cleaned.csv",
+                "counters/Investment_counters.csv",
+                "Investmentmetadata.csv",
+                "Investmentpvmap.csv",
+                "manifest.json"
+            ],
+            "cron_schedule": "30 05 * * 1"
+        }
+    ]
+}