USFEMA_FloodInsuranceClaims

[kartik-s21]

Description

This PR optimizes and stabilizes the data download pipeline for the FEMA NFIP Flood Insurance Claims import.

Key Changes

1. Direct Bulk Download Support:

   • Added bulk_url support to download the full dataset (FimaNfipClaims.csv) directly from OpenFEMA as the primary method, significantly speeding up the download process.
   • Implemented automatic fallback to paginated API download if direct bulk download fails or returns empty.

2. Error Handling & Fallback Visibility:

   • Explicitly raise exceptions upon bulk download failure so that fallback events are properly captured and logged with warnings.
   • Added fatal logging and exceptions on API chunk failures or incomplete downloads to prevent silent partial data ingestion.

3. CSV Chunk Concatenation Fixes:

   • Improved binary chunk merging logic to ensure consistent newline endings across chunks.
   • Guarded chunk concatenation to prevent empty/header-only chunks from injecting blank lines into the merged CSV.
   • Ensured safe temporary directory management (exist_ok=True and cleanup in finally blocks).

4. Testing & Coverage:

   • Added comprehensive unit tests in fema_download_test.py covering:
     • Record count retrieval and error handling.
     • Direct bulk download success flow.
     • Direct bulk download failure with fallback to API pagination.
     • Multi-chunk pagination and header-only chunk merging without blank lines.

Verification

• Validation Report - link
• CRA Report - link
• Differ summary - link (All these deletions are expected and are from Source side changes)

[gemini-code-assist]

Code Review

This pull request introduces a direct bulk download option for the FEMA NFIP claims dataset as a faster and more reliable alternative to API pagination, which remains as a fallback. It also increases the pagination page size, improves error handling, and refines CSV chunk merging. The review feedback highlights two key improvements: raising an exception on bulk download failures to ensure proper logging instead of a silent fallback, and adding a check to prevent writing empty newlines when a downloaded chunk contains only a header.