Batch CSV Pipeline¶
UC-ING-012: Detect CSV Batch Drop¶
Purpose: Watch for new CSV files from LIS/HIS.
| Property | Value |
|---|---|
| Actor | CSV Watcher Service |
| Trigger | File drop in src/data/csv/ |
| Priority | P1 |
Main Success Scenario:
1. Detect file `lab_results_20240101.csv`
2. Validate filename timestamp
3. Move to `processing/`
4. Stream rows to CSV Validation Queue
Acceptance Criteria: 1. [ ] Handles multi-GB files via streaming 2. [ ] Detects duplicate files
UC-ING-013: Validate CSV Schema¶
Purpose: Ensure CSV structure matches expected format.
| Property | Value |
|---|---|
| Actor | CSV Validator |
| Trigger | Row in Validation Queue |
| Priority | P1 |
Main Success Scenario:
1. Check header row matches schema
2. Validate column types (Date, Number, String)
3. Enqueue valid rows to Transformation Queue
Acceptance Criteria: 1. [ ] Rejects malformed rows but continues processing file 2. [ ] Generates error report for rejected rows
UC-ING-014: Transform CSV Row¶
Purpose: Convert CSV row to internal data model.
| Property | Value |
|---|---|
| Actor | CSV Transformer |
| Trigger | Row in Transformation Queue |
| Priority | P1 |
Main Success Scenario:
1. Map CSV columns to `LabResult` or `Patient` object
2. Normalize values (units, codes)
3. Trigger update pipeline (UC-ING-005)
Acceptance Criteria: 1. [ ] Handles missing optional fields 2. [ ] Logs transformation errors