Data Quality Remediation Process

1. Issue Identification
When a potential data quality issue is identified during analysis, ETL development, reporting, or partner review, document the issue.

2. Create Task in Data Quality Board
Create a task on the NSWERS Data Quality board:
https://nswers.teamhood.com/NSWERS/Board/DAQU?view=LIST

The task should include:

• Clear description of the issue
• A reproducible example (query, record sample, or analytic output demonstrating the problem)
• Affected dataset(s), institution(s), and time period if known
• Initial hypothesis about the cause if one exists

3. Monthly Review and Triage
Tasks are reviewed monthly. During review I will:

• Confirm the issue is reproducible
• Determine scope and impact
• Prioritize remediation work
• Assign responsibility for follow-up
• Label appropriate items as WONTFIX
  • If an item is not impactful, we will label as a data quirk and log them.
  • https://ts.nswers.org/kb/research/currently-known-data-issues/

4. Remediation Investigation
Typical remediation actions may include:

• Inspect raw submitted data in GCC to determine if the issue originates in partner submissions
• Review ETL logic and transformations for mapping or processing errors
• Compare against data specifications to determine if the issue reflects a spec ambiguity or misinterpretation
• Validate against historical submissions to determine if the issue is new or longstanding
• Check for upstream system changes at partner institutions that could explain anomalies
• Conduct targeted data profiling to assess scope (counts, distributions, missingness patterns)

5. Remediation Actions
Depending on the findings, remediation may involve:

• Correcting ETL logic and reprocessing affected data
• Requesting a corrected resubmission from the partner institution
• Updating validation rules or QA checks
• Clarifying or revising data specifications
• Adding automated checks to prevent recurrence

6. Documentation and Closure
Once resolved:

• Document the resolution in the task
• Note any specification changes or ETL updates
• Close the task on the board