Data Cleaning Auto Report Pipeline

Dirty data leads to broken pipelines and unreliable analysis. This skill provides a structured framework to profile, clean, and validate tabular datasets without compromising data integrity. It replaces ad-hoc regex and manual spreadsheet editing with a repeatable, audit-logged process.

What it does

• Data profiling generates row/column counts, inferred types, missingness percentages, and outlier detection.
• Conservative cleaning standardizes categories, trims whitespace, and parses dates/numbers using explicit business rules.
• Audit logging records every transformation to ensure every change is reversible and documented.
• Data validation performs before-and-after metric comparisons to verify row-count reconciliation and schema integrity.
• Quarantine management isolates suspicious rows instead of deleting them to prevent silent data loss.

How it works

1. Intake to identify the source file, unique keys, sensitive fields, and required output formats.
2. Profile the raw data to identify inconsistencies and possible errors without making changes.
3. Plan the cleaning rules including the logic, rationale, and expected impact for user approval.
4. Execute transformations using a bundled Python script or manual logic, preserving the original source file.

Frameworks & tools

This skill utilizes a bundled Python 3.8+ script for automated processing of CSV and TSV files. It handles encoding issues, date formatting, and category mapping using standard libraries or common data tools like pandas and openpyxl when available.

Why this beats prompting it yourself

Generic prompts often lead to "hallucinated" data cleaning where the LLM silently drops rows or guesses missing values. This skill enforces a read-only source policy and creates a verifiable audit trail that distinguishes facts from assumptions.

Use cases

• Auditing messy CRM exports before importing them into a new system.
• Standardizing inconsistent date formats and category labels in monthly sales reports.
• Deduplicating customer lists using deterministic keys and quarantine logs.
• Generating automated data quality reports for stakeholders to highlight unresolved risks.

Known limitations

Does not support predictive modeling or automated database migrations. Very large files may require sampling during the profiling phase. Performance depends on the availability of a Python execution environment.