merge messy spreadsheets

The problem

Standard deduplication tools fail when rows aren't byte-identical, leaving you with messy data full of typos, abbreviations, and reordered names. Manually reviewing thousands of rows to catch "Jon Smith" vs "Jonathan Smith" is slow and prone to human error.

What it does

• Consolidates multiple .xlsx, .xlsm, and .csv files into a single unified schema while preserving source file and row metadata.
• Executes a four-pass matching cascade: Exact, Normalized (diacritics/case), Fuzzy (typos/reordering), and Semantic (synonyms/paraphrases).
• Implements mandatory approval gates for non-exact matches, ensuring you review and sign off on fuzzy or semantic clusters before merging.
• Generates a comprehensive audit trail sheet documenting every merge decision, similarity score, and the pass that triggered the match.
• Auto-selects the most complete row (fewest nulls) as the canonical record while logging all dropped duplicates.

Frameworks & tools

Python, Pandas for data orchestration, RapidFuzz for string similarity, and Sentence-Transformers for semantic embedding analysis.

Why this beats prompting it yourself

Writing a script to handle multi-pass deduplication is complex, especially when balancing recall and precision. This skill provides a tested architecture that handles edge cases like Unicode normalization and diacritic folding, while preventing data loss through structured audit logs that simple LLM prompts cannot reliably maintain.

Use cases

• Merging disparate CRM exports where contact names and company titles vary slightly.
• Cleaning up inventory lists with inconsistent naming conventions and abbreviations.
• Deduplicating survey responses or form submissions containing paraphrased text.
• Consolidating legacy spreadsheets into a single source of truth for database migration.

Known limitations

This skill is designed for stacking and deduplicating rows. It does not perform relational joins or VLOOKUP-style operations between different datasets.