Data & Tables · PRACTICAL GUIDE
Deduplicate CSV records by the right columns
Choose whole-row or key-column matching and understand which record survives.
Define what counts as the same record
Whole-row duplicates have identical values in every column. Key duplicates share selected fields even when other values differ. The right choice depends on the data model: a customer ID may identify a person, while an order ID identifies a transaction.
Never discard rows merely because a name looks similar. Shared names and repeated transactions can be legitimate.
An ID with a changed address
id,address 007,Old Street 008,Oak Road 007,New Street
Whole-row deduplication keeps all three records. Keying on column 1 with “First” keeps 007 at Old Street. “Last” keeps 007 at New Street and retains the surviving records in their last-occurrence order: 008, then 007.
The latter may be appropriate for a chronological export, but only if the export’s ordering is actually meaningful.
Use composite keys safely
A record can require two columns to identify it, such as customer_id and product_id. Enter their one-based positions as 1,2. The tool compares a tuple of strings, not a delimiter-joined string, so punctuation inside one field cannot impersonate a field boundary.
Blank values, letter case and surrounding whitespace remain significant. Clean them deliberately at the source if your system considers them equivalent.
Verify the discarded count
Inspect how many rows were removed and compare representative records. Use the viewer to confirm that the surviving values are the intended ones. Then use the column editor if the importer needs a narrower schema.
Keep the original export. Deduplication is not a database constraint or an ongoing synchronization process; future files can reintroduce the same keys. Sensitive real records are unnecessary when reporting a reproducible issue—replace them with a small synthetic example.
When not to deduplicate
Do not deduplicate event logs, transaction tables or any store where repeated keys are legitimate occurrences. Do not deduplicate before you understand the grain of the table: two rows with the same customer_id may be two contacts, not a mistake. When in doubt, run whole-row first, inspect what remains, then consider a narrower key only if true duplicates still exist.
Also skip deduplication as a substitute for fixing the system that produced duplicates. If an integration double-writes rows, deduplicating the export hides the bug until the next run.
Composite key pitfalls
Keys built from concatenated strings can collide when a field contains the delimiter you joined with—another reason this tool compares tuples of fields rather than a joined string. Keys that include a timestamp column almost never match because timestamps differ. Keys that include a free-text comment column almost always match only exact copies.
Choose columns that are stable identifiers in the source system. Test the key on a sample: if more rows disappear than you can explain, widen the key or switch to whole-row mode.
Documenting what was removed
Before discarding anything, export or screenshot: total input records, retained count, removed count, the key columns used, and first/last retention setting. Store a copy of removed rows if they might be needed for audit. A deduplication that cannot be explained later is a data-loss incident waiting to happen.
After deduplication, re-run the same key check on a future export. If duplicates reappear, the upstream process has not been fixed—the cleaned file is a snapshot, not a constraint.
Put it into practice
Choose the tool that matches your next step. Keep an original copy and inspect the result before using it elsewhere.
Remove duplicate CSV rowsRemove duplicate records by selected fields.Data & TablesCSV column editorSelect, rename and reorder fields for an import.Data & TablesCSV viewer and validatorInspect a file before importing it.Data & Tables