Deduplicate CSV records by the right columns

Define what counts as the same record

Whole-row duplicates have identical values in every column. Key duplicates share selected fields even when other values differ. The right choice depends on the data model: a customer ID may identify a person, while an order ID identifies a transaction.

Never discard rows merely because a name looks similar. Shared names and repeated transactions can be legitimate.

An ID with a changed address

id,address
007,Old Street
008,Oak Road
007,New Street

Whole-row deduplication keeps all three records. Keying on column 1 with “First” keeps 007 at Old Street. “Last” keeps 007 at New Street and retains the surviving records in their last-occurrence order: 008, then 007.

The latter may be appropriate for a chronological export, but only if the export’s ordering is actually meaningful.

Use composite keys safely

A record can require two columns to identify it, such as customer_id and product_id. Enter their one-based positions as 1,2. The tool compares a tuple of strings, not a delimiter-joined string, so punctuation inside one field cannot impersonate a field boundary.

Blank values, letter case and surrounding whitespace remain significant. Clean them deliberately at the source if your system considers them equivalent.

Verify the discarded count

Inspect how many rows were removed and compare representative records. Use the viewer to confirm that the surviving values are the intended ones. Then use the column editor if the importer needs a narrower schema.

Keep the original export. Deduplication is not a database constraint or an ongoing synchronization process; future files can reintroduce the same keys. Sensitive real records are unnecessary when reporting a reproducible issue—replace them with a small synthetic example.