How to use this tool
- Load the CSV and choose its delimiter and header setting.
- Enter key column positions, such as 1,2, or leave blank for whole-row comparison.
- Choose first or last retention, then check the removed and retained counts.
Whole row versus key columns
Whole-row duplicates have identical values in every column. Key duplicates share selected fields even when other values differ. A customer ID may identify a person; an order ID identifies a transaction. Choose the key that matches your data model—not merely a column that looks unique.
Never discard rows only because a name looks similar. Shared names and repeated transactions can be legitimate. When unsure, run whole-row first and inspect what remains.
Composite keys and exact matching
Enter one-based column positions such as 1,2 for a composite key. Comparison uses a tuple of field strings, not a delimiter-joined string, so punctuation inside one field cannot impersonate a field boundary. Matching is exact: whitespace, case and leading zeros remain significant.
Blank values are part of the key unless you clean them first. Letter case differences create different keys. Clean at the source only when your system treats those values as equivalent.
Which record survives
id,address 007,Old Street 008,Oak Road 007,New Street
Whole-row: all three records remain (no exact row repeats). Key on column 1, “First”: 007 Old Street survives; 007 New Street is removed. “Last”: 007 New Street survives; retained rows follow last-occurrence order (008 then 007) when that setting is used.
First/last is a deliberate choice, not a quality judgment. Chronological exports may make “last” meaningful; unordered dumps may not.
Verify the discarded count
After deduplication, compare removed and retained counts with the viewer. Spot-check representative records. Then narrow the schema with CSV Column Editor if the importer needs fewer columns.
Keep the original export. Deduplication is not a database constraint—future files can reintroduce the same keys. For schema mismatches before merging, align headers first; see CSV Merge.
Limits and safety
Input limits: 5 MiB per file, 50,000 records, 200 columns. Processing stays in the browser. The tool does not infer business importance, fuzzy-match similar names or sync with a remote database.
Decide what makes two records duplicates
Whole-row mode compares every field, so rows that differ in one cell are kept. Key-column mode compares only the selected fields and ignores the rest; a tuple key is the ordered combination of those values. Case, surrounding whitespace and Unicode normalization can be applied before comparing, which changes which rows collide.
Choose the key from the meaning of the data. For a customer export, an email or customer ID may define a duplicate even when the name or address changed. For an event log, identical values in every column may represent two real events, so deduplication would remove valid records. Inspect the file with CSV Viewer before deleting rows.
Which record is kept, and what is lost
Keep-first retains the earliest occurrence in file order. Keep-last retains the latest occurrence and restores the original order afterwards. Neither mode merges the other fields: the retained row wins entirely, so a later row with a corrected value can replace an earlier row under keep-last, while keep-first discards that correction.
Deduplication is destructive. Download the result to a new file and keep the source. If two rows share a key but contain different values, decide which record is authoritative or reconcile the fields with CSV Column Editor before removing anything.
Processing and privacy
CSV Deduplicate compares rows in the page; key columns and retained records stay on the device. Input stays in browser memory until you reset or leave the page. Copy and download run only when you choose them. No tool input is sent to a processing service. Website pages and scripts still make ordinary hosting requests; the privacy policy separates those from tool processing.
Frequently asked questions
Are column positions zero-based?
No. Position 1 means the first column shown in the table.
Will it remove rows that only differ by whitespace?
Not unless you enable trimming as part of the key rule at the source. Matching is exact by default.
Can I undo a deduplication?
Not inside the tool. Keep the original file; re-load it if you need the discarded rows.
What if two records share an ID but different addresses?
Whole-row keeps both. Keying on ID alone discards one according to first/last. Choose deliberately.
Does this work on TSV or pipe files?
Yes—select the correct delimiter and header mode before running deduplication.
Does deduplication merge the duplicate rows?
No. It keeps one record and drops the others; it does not combine or reconcile their field values.
Why were two similar rows not treated as duplicates?
They differ in a compared field, or in case, whitespace or Unicode normalization. Select the right key columns or enable the matching options.
How this tool was checked
CSV Deduplicate was checked with whole-row and key-column modes, tuple keys containing the delimiter, case and whitespace differences, first or last retention and large files. See the testing method or report a failure.