How to remove duplicate lines
- Paste one complete entry per line.
- Leave matching options off for exact comparison, or choose the differences that should be ignored.
- Select First or Last under Occurrence to keep.
- Select Remove duplicates and check the retained and removed counts.
- Copy result or Download the unique lines. The original input stays available.
A repeated line is one complete entry that matches an earlier comparison key. The tool does not remove repeated words within a sentence, find similar names or merge partial entries. It accepts up to 1 MiB of text and runs in the browser.
How duplicate lines are detected
Default matching is exact and case-sensitive. A line containing Apple differs from apple, and apple with a trailing space differs from apple. Punctuation also matters.
- Ignore letter case: lowercase the comparison key using the tool’s English locale default. Original output spelling is retained.
- Ignore surrounding spaces: trim whitespace from the beginning and end of the comparison key. Internal spaces remain significant.
- Normalize Unicode (NFC): compare canonically normalized keys, allowing a precomposed accent to match its equivalent base-letter-plus-accent spelling.
When enabled together, trimming happens first, NFC normalization second and lowercasing third. These operations change the key used to detect duplicates, not the surviving line. NFC does not remove accents, translate text or make all look-alike characters equal.
Duplicate line remover example
Input
apple pear apple Apple
Result
Default, keep First: apple pear Apple Ignore letter case, keep First: apple pear
The default removes one repeated line; case-insensitive matching removes two. Both keep the first apple spelling. If Ignore surrounding spaces is enabled and the first occurrence includes edge spaces, those spaces remain in the output. To change the visible text as well, clean the list separately.
Keep the first or last occurrence
First retains the first matching entry and preserves the order of those first appearances. Last retains the final matching entry and preserves the order of the surviving last appearances. It does not keep the first position while swapping in the final spelling.
Input
apple pear Apple
Result
Ignore letter case, keep First: apple pear Ignore letter case, keep Last: pear Apple
Choose the mode based on how the list was assembled. Last can retain a later spelling, but the tool has no timestamps and cannot prove the later line is more accurate. If chronology matters, confirm that the pasted order actually represents it.
Blank lines, line endings and whitespace
The processor treats CRLF and CR as newline boundaries. One final newline is a terminator rather than an extra empty entry. Internal blank lines participate in duplicate matching, so repeated empty entries collapse to one retained blank entry.
A line of spaces and an empty line are different under exact matching. With Ignore surrounding spaces enabled, they share an empty comparison key, and the chosen original line survives. This behavior does not make the tool a remove-all-blank-lines utility. Use the line counter to inspect the result if blank entries affect the receiving workflow.
Remove duplicate text online: useful list workflows
Deduplicate a repeated list of labels, pasted filenames or simple identifiers when one line represents one item. Start with exact matching for identifiers whose case might be significant. Enable broader matching only when those differences are irrelevant to your task.
After removing duplicates, alphabetize the list if you need a lookup-friendly order. Sorting is separate because the original sequence may matter. To find entries present in one list but missing in another, use Compare Lists; that is a membership comparison, not just deduplication.
When line deduplication is the wrong tool
Do not use this on prose to remove duplicate words. A repeated sentence can be intentional, and two words on one line are not separate entries. There is no fuzzy matching, spelling correction or meaning-based grouping.
CSV records may contain quoted newlines, so a physical line is not always a complete row. Use CSV deduplication for whole records or selected key columns. Always inspect the surviving entries before replacing an original list, especially when broader matching rules combine differently spelled values.
Processing and privacy
Remove Duplicate Lines compares and reorders the list locally; no line or file leaves the browser. Input stays in browser memory until you reset or leave the page. Copy and download run only when you choose them. No tool input is sent to a processing service. Website pages and scripts still make ordinary hosting requests; the privacy policy separates those from tool processing.
Frequently asked questions
Does this sort the unique lines?
No. It preserves the order of the selected first or last occurrences. Use Alphabetize List & Sort Lines afterward if you need a sorted output.
Can it remove duplicate words in a paragraph?
No. Matching operates on complete lines. It does not tokenize prose into words or decide which repeated words should be removed.
Why did a retained entry still have spaces?
Ignore surrounding spaces changes the comparison key only. The retained output is the original line, including its spaces. Use Text Cleaner if those should also be removed.
How this tool was checked
Remove Duplicate Lines was checked with case variants, surrounding spaces, Unicode normalization forms, kept-first versus kept-last order and lists close to the 50,000-record ceiling. See the testing method or report a failure.