UtilityPilot.Find a tool

Text & Lists

Remove Duplicate Lines

Remove duplicate lines from a pasted list while keeping the first or last occurrence. Choose exact matching or ignore case, surrounding whitespace and Unicode representation differences.

Processed in your browserPrivacy details ↗

1 MiB per input

Adjust your options

No sign-up. No upload.

How to remove duplicate lines

  1. Paste one complete entry per line.
  2. Leave matching options off for exact comparison, or choose the differences that should be ignored.
  3. Select First or Last under Occurrence to keep.
  4. Select Remove duplicates and check the retained and removed counts.
  5. Copy result or Download the unique lines. The original input stays available.

A repeated line is one complete entry that matches an earlier comparison key. The tool does not remove repeated words within a sentence, find similar names or merge partial entries. It accepts up to 1 MiB of text and runs in the browser.

How duplicate lines are detected

Default matching is exact and case-sensitive. A line containing Apple differs from apple, and apple with a trailing space differs from apple. Punctuation also matters.

  • Ignore letter case: lowercase the comparison key using the tool’s English locale default. Original output spelling is retained.
  • Ignore surrounding spaces: trim whitespace from the beginning and end of the comparison key. Internal spaces remain significant.
  • Normalize Unicode (NFC): compare canonically normalized keys, allowing a precomposed accent to match its equivalent base-letter-plus-accent spelling.

When enabled together, trimming happens first, NFC normalization second and lowercasing third. These operations change the key used to detect duplicates, not the surviving line. NFC does not remove accents, translate text or make all look-alike characters equal.

Duplicate line remover example

Input

apple
pear
apple
Apple

Result

Default, keep First:
apple
pear
Apple

Ignore letter case, keep First:
apple
pear

The default removes one repeated line; case-insensitive matching removes two. Both keep the first apple spelling. If Ignore surrounding spaces is enabled and the first occurrence includes edge spaces, those spaces remain in the output. To change the visible text as well, clean the list separately.

Keep the first or last occurrence

First retains the first matching entry and preserves the order of those first appearances. Last retains the final matching entry and preserves the order of the surviving last appearances. It does not keep the first position while swapping in the final spelling.

Input

apple
pear
Apple

Result

Ignore letter case, keep First:
apple
pear

Ignore letter case, keep Last:
pear
Apple

Choose the mode based on how the list was assembled. Last can retain a later spelling, but the tool has no timestamps and cannot prove the later line is more accurate. If chronology matters, confirm that the pasted order actually represents it.

Blank lines, line endings and whitespace

The processor treats CRLF and CR as newline boundaries. One final newline is a terminator rather than an extra empty entry. Internal blank lines participate in duplicate matching, so repeated empty entries collapse to one retained blank entry.

A line of spaces and an empty line are different under exact matching. With Ignore surrounding spaces enabled, they share an empty comparison key, and the chosen original line survives. This behavior does not make the tool a remove-all-blank-lines utility. Use the line counter to inspect the result if blank entries affect the receiving workflow.

Remove duplicate text online: useful list workflows

Deduplicate a repeated list of labels, pasted filenames or simple identifiers when one line represents one item. Start with exact matching for identifiers whose case might be significant. Enable broader matching only when those differences are irrelevant to your task.

After removing duplicates, alphabetize the list if you need a lookup-friendly order. Sorting is separate because the original sequence may matter. To find entries present in one list but missing in another, use Compare Lists; that is a membership comparison, not just deduplication.

When line deduplication is the wrong tool

Do not use this on prose to remove duplicate words. A repeated sentence can be intentional, and two words on one line are not separate entries. There is no fuzzy matching, spelling correction or meaning-based grouping.

CSV records may contain quoted newlines, so a physical line is not always a complete row. Use CSV deduplication for whole records or selected key columns. Always inspect the surviving entries before replacing an original list, especially when broader matching rules combine differently spelled values.

Processing and privacy

Remove Duplicate Lines compares and reorders the list locally; no line or file leaves the browser. Input stays in browser memory until you reset or leave the page. Copy and download run only when you choose them. No tool input is sent to a processing service. Website pages and scripts still make ordinary hosting requests; the privacy policy separates those from tool processing.

Frequently asked questions

Does this sort the unique lines?

No. It preserves the order of the selected first or last occurrences. Use Alphabetize List & Sort Lines afterward if you need a sorted output.

Can it remove duplicate words in a paragraph?

No. Matching operates on complete lines. It does not tokenize prose into words or decide which repeated words should be removed.

Why did a retained entry still have spaces?

Ignore surrounding spaces changes the comparison key only. The retained output is the original line, including its spaces. Use Text Cleaner if those should also be removed.

How this tool was checked

Remove Duplicate Lines was checked with case variants, surrounding spaces, Unicode normalization forms, kept-first versus kept-last order and lists close to the 50,000-record ceiling. See the testing method or report a failure.