The hard part of merging several lists is not spotting the duplicates but removing them without disturbing the order — sort | uniq and the spreadsheet habit of sorting before deleting both destroy the sequence along the way. This tool compares line by line and keeps each value where it first appeared. The comparison runs in your browser; the list is never transmitted.
How to use it
- Paste a list with one record per line into the Input box on the left. The Output box updates live — there is nothing to run.
- Tick the options above as needed. "Ignore case" treats
Alice@x.comandalice@x.comas the same record; "Ignore leading/trailing spaces" stops stray indentation from defeating the match. With neither ticked, only a character-for-character match counts as a duplicate. - When something is dropped, a line reading "Removed N duplicate lines" appears under the output. It is absent when nothing was removed.
- Take the result with Copy at the bottom right; Clear on the left empties the input box.
When it comes in handy
Merging mailing lists before a send
Event sign-ups, newsletter subscribers and the addresses your sales colleague collected: stack three exports together and somebody is getting the mail twice. Paste the address column, tick Ignore case — the domain part is case-insensitive by definition — and you get one clean list in its original order, which matters when you want the earliest registrations at the top.
Tidying keyword and negative-keyword lists
Keyword sets for SEO or paid search are usually stitched together from several exports. Duplicates waste review time and can be rejected outright by the ad platform on upload. Removing them here leaves the remaining terms in the priority order you built.
Counting how many distinct errors a log contains
Several thousand error lines usually boil down to five or six real problems. Deduplicate the message column and the remaining line count is your number of distinct failures, still ordered by first occurrence. Strip the timestamps first — with them attached, every line is unique.
Limits and behavior
- Matching is whole-line and exact. Repetition inside a line is not touched:
apple,apple,bananais one unique line as far as this tool is concerned. Comma- or semicolon-separated lists have to be split onto separate lines before pasting. - Blank lines and whitespace-only lines are always discarded, with no switch on screen to keep them, so data that uses blank lines as group separators loses that grouping. The "Removed N duplicate lines" figure counts only lines dropped for being duplicates and excludes those blanks, which is why it is usually smaller than the difference between the input and output line counts.
- With "Ignore leading/trailing spaces" ticked, the comparison uses the trimmed value but the output keeps the first occurrence exactly as written, indentation included. Likewise, with "Ignore case" ticked the surviving line keeps its original capitalisation, so if
TAIPEIcame first,TAIPEIis what you get back. - Comparison is character-by-character with no Unicode normalisation and no width folding:
ａｂｃandabcare two separate records, as are the composed and decomposed forms of an accented letter. Normalise such data with the full/half-width converter first, then deduplicate. - This tool does not sort, does not report how many times each value occurred, and offers no "show only the duplicates" mode. All of it executes in this tab on your own machine, so the list is never uploaded — though the page itself does load third-party resources such as advertising, and those requests carry none of what you pasted.
Frequently asked questions
The "Removed N" figure does not match how many lines disappeared. Is it wrong?
No. Ten input lines containing two duplicates and three blanks leave five lines of output under a message reading "Removed 2 duplicate lines", because the blanks are outside that counter. If you need the total reduction, run the input and the output through the word count tool and compare the line figures.
Will it sort my list?
No, deliberately. sort -u on the command line and Remove Duplicates in a spreadsheet usually reorder as a side effect, but plenty of lists carry meaning in their sequence — registration order, priority, the structure of the source document. When you do want sorting, deduplicate here and let the spreadsheet handle the ordering afterwards.
Why were my blank lines removed? I was using them as separators.
There is currently no option to keep them; they are filtered out before comparison begins. The alternative is worse — a second blank line would be treated as a duplicate of the first, which is harder to predict. When blank lines are structurally meaningful, substitute a marker that cannot occur in your data, such as ---, deduplicate, then substitute it back.
Is my list uploaded? It contains customer addresses.
It is not. The deduplication logic lives in JavaScript loaded by this tab: it reads the input box, compares, and writes the output box, entirely on your machine — disconnect from the network and it still works. Worth stating plainly: the page itself loads third-party resources such as advertising, and those connections carry page information rather than your records.