About Remove Duplicate Lines
Remove Duplicate Lines drops repeated lines from a list, keeping the first occurrence of each unique value and discarding the rest. Order is preserved: surviving lines are emitted in the order they first appeared in the input. The result is a list of distinct lines.
This is the everyday cleanup operation for noisy data: collapse log lines that repeat under bursty traffic, dedupe a hand-edited tag list, build a unique inventory from a concatenated export, prune an email list before a mail-merge. It works on a single list — unlike Full-Inclusive Join, which dedupes across two lists. The comparison is exact by default — case- and whitespace-sensitive — so `Apple` and `apple` are different items, and `apple ` (with a trailing space) is different from `apple`.
Examples
apple
banana
apple
cherry
bananaapple
banana
cherryFirst occurrence wins. Original order is preserved.
Frequently asked questions
How is this different from Sort Lines?
Sort Lines reorders without removing anything. Remove Duplicate Lines removes without reordering. Run Sort Lines after deduping if you also want the result alphabetised.
Why is my output still showing what looks like duplicates?
Most often because of trailing whitespace or invisible characters. `apple` and `apple ` (trailing space) are different lines to the dedupe tool. Run Trim Each Line first to normalise whitespace, then dedupe.
Is the comparison case-sensitive?
Yes by default. Use the case-insensitive option in the comparison bar to fold `Apple` and `apple` together; the version preserved in the output is the first occurrence as written.
How is this different from Full-Inclusive Join?
Full-Inclusive Join expects two lists and dedupes across both. Remove Duplicate Lines works on a single list. They share the dedupe semantics; they differ in arity.