A list exported from somewhere real — a signup form, a spreadsheet column, a product feed, a tag dump — is almost never clean. It has duplicates, blank rows, inconsistent order, and rows that differ only by a stray space or a capital letter. Tidying it up sounds trivial until you hit the cases that quietly corrupt the result: a dedupe that scrambles the order, a sort that puts item 10 before item 2, a blank line that survives because it holds a space. This guide covers doing it without those surprises.
Dedupe without losing the order
The obvious way to remove duplicates is to sort the list and drop adjacent repeats, but sorting destroys any order that mattered — the sequence a form was filled in, the priority a list was written in, the grouping a colleague set up. Order-preserving dedupe keeps the first occurrence of each item where it was and removes only the later repeats. The list comes out shorter but in the same shape, which is usually what you actually wanted.
The harder problem is deciding what counts as a duplicate. Two entries can be the same to a human and different to a computer because one has a trailing space, or one is capitalised. “ALICE@example.com”, “alice@example.com”, and “alice@example.com ” are one person and three distinct strings. Ignoring case and surrounding whitespace during the comparison catches these near-duplicates that an exact match misses.
Remove Duplicate LinesDeduplicate a list while keeping first-appearance order, and ignore case and stray spaces so near-duplicates collapse too.When you want the duplicates instead
Sometimes the repeats are the point. If you are checking whether an email appears twice in a signup export, or which SKU got entered more than once, you want to keep only the lines that appeared more than once and discard the singletons. Flipping the mode this way turns a dedupe into a duplicate-finder, which is the fastest way to audit a list for accidental repeats before you import it somewhere that will reject them.
Natural sort, and why it matters
A plain alphabetical sort compares text character by character, which means it sorts “item 10” before “item 2”, because the character “1” comes before “2”. For anything with numbers in it — filenames, version tags, SKUs, chapter labels — that ordering is wrong in the way that matters. Natural sort reads the digits as numbers, so “item 2” comes before “item 10”, which is what a human expects.
- Plain sort: file1, file10, file2, file20, file3 — the numbers scatter.
- Natural sort: file1, file2, file3, file10, file20 — the numbers read in order.
- Use natural sort for versions, filenames, and any label ending in a number.
- Use plain alphabetical sort for words, names, and tags with no numeric part.
Sorting has other modes worth knowing. Sorting by line length groups short entries together and is useful for spotting outliers — a row that is far longer than the rest is often a formatting error or two rows that got joined. A simple reverse flips the current order, which is the quickest way to get a most-recent-first view of a list that was appended chronologically.
Sort LinesSort alphabetically, numerically, or by length, with natural ordering so numbered items land where you expect.Blank lines that aren’t empty
Empty rows are the classic import-breaker: a blank line becomes an empty record, a null email, a row with no SKU. The catch is that many “blank” lines are not truly empty — they contain a space, a tab, or a run of both, which is invisible on screen but real to any tool reading the file. A naive removal that only drops zero-length lines leaves these behind, and they cause exactly the failures you were trying to prevent.
Treating any line that contains only whitespace as empty is what catches them all. There is also the related case of long runs of blank lines — big vertical gaps left by an export — which you often want collapsed to a single separator rather than removed entirely, so the structure of the list survives while the gaps close up.
Remove Empty LinesStrip blank lines — including those holding only spaces or tabs — or collapse long runs down to a single gap.The order to do it in
The steps interact, so sequence matters. A dependable order is: first remove blank lines, so empty rows do not get counted or sorted as if they were data; then dedupe, so you are removing repeats from a list that is already free of junk rows; then sort last, once the list holds only the real, unique entries. Sorting first and deduping after works too, but sorting a list still full of blanks and duplicates just wastes the sort on rows you are about to delete.
One caution for anything you will import: check whether the destination cares about a trailing newline or a header row, and preserve or strip it deliberately rather than by accident. A tidy list with the wrong first line still fails the import.
All of this runs in your browser, so a list of customer emails, internal SKUs, or subscriber addresses is not uploaded anywhere to be cleaned. The list goes in, the tidy version comes out, and nothing about your data leaves the page.