Cleaning Up Text: Case, Duplicates and Invisible Characters

Copied text rarely arrives clean. It may contain curly quotes, non-breaking spaces, zero-width characters, inconsistent capitalisation, duplicated lines from a bad export, or line breaks in the middle of sentences. The visible content looks fine, but searches fail, spreadsheets misalign and publishing tools complain.

Start by identifying the source

The artefacts you need to remove depend on where the text came from:

• From a PDF: line breaks at the end of every visual line, hyphenated words split across lines, and occasional ligature characters.
• From a web page: non-breaking spaces, soft hyphens and mixed quote styles.
• From a spreadsheet export: stray tabs, duplicated header rows and trailing delimiters.
• From OCR: misread characters, especially 1 and l, 0 and O, and broken word spacing.

Normalise case deliberately

Sentence case, title case and upper case each have a place. Sentence case is the safest default for body text and headings in most editorial styles. Title case rules vary between style guides, especially for short prepositions, so check the guide you are following rather than trusting a converter's default.

Case conversion should not touch email addresses, URLs or code. Convert the prose first, then restore those elements, or exclude them from the selection. Acronyms and product names also need a manual pass; an automatic converter will happily turn them into something wrong.

Remove duplicate lines

Duplicate lines typically come from merged exports, repeated headers, or copy-paste that captured a table twice. A duplicate checker that compares whole lines is safer than one that removes repeated words, because repeated words are often intentional. Decide whether comparison should be case-sensitive and whether leading or trailing spaces should be ignored; "Total" and "total " are different strings to a strict comparison but the same line to a human reader.

Find invisible characters

These are the characters you cannot see but that break matching:

• Non-breaking space (U+00A0) looks like a space but is not one. Searching for a normal space will not find it.
• Zero-width space (U+200B) is used to allow line breaks inside long words and is invisible everywhere else.
• Soft hyphen (U+00AD) appears only when a word is broken at the end of a line.
• Byte order mark (U+FEFF) at the start of a file can stop the first line from being recognised.
• Windows line endings (carriage return plus line feed) can add a stray character when text is pasted into a Unix-style system.

Use a character inspector or a replace operation with an explicit code point to locate them. Replacing a non-breaking space with a normal space is usually safe; removing zero-width characters is safe in most prose but should be checked in languages that rely on them for line breaking.

Fix line breaks and spacing

Text pasted from a PDF often has a hard line break at the end of every visual line. Joining lines is the fix, but joining indiscriminately merges headings with paragraphs and destroys lists. A reliable sequence is:

1. Replace runs of more than one space with a single space.
2. Replace tabs with a space or the delimiter your target format expects.
3. Join lines, then re-insert paragraph breaks at headings and list items.
4. Remove spaces before punctuation and ensure one space after it.

For Chinese and English mixed text, typography rules also call for specific spacing between scripts and full-width punctuation. Apply those rules with a dedicated tool rather than global search and replace, which tends to over-correct.

Proofread after processing

Automated clean-up can introduce errors of its own: a case converter may capitalise after an abbreviation, a duplicate remover may delete a legitimately repeated line, and a smart-quote conversion may break code. Read the result, check the beginning and end, and search for the patterns you changed to confirm the outcome.