Programmatic text comparisons

Combine Two Text Files While Keeping One Copy of Each Line

Combine Two Text Files While Keeping One Copy of Each Line. Apply explicit data rules and checked examples while preserving the original inputs.

By Uddhav Gopani, FounderPublished 27 September 2026Updated 29 September 2026

To combine two text files and remove duplicate lines, read both inputs, retain the first occurrence of each exact line and write the unique sequence to a new file. That operation is a deduplicated union. It is different from keeping only lines unique to one file or removing every value that appeared more than once.

Choose which duplicate operation you intend

Suppose A contains red, blue and red, while B contains blue, green and green. A first-seen union is red, blue, green. Values present only in A would be just red. Removing every value repeated anywhere in the combined inputs would produce an empty result for this fixture.

These are three legitimate but different operations. State the intended rule before deleting anything, and preserve the originals. A command that produces fewer lines is not necessarily correct merely because the request used the word duplicates.

Preserve first-seen order in the combined output

from pathlib import Path

a = Path("a.txt").read_text(encoding="utf-8").splitlines()
b = Path("b.txt").read_text(encoding="utf-8").splitlines()
seen = set()
merged = []
for value in a + b:
    if value not in seen:
        seen.add(value)
        merged.append(value)
payload = "\n".join(merged) + ("\n" if merged else "")
Path("unique.txt").write_text(payload, encoding="utf-8")
assert Path("unique.txt").read_text(encoding="utf-8").splitlines() == merged

The set tracks membership, while the list retains order. A is processed before B, so a value’s first appearance in A wins over its later appearance in B. The output file uses LF line endings and a final newline when nonempty. That is a deliberate output convention, not a byte-for-byte preservation of either source.

Keep the equality rule visible

The example uses exact decoded line values after removing line terminators. Case and leading or trailing spaces remain significant. Red and red are therefore separate items, and 00123 does not become 123. Do not strip or lowercase every line unless those transformations match the data contract.

If blank lines occur, the first blank line is retained as an empty-string item. An empty source file contributes no items. A requirement to remove blank records is an additional filter and should be tested independently from duplicate removal.

Do not apply a line rule to every structured file

A CSV record may span several physical lines when a quoted field contains a newline. Deduplicating those physical lines can corrupt the record. Use a CSV parser and a defined record key when the input is CSV, and make the same format-aware decision for JSON, XML or other structured data.

Two records can share one identifier but differ in another field. Decide whether that is a duplicate, a conflict or an updated record. Keeping the first entire line does not resolve such a conflict automatically. Retain enough context to review the differing values before choosing a final record.

Verify content as well as the reduced count

For the red/blue/green fixture, expect exactly those three lines in that order. As a second test, use A containing oak and pine and B containing pine and birch; expect oak, pine, birch. Confirm the input files still contain their original contents after the run.

For a larger list, compare representative identifiers and check that the output has no repeated exact lines under the chosen rule. A small output count alone cannot reveal that the wrong input file was read or a normalization step merged distinct identifiers.

Compare the result with the intended baseline

The Text Diff Checker can help review what disappeared from each source, but its ordinary diff does not itself perform this first-seen deduplication. Keep the generated unique file separate until the record policy and acceptance checks are satisfied.

Interpret an empty output correctly

Two empty source files produce an empty union. A file containing a blank line contains an empty-string item under this rule and produces a retained blank line. Keep that distinction in an acceptance test so a successful zero-item result is not confused with an input that failed to load.