Unique Rows Dedupe First Occurrence
Remove exact duplicate decoded CSV records after an optional header, retaining first occurrences and their original order; cell boundaries remain significant.
Description
CSV rows are sequences of decoded cells.
CSV rows are sequences of decoded cells. Exact deduplication should preserve cell boundaries and retain a defined occurrence order.1
Remove exact duplicate decoded CSV records after an optional header, retaining first occurrences and their original order; cell boundaries remain significant.
When to use Unique Rows Dedupe First Occurrence
Use this for the following task: Remove exact duplicate decoded CSV records after an optional header, retaining first occurrences and their original order; cell boundaries remain significant.
Prepare the CSV text, decoded cells or byte values requested below. Choose delimiter and header flags explicitly when those fields are present.
How to use Unique Rows Dedupe First Occurrence and understand its results
- Prepare the CSV text, decoded cells or byte values requested below. Choose delimiter and header flags explicitly when those fields are present.
- Enter csv, delimiter, hasHeader using the field table, then run the tool.
- Read csv, removedCount. Interpret flags, units, ordering and placeholder values using the rules below.
- Review any reported issue or failed check, correct the supplied values when appropriate, and verify the result in its intended context.
| Input field | Meaning | Value type and bounds | Requirement |
|---|---|---|---|
| csv | CSV text | string; maximum text length 100000 | Required |
| delimiter | Delimiter | string; choose ",", ";", "\t" | Required |
| hasHeader | Has header | boolean | Required |
Parse records and set the header flag explicitly. Use JSON array text as an exact row key, retain the first occurrence of each data row and re-encode retained rows.2
Parse delimiters only outside quoted fields and decode doubled quotes. A quoted line break stays inside its cell; malformed quoting is returned as an input error.
The shared writer doubles interior quotes and quotes cells when needed for delimiters or line breaks, including a single empty-field record. It normally emits LF record endings without a terminal newline; any tool-specific quote-all behavior is stated above.
remove every row whose decoded cell array has appeared earlier This summarizes the declared operation. Units, indexing, inclusivity and exceptional cases are explained in the surrounding text.
| Output field | Meaning | Value type |
|---|---|---|
| csv | CSV text | string |
| removedCount | Removed count | integer |
{
"input": {
"csv": "name,age\nAda,36\nGrace,85\nAda,36",
"delimiter": ",",
"hasHeader": true
},
"output": {
"csv": "name,age\nAda,36\nGrace,85",
"removedCount": 1
}
}Limitations and assumptions
Whitespace, case and different decoded cell boundaries remain significant. Header rows are excluded from deduplication.
Where CSV is parsed, the shared parser accepts comma, semicolon or tab separators, doubled quotes and quoted line breaks, with CR/LF/CRLF record endings. Blank records remain records. CSV inputs are bounded to 100000 UTF-16 units and 10000 fields. Malformed requests produce input errors. A successfully evaluated request can still contain false flags, empty result lists or a result outside the supplied threshold; those outcomes have the meanings stated above.
Alternative or Complementary methods
Choose Duplicate Rows Extractor for this operation: Extract every CSV data record whose exact decoded row occurs more than once, retaining all members of duplicate groups in original order and preserving an optional header. For a complementary task, use Row Hash Digest Dedupe Key for this operation: Fingerprint one decoded row using legacy MD5 over UTF-8 JSON array text, preserving cell boundaries; confirm row equality before deduplication because MD5 is not collision resistant.
Further background: Format and language reference: Unique Rows Dedupe First Occurrence.
References
Similar or alternative tools
- Duplicate Rows Extractor
Extract every CSV data record whose exact decoded row occurs more than once, retaining all members of duplicate groups in original order and preserving an optional header.
- Row Hash Digest Dedupe Key
Fingerprint one decoded row using legacy MD5 over UTF-8 JSON array text, preserving cell boundaries; confirm row equality before deduplication because MD5 is not collision resistant.