High-Cardinality Column Flagger
Flag columns whose distinct-to-observed ratio exceeds an explicit threshold, counting exact nonempty data cells and excluding an explicit header; zero observations never flag.
Description
A column’s distinct-to-observed ratio measures how frequently values are unique.
A column’s distinct-to-observed ratio measures how frequently values are unique. High cardinality can prompt review of whether a field behaves like an identifier.1
Flag columns whose distinct-to-observed ratio exceeds an explicit threshold, counting exact nonempty data cells and excluding an explicit header; zero observations never flag.
When to use High-Cardinality Column Flagger
Use this for the following task: Flag columns whose distinct-to-observed ratio exceeds an explicit threshold, counting exact nonempty data cells and excluding an explicit header; zero observations never flag.
Prepare the CSV text, decoded cells or byte values requested below. Choose delimiter and header flags explicitly when those fields are present.
How to use High-Cardinality Column Flagger and understand its results
- Prepare the CSV text, decoded cells or byte values requested below. Choose delimiter and header flags explicitly when those fields are present.
- Enter csv, delimiter, hasHeader, threshold using the field table, then run the tool.
- Read columns. Interpret flags, units, ordering and placeholder values using the rules below.
- Review any reported issue or failed check, correct the supplied values when appropriate, and verify the result in its intended context.
| Input field | Meaning | Value type and bounds | Requirement |
|---|---|---|---|
| csv | CSV text | string; maximum text length 100000 | Required |
| delimiter | Delimiter | string; choose ",", ";", "\t" | Required |
| hasHeader | Has header | boolean | Required |
| threshold | Threshold | number; minimum 0; maximum 1 | Required |
Parse rectangular data excluding an explicit header. Ignore exactly empty cells, tally exact distinct values and compare their ratio with the supplied fraction threshold.2
Parse delimiters only outside quoted fields and decode doubled quotes. A quoted line break stays inside its cell; malformed quoting is returned as an input error.
ratio=distinctNonempty/observedNonempty; high if observations>0 AND ratio>threshold This summarizes the declared operation. Units, indexing, inclusivity and exceptional cases are explained in the surrounding text.
| Output field | Meaning | Value type |
|---|---|---|
| columns | Columns | list of record: column (integer), distinctCount (integer), observedCount (integer), ratio (number), high (boolean) |
{
"input": {
"csv": "id,group\n1,A\n2,A",
"delimiter": ",",
"hasHeader": true,
"threshold": 0.75
},
"output": {
"columns": [
{
"column": 0,
"distinctCount": 2,
"observedCount": 2,
"ratio": 1,
"high": true
},
{
"column": 1,
"distinctCount": 1,
"observedCount": 2,
"ratio": 0.5,
"high": false
}
]
}
}Limitations and assumptions
Equality is not flagged and zero observations return ratio 0/high false. This does not prove a column is a key.
Where CSV is parsed, the shared parser accepts comma, semicolon or tab separators, doubled quotes and quoted line breaks, with CR/LF/CRLF record endings. Blank records remain records. CSV inputs are bounded to 100000 UTF-16 units and 10000 fields. Malformed requests produce input errors. A successfully evaluated request can still contain false flags, empty result lists or a result outside the supplied threshold; those outcomes have the meanings stated above.
Alternative or Complementary methods
Choose Low-Cardinality Category Lister for this operation: List first-encounter distinct nonempty values for data columns whose observed category count is at most an explicit maximum, excluding an explicit header. For a complementary task, use First-N Distinct Values Per Column for this operation: Sample the first N distinct decoded data values per column in encounter order, counting empty strings as values and reporting the full distinct count.
Further background: Format and language reference: High-Cardinality Column Flagger.
References
-
Format and language reference: High-Cardinality Column Flagger — RFC Editor
-
High-Cardinality Column Flagger — tool.io
Similar or alternative tools
- Low-Cardinality Category Lister
List first-encounter distinct nonempty values for data columns whose observed category count is at most an explicit maximum, excluding an explicit header.
- First-N Distinct Values Per Column
Sample the first N distinct decoded data values per column in encounter order, counting empty strings as values and reporting the full distinct count.