Random CSV Unicode Normalization Corpus Generator

Description

Normalize supplied headers and cells with a sampled Unicode form and retain UTF-8 bytes and canonical-equivalence comparisons. Parsed JSON preserves the disclosed legacy empty-cell and blank-record conversion, not a lossless round trip.

CSV stores records as delimited text fields. Quoting can protect embedded delimiters and newlines, so a record need not equal one physical line. Cleaning field contents, interpreting their types, and preserving an exact import/export round trip are separate operations.

When to use this tool

Use this generator to create reproducible csv unicode normalization corpus cases from the supplied model or bank. Inspect the returned evidence before using cases in a test or exercise.

How to use the tool and how it works

Supply headers, rows, forms in the field shapes below. Set seed when you need an explicit replay.
Run the tool and correct any input error. For generated controls, distinguish a successful fixture containing an invalid case from failure to generate the fixture.
Read form, headers, rows, comparisons, csv, parsedJson, conversionConvention. Use the conventions below. Keep answer-bearing fields separate when presenting an exercise.

Select one normalization form and normalize headers and cells. Return before/after UTF-8 bytes, byte equality, NFC-based canonical equivalence, serialized CSV, and the delegated nonnumeric CSV-to-JSON import.^[1]

Delegated operations used for this result: normalize unicode; csv to json. Their calculation or parsing conventions apply to the corresponding result fields.^{[2, 3]}

Input field	Meaning and units	Accepted input
headers	Headers	Required. Array of Text; minimum UTF-16 length 1; maximum UTF-16 length 32.; minimum items 1; maximum items 8.
rows	Rows	Required. Array of Array of Text; maximum UTF-16 length 32.; minimum items 1; maximum items 8.; minimum items 1; maximum items 16.
forms	Forms	Required. Array of Text; one of "NFC", "NFD", "NFKC", "NFKD".; minimum items 1; maximum items 4.
seed	Optional numeric seed; takes precedence over context randomness. Without either, use the fixed tool.io default seed.	Optional. Safe integer.

Output field	Interpretation	Returned shape
form	Form	Required. Text; one of "NFC", "NFD", "NFKC", "NFKD".
headers	Headers	Required. Array of Text; maximum UTF-16 length 256.; minimum items 0; maximum items 8.
rows	Rows	Required. Array of Array of Text; maximum UTF-16 length 256.; minimum items 1; maximum items 10.; minimum items 0; maximum items 128.
comparisons	Comparisons	Required. Array of Object; no unknown keys.; minimum items 0; maximum items 136.
comparisons[].original	Original	Required. Text; maximum UTF-16 length 32.
comparisons[].normalized	Normalized	Required. Text; maximum UTF-16 length 256.
comparisons[].originalBytes	Original Bytes	Required. Array of Safe integer; minimum 0; maximum 255.; minimum items 0; maximum items 128.
comparisons[].normalizedBytes	Normalized Bytes	Required. Array of Safe integer; minimum 0; maximum 255.; minimum items 0; maximum items 1024.
comparisons[].byteEqual	Byte Equal	Required. Boolean.
comparisons[].canonicalEquivalent	Canonical Equivalent	Required. Boolean.
csv	Csv	Required. Text; maximum UTF-16 length 100000.
parsedJson	Parsed Json	Required. Text; maximum UTF-16 length 100000.
conversionConvention	Conversion Convention	Required. Text; exactly "legacy CSV-to-JSON: empty cells become null; single-empty-cell records may be dropped".
randomness	Replay source and normalized 32-bit seed, or null seed for an advancing context source.	Required. Object; no unknown keys.
randomness.seed	Optional safe-integer replay seed; normalization and precedence are explained below.	Required. Safe integer. or Null.
randomness.source	Source	Required. Text; one of "context", "seed".

An explicit safe-integer seed takes precedence over injected context randomness and is normalized to an unsigned 32-bit integer. With no seed and no context source, replay uses the fixed tool.io default seed. A context source must return finite values from zero inclusive to one exclusive; its advancing state must be retained separately. randomness records the source and normalized seed, or null for context. These generators are not cryptographic random sources.

json

{
  "input": {
    "headers": [
      "café"
    ],
    "rows": [
      [
        "é"
      ]
    ],
    "forms": [
      "NFC",
      "NFD"
    ],
    "seed": 42
  },
  "output": {
    "form": "NFD",
    "headers": [
      "café"
    ],
    "rows": [
      [
        "é"
      ]
    ],
    "comparisons": [
      {
        "original": "café",
        "normalized": "café",
        "originalBytes": [
          99,
          97,
          102,
          195,
          169
        ],
        "normalizedBytes": [
          99,
          97,
          102,
          101,
          204,
          129
        ],
        "byteEqual": false,
        "canonicalEquivalent": true
      },
      {
        "original": "é",
        "normalized": "é",
        "originalBytes": [
          195,
          169
        ],
        "normalizedBytes": [
          101,
          204,
          129
        ],
        "byteEqual": false,
        "canonicalEquivalent": true
      }
    ],
    "csv": "café\né",
    "parsedJson": "[{\"café\":\"é\"}]",
    "conversionConvention": "legacy CSV-to-JSON: empty cells become null; single-empty-cell records may be dropped",
    "randomness": {
      "seed": 42,
      "source": "seed"
    }
  }
}

Limitations and assumptions

Compatibility normalization can change canonical equivalence and normalized headers must stay unique. The legacy converter turns empty cells into null and may drop single-empty-cell records; normalization does not guarantee an exact table round trip.
A replay requires the same complete input and implementation. The stated sampling convention applies within its eligible space. A finite draw does not exhaust all possible outputs; repeats remain possible unless a distinct-selection rule is explicit.
The field tables state structural bounds; cross-field constraints and bounded search or output budgets also apply. Unknown keys are rejected where the object contract disallows them. Input errors are distinct from null fields and deliberate invalid controls in a successful output.

Alternative or Complementary methods

For a supplied example, inspect the returned construction or delegated check directly. To evaluate a consumer, also test cases chosen for its own boundaries; a generated finite corpus does not establish exhaustive coverage.

References

Tool behavior and example: Random CSV Unicode Normalization Corpus Generator — tool.io
Calculation or validation: text/normalize-unicode — tool.io
Calculation or validation: convert/csv-to-json — tool.io

Similar or alternative tools

Random CSV Aggregate Reconciliation Exercise Generator
Generate synthetic grouped integer observations, plant one declared subtotal discrepancy and retain exact decimal-string subtotal, grand-total and checksum-row reconciliation answers.
Random CSV Boolean Token Cleaning Exercise Generator
Sample caller-selected Boolean spellings, blank cells and unknown controls, retaining explicit classifications and nullable normalization answers from the Boolean token core.