Stopword-Based Language Guess Score

Score bounded caller multilingual stopword profiles against actual NFC lexical tokens with tied winners and absent evidence retained; unlike existing fixed English whitespace counter, no confidence calibration or definitive language identification.

Description

Stopword overlap supplies lexical evidence for caller-defined profiles.

Stopword overlap supplies lexical evidence for caller-defined profiles. Shared words can produce ties and zero matches can mean no evidence.^[1]

Enter observed text, distinct ASCII profile identifiers and aligned whitespace-separated lexical dictionaries. NFC-normalize/lowercase letter-and-mark tokens. Return match counts/shares, all tied best labels only when positive, token count and evidence flag.

When to use Stopword-Based Language Guess Score

Use Stopword-Based Language Guess Score when you need to evaluate the supplied records under the calculation and evidence rules below. Empty observed text produces zero shares and no winners. Scores are not calibrated probabilities or definitive language identification.

Prepare explicit text tokens, annotated language observations, caller mappings or a supported locale as requested. Preserve source evidence and the stated Unicode counting convention.

How to use Stopword-Based Language Guess Score and understand its results

Prepare explicit text tokens, annotated language observations, caller mappings or a supported locale as requested. Preserve source evidence and the stated Unicode counting convention.
Enter observedText, callerProfileLabels, callerStopwordProfiles using the field table and calculation rules, then run the tool.
Read profileMatchedTokenCounts, profileObservedTokenShares, tiedBestProfileLabels, observedLexicalTokenCount, positiveProfileEvidenceAvailable. The explanation below defines the units, signs, ordering and any flags or placeholder values.
If the tool reports an input issue, correct the values and run it again. Apply the stated assumptions when interpreting a successfully returned result.

Input field	Meaning	Value type; see rules below	Requirement
observedText	Observed text	string	Required
callerProfileLabels	Caller profile labels	list of string	Required
callerStopwordProfiles	Caller stopword profiles	list of string	Required

score=count matched observed tokens; share=score/observed token count

Units, indexing, boundary conditions and exceptional cases are defined in the calculation rules above.

Output field	Meaning	Value type
profileMatchedTokenCounts	Profile matched token counts	list of number
profileObservedTokenShares	Profile observed token shares	list of number
tiedBestProfileLabels	Tied best profile labels	list of string
observedLexicalTokenCount	Observed lexical token count	number
positiveProfileEvidenceAvailable	Positive profile evidence available	boolean

json

{
  "input": {
    "observedText": "the and le",
    "callerProfileLabels": [
      "English",
      "French"
    ],
    "callerStopwordProfiles": [
      "the and",
      "le et"
    ]
  },
  "output": {
    "profileMatchedTokenCounts": [
      2,
      1
    ],
    "profileObservedTokenShares": [
      0.6666666666666666,
      0.3333333333333333
    ],
    "tiedBestProfileLabels": [
      "English"
    ],
    "observedLexicalTokenCount": 3,
    "positiveProfileEvidenceAvailable": true
  }
}

Limitations and assumptions

Empty observed text produces zero shares and no winners. Scores are not calibrated probabilities or definitive language identification.

Numeric inputs, where present, must be finite. Follow the calculation-specific rules for overflow, underflow and exceptional cases. Numeric lists are limited to 10,000 entries; string lists to 128 strings of at most 128 characters each. The rules above may impose smaller limits. Extremely large text inputs are rejected. Missing required fields, extra fields, wrong value types and violations of the calculation rules are input errors. A successfully returned false flag is an evaluated outcome; its meaning is explained above.

Alternative or Complementary methods

Use Letter Frequency Rank By Language Lookup Table: Letter-frequency ranks describe the supplied corpus and alphabet. Ties use competition rank rather than an arbitrary alphabet order. For a complementary task, use RTL/LTR Script Direction Classifier: Script-level direction counts provide a limited text inventory. Actual Unicode bidirectional layout also depends on marks, numbers and formatting controls.

References

Stopword-Based Language Guess Score — tool.io

Similar or alternative tools

Letter Frequency Rank By Language Lookup Table
Letter-frequency ranks describe the supplied corpus and alphabet. Ties use competition rank rather than an arbitrary alphabet order.
RTL/LTR Script Direction Classifier
Script-level direction counts provide a limited text inventory. Actual Unicode bidirectional layout also depends on marks, numbers and formatting controls.