Computational Linguistics Tools
Browse all available computational linguistics tools.
Available tools
- Alignment Null Support Share Tool Account for supplied null-index alignment support on both sides without merging one-sided and double-null rows.
- BIOES Single Token Transition Label Check one local BIOES transition under an explicit strict span-closing convention, excluding sentence boundary validation.
- CoNLL-U Decimal Empty Node Key Split a CoNLL-U empty-node ID without parsing its dot suffix as a fractional decimal or checking sentence-order constraints.
- CoNLL-U Range Token Boundary Parse a CoNLL-U multiword-token range into positive inclusive word-ID endpoints without validating a sentence.
- CoNLL-U Root Label Head Zero Check Tool Check a parsed CoNLL-U dependency label/head pair; does not validate an entire dependency tree.
- Constituency Empty Node Label Append an explicit positive coindex to a supplied empty-category label using a label-index convention; zero denotes no coindex and no treebank inventory is inferred.
- Coreference Mention Number Label Label an explicitly assigned coreference mention and cluster without resolving referents.
- Corpus Character Offset Newline Adjustment Adjust an explicit offset when each counted preceding LF becomes CRLF, adding one code unit per supplied preceding newline.
- Dependency Head Distance Tool Resolve one supplied dependency edge in a common token-index convention, preserving a root-head flag.
- Dependency One Based Head Crosswalk Map a dependency head index with root=0 into a zero-based token index with root=-1, without validating an entire dependency tree.
- Dependency Projectivity Crossing Flag Tool Detect strict interleaving of two supplied dependency arcs; shared endpoints and nesting do not cross.
- Entity Span Inclusive Exclusive Crosswalk Convert an explicit nonempty inclusive entity-token span into an exclusive endpoint and length without retokenizing text.
- IOB Entity Tag Prefix Split Split an explicit IOB B-/I- entity tag or outside O token without validating sequence transitions.
- IPA Secondary Stress Position Tool Locate literal U+02CC secondary-stress markers using Unicode code-point indices; no phonological analysis.
- Language Model Log Base Perplexity Crosswalk Convert explicit mean negative log-likelihood between logarithm bases and perplexity without scoring a language model or changing tokenization.
- Lexical Surprisal Token Bits Calculate one supplied positive token probability surprisal in bits without estimating a language model probability.
- Linguistic Tag Attribute Value Token Format one explicit key=value linguistic feature token under a no-whitespace/no-pipe value grammar without validating a feature inventory.
- Morph Feature Empty Value Token Tool Keep empty feature strings distinct from an explicit missing marker.
- Morpheme Boundary Byte Offset Count UTF-8 bytes preceding an explicit morpheme boundary, requiring each morpheme to contain complete Unicode scalar values; the terminal boundary is allowed.
- Morphological Gloss Abbreviation Source Row Preserve an explicit gloss abbreviation, supplied expansion and source without inventing a universal gloss vocabulary.
- Parse Node Span Width Tool Calculate parse node span width from explicit caller-supplied inputs.
- Phoneme Feature Single Contrast Compare one explicitly supplied phoneme-feature value pair without inferring a phonological feature system.
- Pronunciation Lexicon Alternative Index Append an explicit parenthesized positive pronunciation alternative index to a supplied headword without looking up pronunciation.
- Subword Token Byte Offset Tool Compute UTF-8 byte boundaries in literally concatenated supplied tokens containing complete Unicode scalar values; no model-specific detokenization.
- Subword Word Boundary Marker Token Remove one supplied literal leading subword boundary marker when present without selecting a tokenizer dialect.
- Token Byte Character Ratio Tool Compare portable UTF-8 bytes to code points; isolated surrogates use replacement encoding.
- Token Offset Domain Gap Tool Extract a half-open token span with an explicit code-point versus UTF-16 offset domain.
- Word Alignment Fertility Count Tool Count distinct target IDs for one source token while preserving explicit null-alignment metadata.
- Word Alignment Null Partner Flag Tool Flag a supplied null-partner alignment convention without discovering word correspondences.
- Word Alignment Pair Token Format one source-target word alignment pair in zero-based i-j notation from an explicit indexing convention.