Indexing Stopword Position Increment Tool

Description

Retain unflagged tokens with zero-based originalPositions and increments from the prior retained position, beginning from -1.

Search metrics and positional checks depend on rank origins, token positions, relevance definitions, and tie or gap conventions. One term is not a complete query or retrieval evaluation.

Retain unflagged tokens with zero-based originalPositions and increments from the prior retained position, beginning from -1.

When to use this tool

Use Indexing Stopword Position Increment when you need one rank contribution or token-position diagnostic.

How to use the tool and how it works

Enter tokens, stopped. Keep parallel arrays aligned. Use the units and conventions below.
Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
Read retained, increments, originalPositions. Use the output convention below.

Require aligned tokens and stopped flags. Retain unflagged tokens with zero-based originalPositions and increments from the prior retained position, beginning from -1.^[1]

Input field	Meaning and units	Accepted input
tokens	Tokens	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.
stopped	Stopped	Required. Array of boolean; maximum items 256.

Output field	Interpretation	Returned shape
retained	Retained	Required. Array of text; maximum UTF-16 length 8192; maximum items 256.
increments	Increments	Required. Array of safe integer; maximum items 256.
originalPositions	Original Positions	Required. Array of safe integer; maximum items 256.

json

{
  "input": {
    "tokens": [
      "the",
      "red",
      "and",
      "blue"
    ],
    "stopped": [
      true,
      false,
      true,
      false
    ]
  },
  "output": {
    "retained": [
      "red",
      "blue"
    ],
    "increments": [
      2,
      2
    ],
    "originalPositions": [
      1,
      3
    ]
  }
}

Limitations and assumptions

Gaps preserve removed-token positions; no stopword vocabulary, tokenization, or indexing system is supplied.
Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.

Alternative or Complementary methods

Evaluate complete queries and result sets under a documented tokenization, relevance, and aggregation policy.

References

Calculation and input contract: information-science/indexing-stopword-position-increment — tool.io

Similar or alternative tools

Query Phrase Slop One Gap
Subtract firstPosition from secondPosition for observedGap, then compare its absolute difference from expectedGap inclusively with allowedSlop.
Query Phrase Missing Token Position Flag
For each phrase token, test exact membership in availableTokens and return zero-based missing positions and their tokens.