Description
For GLU=value*sigmoid(gate), set s=sigmoid(gate).
A scalar gated linear unit multiplies a value coordinate by a sigmoid of a gate coordinate. Differentiating the value holds the gate fixed; differentiating the gate holds the value fixed. These two partials are different components of a gradient.
For GLU=value*sigmoid(gate), set s=sigmoid(gate).
When to use this tool
Use Activation: Glu Gate Partial Derivative when checking the stated gated-linear-unit scalar partial derivative convention.
How to use the tool and how it works
- Enter value, gate. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read result. Expand result to inspect the returned fields.
For GLU=value*sigmoid(gate), set s=sigmoid(gate). Return valuePartial=s and gatePartial=value*s*(1-s).1
A gated linear unit combines a value with a sigmoid gate. This tool exposes the two scalar partial derivatives without performing the complete convolutional or tensor computation.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| value | Value | Required. Finite number. |
| gate | Gate | Required. Finite number. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| result | Object containing the operation-specific returned members listed below. | Required. Object. |
| result.valuePartial | Value Partial | Calculated result member: number. |
| result.gatePartial | Gate Partial | Calculated result member: number. |
{
"input": {
"value": 4,
"gate": 0
},
"output": {
"result": {
"valuePartial": 0.5,
"gatePartial": 1
}
}
}Limitations and assumptions
- These are two scalar partial derivatives with the other coordinate held fixed; no tensor split, network gradient or gate training is performed.
- Outputs are nested inside result. Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Use the complete explicit function and the intended parameter, branch and numerical conventions for symbolic, tensor or model-level differentiation; assess training behavior independently.
References
-
Calculation and input contract: activation/glu-gate-partial-derivative — tool.io
-
Language Modeling with Gated Convolutional Networks — Yann N. Dauphin and coauthors
Similar or alternative tools
- Leaky ReLU Slope Partial Derivative
For leaky ReLU, return x when x<0 and zero otherwise as the partial derivative with respect to the negative-branch slope.
- Swish Beta Partial Derivative
Return zero before squaring when s rounds to zero or one.