Description
For exact GELU x*Φ(x), return (2-x²)*exp(-x²/2)/sqrt(2π) as its second input derivative.
Exact GELU is x*Φ(x), where Φ is the standard-normal cumulative distribution function. It differs from the common tanh approximation. Differentiating the exact mathematical expression and imposing a finite-precision tail cutoff are separate choices.
For exact GELU x*Φ(x), return (2-x²)*exp(-x²/2)/sqrt(2π) as its second input derivative.
When to use this tool
Use GELU Exact Scalar Second Derivative when checking the stated exact-GELU second input derivative convention.
How to use the tool and how it works
- Enter x. Use the units and conventions below.
- Run the tool. Correct any format, range, or cross-field input error before interpreting the output.
- Read derivative. Use the output convention below.
For exact GELU x*Φ(x), return (2-x²)*exp(-x²/2)/sqrt(2π) as its second input derivative. The implementation returns zero directly for abs(x)>40.1
Exact GELU uses the standard-normal cumulative distribution function. Its analytic derivative differs from the tanh approximation; the numerical tail cutoff here is an implementation detail.2
| Input field | Meaning and units | Accepted input |
|---|---|---|
| x | Finite scalar input coordinate; use the mathematical scale specified above. | Required. Finite number. |
| Output field | Interpretation | Returned shape |
|---|---|---|
| derivative | Analytic exact-GELU second derivative with an explicit numeric zero cutoff when abs(x)>40. | Required. Finite number. |
{
"input": {
"x": 0
},
"output": {
"derivative": 0.7978845608028654
}
}Limitations and assumptions
- The formula differs from a tanh-approximate GELU derivative; its explicit numerical tail cutoff is not an exact mathematical zero at finite x.
- Inputs must satisfy both the field types and the cross-field conditions above. Invalid input causes an error; a successful false flag or documented placeholder is a different result.
- Numeric calculations use binary floating point unless exact decimal-integer arithmetic is explicitly stated. No hidden measurement tolerance or decimal rounding is added; nonfinite or out-of-contract results are rejected. Numeric signed-zero results are exported as zero.
Alternative or Complementary methods
Use the complete explicit function and the intended parameter, branch and numerical conventions for symbolic, tensor or model-level differentiation; assess training behavior independently.
References
-
Calculation and input contract: activation/gelu-exact-scalar-second-derivative — tool.io
-
Gaussian Error Linear Units (GELUs) — Dan Hendrycks and Kevin Gimpel
Similar or alternative tools
- Mish Scalar Second Derivative
Let p=sigmoid(x), t=tanh(softplus(x)) and a=1-t².
- Activation: Tanh Scalar Second Derivative
Let t=tanh(x).