LayerNorm {#dev_guide_op_layernorm}
===================================

## General

LayerNorm performs a layer normalization operation on \src tensor.

The layerNorm operation performs normalization from `begin_norm_axis` to last
dimension of the data tensor. It is defined by the following formulas which is
the same as @ref dev_guide_layer_normalization.

\f[
    \dst(t, n, c) =
       \gamma(c) \cdot
       \frac{\src(t, n, c) - \mu(t, n)} {\sqrt{\sigma^2(t, n) + \epsilon}}
       + \beta(c),
\f]

where

- \f$\gamma(c), \beta(c)\f$ are optional scale and shift for a channel

- \f$\mu(t, n), \sigma^2(t, n)\f$ are mean and variance (see

- \f$\epsilon\f$ is a constant to improve numerical stability.

Mean and variance are computed at runtime or provided by a user. When mean and
variance are computed at runtime, the following formulas are used:

- \f$\mu(t, n) = \frac{1}{C} \sum\limits_{c} \src(t, n, c)_{}\f$,

- \f$\sigma^2(t, n) = \frac{1}{C} \sum\limits_{c} {}_{} (\src(t, n, c) - \mu(t, n))^2\f$.

## Operation attributes

| Attribute Name                                                 | Description                                                                                                                                                                                                                                                                                   | Value Type | Supported Values                              | Required or Optional |
|:---------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------|:----------------------------------------------|:---------------------|
| [keep_stats](@ref dnnl::graph::op::attr::keep_stats)           | Indicate whether to output mean and variance which can be later passed to backward op.                                                                                                                                                                                                        | bool       | `false`,`true` (default)                      | Optional             |
| [begin_norm_axis](@ref dnnl::graph::op::attr::begin_norm_axis) | `begin_norm_axis` is used to indicate which axis to start layer normalization. The normalization is from `begin_norm_axis` to last dimension. Negative values means indexing from right to left. This op normalizes over the last dimension by default, e.g. C in TNC for 3D and LDNC for 4D. | s64        | [-r,r-1],where r=rank(src). -1 is default     | Optional             |
| [use_affine](@ref dnnl::graph::op::attr::use_affine)           | When set to True, this module has learnable per-element affine parameters.                                                                                                                                                                                                                    | bool       | `false`, `true` (default)                     | Optional             |
| [epsilon](@ref dnnl::graph::op::attr::epsilon)                 | The constant to improve numerical stability.                                                                                                                                                                                                                                                  | f32        | Arbitrary positive f32 value, `1e-5`(default) | Optional             |

## Execution arguments

The inputs and outputs must be provided according to below index order when
constructing an operation.

### Inputs

| Index | Argument Name | Required or Optional |
|:------|:--------------|:---------------------|
| 0     | `src`         | Required             |
| 1     | `gamma`       | Optional             |
| 2     | `beta`        | Optional             |

@note `gamma` is scaling for normalized value. `beta` is the bias added to
the scaled normalized value. They are both 1D tensor with the same span as src’s
channel axis and required if attribute `use_affine` is set to True.

### Outputs

| Index | Argument Name | Required or Optional |
|:------|:--------------|:---------------------|
| 0     | `dst`         | Required             |
| 1     | `mean`        | Optional             |
| 2     | `variance`    | Optional             |

@note Both `mean` and `variance` are required if attribute `keep_stats` is set to
True.

## Supported data types

LayerNorm operation supports the following data type combinations.

| Src / Dst | Gamma / Beta / Mean / Variance |
|:----------|:-------------------------------|
| f32       | f32                            |
| bf16      | f32, bf16                      |
| f16       | f32                            |