# Normalizations (batchnorm-forward)= ## Batchnorm Forward The batchnorm operation computes: $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Optionally the operation also computes: $$next\_running\_mean = (1 - momentum)*previous\_running\_mean + momentum*current\_running\_mean$$ $$next\_running\_variance = (1 - momentum)*previous\_running\_variance + momentum*current\_running\_variance$$ ### C++ API ``` std::array, 5> batchnorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Batchnorm_attributes attributes); ``` Where the output array has tensors in order of: `[output, saved_mean, saved_invariance, next_running_mean, next_running_variance]` Batchnorm attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Batchnorm_attributes& set_previous_running_stats(std::shared_ptr& previous_running_mean, std::shared_ptr& previous_running_variance, std::shared_ptr& momentum) Batchnorm_attributes& set_name(std::string const&) Batchnorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - batchnorm - input - scale - bias - in_running_mean - in_running_var - epsilon - momentum - compute_data_type - name (batchnorm-finalize)= ## Batchnorm Finalize The `bn_finalize` operation calculates the statistics required for the next iteration from the statistics generated by the `genstat` operation. ### C++ API ``` std::array, 6> bn_finalize(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, BN_finalize_attributes); ``` The outputs are `[EQ_SCALE, EQ_BIAS, MEAN, INV_VARIANCE, NEXT_RUNNING_MEAN, NEXT_RUNNING_VAR]`. (batchnorm-backward-dbn)= ## Batchnorm Backward (DBN) The DBN operation computes data gradient, scale gradient, bias gradient during backpropagation of batchnorm forward operation. ### C++ API ``` std::array, 3> batchnorm_backward(std::shared_ptr loss, std::shared_ptr input, std::shared_ptr scale, Batchnorm_backward_attributes); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. DBN attributes is a lightweight structure with setters: ``` Batchnorm_backward_attributes& set_saved_mean_and_inv_variance(std::shared_ptr saved_mean, std::shared_ptr saved_inverse_variance) Batchnorm_backward_attributes& set_epsilon(std::shared_ptr epsilon) Batchnorm_backward_attributes& set_name(std::string const&) Batchnorm_backward_attributes& set_compute_data_type(DataType_t value) ``` Only setting either (saved mean and inverse_variance) or (epsilon) is necessary. (generate-stats)= ## Generate Stats The Genstats operation computes the sum and sum of squares per-channel dimension. ### C++ API ``` std::array, 2> cudnn_frontend::graph::genstats(std::shared_ptr, Genstats_attributes); ``` The output array has tensors in order of: `[sum, square_sum]` Genstats attributes is a lightweight structure with setters: ``` Genstats_attributes& set_name(std::string const&) Genstats_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - genstats - input - compute_data_type - name (layernorm-forward)= ## Layernorm Forward The layernorm operation computes $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Normalization happens across features, independently for each sample. ### C++ API ``` std::array, 3> layernorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Layernorm_attributes attributes); ``` The output array has tensors in order of: `[output, mean, variance]` Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Layernorm_attributes& set_name(std::string const&) Layernorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - layernorm - norm_forward_phase - input - scale - bias - epsilon - compute_data_type - name ## Layernorm Backward (DLN) The DLN operation computes data gradient, scale gradient, and bias gradient during backpropagation of a layernorm forward operation. ### C++ API ``` std::array, 3> layernorm_backward(std::shared_ptr dy, std::shared_ptr x, std::shared_ptr scale, Layernorm_backward_attributes options); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. Layernorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Layernorm_attributes& set_name(std::string const&) Layernorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - layernorm - input - scale - loss - compute_data_type - name ## Instancenorm Forward The instancenorm operation computes $$ output = scale*{input - mean \over \sqrt{variance + epsilon}} + bias $$ Normalization happens across each sample. ### C++ API ``` std::array, 3> instancenorm(std::shared_ptr& input, std::shared_ptr& scale, std::shared_ptr& bias, Instancenorm_attributes attributes); ``` The output array has tensors in order of: `[output, mean, variance]`. Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Instancenorm_attributes& set_name(std::string const&) Instancenorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - instancenorm - norm_forward_phase - input - scale - bias - epsilon - compute_data_type - name ## Instancenorm Backward (DIN) The DIN operation computes data gradient, scale gradient, and bias gradient during backpropagation of an instancenorm forward operation. ### C++ API ``` std::array, 3> instancenorm_backward(std::shared_ptr dy, std::shared_ptr x, std::shared_ptr scale, Instancenorm_backward_attributes options); ``` The output array has tensors in order of: `[input gradient, scale gradient, bias gradient]`. Instancenorm_attributes is a lightweight structure with setters for providing optional input tensors and other operation attributes: ``` Instancenorm_attributes& set_name(std::string const&) Instancenorm_attributes& set_compute_data_type(DataType_t value) ``` ### Python API - layernorm - input - scale - loss - compute_data_type - name