Local Response Normalization#

The LRN primitive performs a forward or backward local response normalization operation defined by the following formulas. Variable names follow the standard Conventions.

Forward#

LRN with algorithm lrn_across_channels:

\[\dst(n, c, h, w) = \left\{k + \frac{\alpha}{n_{l}} \sum\limits_{i=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} (\src(n, c+i, h, w))^2 \right\}^{-\beta} \cdot \src(n, c, h, w),\]

LRN with algorithm lrn_within_channel:

\[\dst(n, c, h, w) = \left\{k + \frac{\alpha}{n_{l}} \sum\limits_{i=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} \sum\limits_{j=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} (\src(n, c, h+i, w+j))^2 \right\}^{-\beta} \cdot \src(n, c, h, w),\]

where \(n_{l}\) is the local_size. Formulas are provided for 2D spatial data case.

Backward#

The backward propagation computes \(\diffsrc(n, c, h, w)\), based on \(\diffdst(n, c, h, w)\) and \(\src(n, c, h, w)\).

Execution Arguments#

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.

Primitive input/output	Execution argument index
\(\src\)	`DNNL_ARG_SRC`
\(\dst\)	`DNNL_ARG_DST`
workspace	`DNNL_ARG_WORKSPACE`
\(\diffsrc\)	`DNNL_ARG_DIFF_SRC`
\(\diffdst\)	`DNNL_ARG_DIFF_DST`

Operation Details#

During training, LRN might or might not require a workspace on forward and backward passes. The behavior is implementation specific. Optimized implementations typically require a workspace and use it to save some intermediate results from the forward pass that accelerate computations on the backward pass. To check whether a workspace is required, query the LRN primitive descriptor for the workspace. Success indicates that the workspace is required and its description will be returned.

Data Type Support#

The LRN primitive supports the following combinations of data types.

Note

Here we abbreviate data types names for readability. For example, dnnl::memory::data_type::f32 is abbreviated to f32.

Propagation	Source / Destination
forward / backward	`f32`, `bf16`
forward	`f16`

Data Representation#

Source, Destination, and Their Gradients#

Like most other primitives, the LRN primitive expects the following tensors:

Spatial	Source / Destination
0D	\(N \times C\)
1D	\(N \times C \times W\)
2D	\(N \times C \times H \times W\)
3D	\(N \times C \times D \times H \times W\)

The LRN primitive is optimized for the following memory formats:

Spatial	Logical tensor	Implementations optimized for memory formats
2D	NCHW	`nchw` (`abcd`), `nhwc` (`acdb`), optimized

Here optimized means the format chosen by the preceding compute-intensive primitive.

Post-ops and Attributes#

The LRN primitive does not support any post-ops or attributes.

API#

struct lrn_forward : public dnnl::primitive #

Local response normalization (LRN) forward propagation primitive.

Public Functions

lrn_forward()#: Default constructor. Produces an empty object.

lrn_forward(const primitive_desc &pd)#

Constructs an LRN forward propagation primitive.

Parameters:: pd – Primitive descriptor for an LRN forward propagation primitive.

struct primitive_desc : public dnnl::primitive_desc #

Primitive descriptor for an LRN forward propagation primitive.

Public Functions

primitive_desc() = default#: Default constructor. Produces an empty object.

primitive_desc(const engine &aengine, prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &src_desc, const memory::desc &dst_desc, memory::dim local_size, float alpha, float beta, float k, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for an LRN forward propagation primitive.

Parameters:

aengine – Engine to use.
aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.
aalgorithm – LRN algorithm kind: either dnnl::algorithm::lrn_across_channels, or dnnl::algorithm::lrn_within_channel.
src_desc – Source memory descriptor.
dst_desc – Destination memory descriptor.
local_size – Regularization local size.
alpha – The alpha regularization parameter.
beta – The beta regularization parameter.
k – The k regularization parameter.
attr – Primitive attributes to use. Attributes are optional and default to empty attributes.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns:: Source memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a source parameter.

memory::desc dst_desc() const#

Returns a destination memory descriptor.

Returns:: Destination memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a destination parameter.

memory::desc workspace_desc() const#

Returns the workspace memory descriptor.

Returns:: Workspace memory descriptor.
Returns:: A zero memory descriptor if the primitive does not require workspace parameter.

algorithm get_algorithm() const#

Returns an algorithm kind.

Returns:: An algorithm kind.
Returns:: dnnl::algorithm::undef if the primitive does not have an algorithm parameter.

prop_kind get_prop_kind() const#

Returns a propagation kind.

Returns:: A propagation kind.
Returns:: dnnl::prop_kind::undef if the primitive does not have a propagation parameter.

float get_alpha() const#

Returns an alpha.

Returns:: An alpha.
Returns:: Zero if the primitive does not have an alpha parameter.

float get_beta() const#

Returns a beta.

Returns:: A beta.
Returns:: Zero if the primitive does not have a beta parameter.

memory::dim get_local_size() const#

Returns an LRN local size parameter.

Returns:: An LRN local size parameter.
Returns:: Zero if the primitive does not have an LRN local size parameter.

float get_k() const#

Returns an LRN K parameter.

Returns:: An LRN K parameter.
Returns:: Zero if the primitive does not have an LRN K parameter.

struct lrn_backward : public dnnl::primitive #

Local response normalization (LRN) backward propagation primitive.

Public Functions

lrn_backward()#: Default constructor. Produces an empty object.

lrn_backward(const primitive_desc &pd)#

Constructs an LRN backward propagation primitive.

Parameters:: pd – Primitive descriptor for an LRN backward propagation primitive.

struct primitive_desc : public dnnl::primitive_desc #

Primitive descriptor for an LRN backward propagation primitive.

Public Functions

primitive_desc() = default#: Default constructor. Produces an empty object.

primitive_desc(const engine &aengine, algorithm aalgorithm, const memory::desc &diff_src_desc, const memory::desc &diff_dst_desc, const memory::desc &src_desc, memory::dim local_size, float alpha, float beta, float k, const lrn_forward::primitive_desc &hint_fwd_pd, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for an LRN backward propagation primitive.

Parameters:

aengine – Engine to use.
aalgorithm – LRN algorithm kind: either dnnl::algorithm::lrn_across_channels, or dnnl::algorithm::lrn_within_channel.
diff_src_desc – Diff source memory descriptor.
diff_dst_desc – Diff destination memory descriptor.
src_desc – Source memory descriptor.
local_size – Regularization local size.
alpha – The alpha regularization parameter.
beta – The beta regularization parameter.
k – The k regularization parameter.
hint_fwd_pd – Primitive descriptor for an LRN forward propagation primitive. It is used as a hint for deciding which memory format to use.
attr – Primitive attributes to use. Attributes are optional and default to empty attributes.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc diff_src_desc() const#

Returns a source memory descriptor.

Returns:: Source memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a source parameter.

memory::desc diff_dst_desc() const#

Returns a diff destination memory descriptor.

Returns:: Diff destination memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a diff destination parameter.

memory::desc workspace_desc() const#

Returns the workspace memory descriptor.

Returns:: Workspace memory descriptor.
Returns:: A zero memory descriptor if the primitive does not require workspace parameter.

algorithm get_algorithm() const#

Returns an algorithm kind.

Returns:: An algorithm kind.
Returns:: dnnl::algorithm::undef if the primitive does not have an algorithm parameter.

prop_kind get_prop_kind() const#

Returns a propagation kind.

Returns:: A propagation kind.
Returns:: dnnl::prop_kind::undef if the primitive does not have a propagation parameter.

float get_alpha() const#

Returns an alpha.

Returns:: An alpha.
Returns:: Zero if the primitive does not have an alpha parameter.

float get_beta() const#

Returns a beta.

Returns:: A beta.
Returns:: Zero if the primitive does not have a beta parameter.

memory::dim get_local_size() const#

Returns an LRN local size parameter.

Returns:: An LRN local size parameter.
Returns:: Zero if the primitive does not have an LRN local size parameter.

float get_k() const#

Returns an LRN K parameter.

Returns:: An LRN K parameter.
Returns:: Zero if the primitive does not have an LRN K parameter.

Local Response Normalization

Contents

Local Response Normalization#

Forward#

Backward#

Execution Arguments#

Operation Details#

Data Type Support#

Data Representation#

Source, Destination, and Their Gradients#

Post-ops and Attributes#

API#