Prelu#

The prelu primitive (Leaky ReLU with trainable alpha parameter) performs forward or backward operation on data tensor. Weights (alpha) tensor supports broadcast-semantics.

Broadcast configuration is assumed based on src and weights dimensions.

Forward#

The prelu operation is defined by the following formulas. We show formulas only for 2D spatial data which are straightforward to generalize to cases of higher and lower dimensions. Variable names follow the standard Conventions.

\[\begin{split}\dst(n, c, h, w) = \begin{cases} \src(n, c, h, w) & \mbox{if } \src(n, c, h, w) > 0 \\ \src(n, c, h, w) \cdot \weights(n, c, h, w) & \mbox{if } \src(n, c, h, w) \leq 0 \end{cases}\end{split}\]

Difference Between Forward Training and Forward Inference#

There is no difference between the forward_training and forward_inference propagation kinds.

Backward#

The backward propagation computes \(\diffsrc\) and \(\diffweights\). For no broadcast case, results are calculated using formula:

\[\begin{split}\diffsrc(n, c, h, w) &= \begin{cases} \diffdst(n, c, h, w) & if \src(n, c, h, w) > 0 \\ \diffdst(n, c, h, w) \cdot \weights(n, c, h, w) & if \src(n, c, h, w) \leq 0 \end{cases}\\\\\end{split}\]
\[\diffweights(n, c, h, w) = \min(\src(n, c, h, w), 0) \cdot \diffdst(n, c, h, w)\]

Execution Arguments#

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.

Primitive input/output

Execution argument index

\(\src\)

DNNL_ARG_SRC

\(\dst\)

DNNL_ARG_DST

\(\weights\)

DNNL_ARG_WEIGHTS

\(\diffsrc\)

DNNL_ARG_DIFF_SRC

\(\diffdst\)

DNNL_ARG_DIFF_DST

\(\diffweights\)

DNNL_ARG_DIFF_WEIGHTS

Operation Details#

  1. All input and output tensors must have the same number of dimensions.

  2. \(\weights\) tensor dimensions must follow broadcast semantics. Each dimension can either be equal to the corresponding data dimension or equal to 1 to indicate a broadcasted dimension.

Post-ops and Attributes#

The prelu primitive does not have to support any post-ops or attributes.

Data Types Support#

The PReLU primitive supports the following combinations of data types:

Note

Here we abbreviate data types names for readability. For example, dnnl::memory::data_type::f32 is abbreviated to f32.

Propagation

Source / Destination

forward / backward

f32, s32, bf16, f16, s8, u8

Data Representation#

Source, Destination, and Their Gradients#

The PReLU primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions.

API#

struct prelu_forward : public dnnl::primitive#

PReLU forward propagation primitive.

Public Functions

prelu_forward() = default#

Default constructor. Produces an empty object.

prelu_forward(const primitive_desc &pd)#

Constructs a prelu forward propagation primitive.

Parameters:

pd – Primitive descriptor for a prelu forward propagation primitive.

struct primitive_desc : public dnnl::primitive_desc#

Primitive descriptor for a PReLU forward propagation primitive.

Public Functions

primitive_desc() = default#

Default constructor. Produces an empty object.

primitive_desc(const engine &aengine, prop_kind aprop_kind, const memory::desc &src_desc, const memory::desc &weight_desc, const memory::desc &dst_desc, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for a PReLU forward propagation primitive.

Parameters:

  • aengine – Engine to use.

  • aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.

  • src_desc – Source memory descriptor.

  • weight_desc – Alpha parameters memory descriptor.

  • dst_desc – Destination memory descriptor.

  • attr – Primitive attributes to use. Attributes are optional and default to empty attributes.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns:

Source memory descriptor.

Returns:

A zero memory descriptor if the primitive does not have a source parameter.

memory::desc dst_desc() const#

Returns a destination memory descriptor.

Returns:

Destination memory descriptor.

Returns:

A zero memory descriptor if the primitive does not have a destination parameter.

prop_kind get_prop_kind() const#

Returns a propagation kind.

Returns:

A propagation kind.

Returns:

dnnl::prop_kind::undef if the primitive does not have a propagation parameter.

struct prelu_backward : public dnnl::primitive#

PReLU backward propagation primitive.

Public Functions

prelu_backward() = default#

Default constructor. Produces an empty object.

prelu_backward(const primitive_desc &pd)#

Constructs a prelu backward propagation primitive.

Parameters:

pd – Primitive descriptor for a prelu backward propagation primitive.

struct primitive_desc : public dnnl::primitive_desc#

Primitive descriptor for prelu backward propagation.

Public Functions

primitive_desc() = default#

Default constructor. Produces an empty object.

primitive_desc(const engine &aengine, const memory::desc &src_desc, const memory::desc &weight_desc, const memory::desc &diff_src_desc, const memory::desc &diff_weights_desc, const memory::desc &diff_dst_desc, const prelu_forward::primitive_desc &hint_fwd_pd, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a descriptor for a PReLU backward propagation primitive.

Parameters:

  • aengine – Engine to use.

  • src_desc – Source memory descriptor.

  • weight_desc – Alpha parameters memory descriptor.

  • diff_src_desc – Diff source memory descriptor.

  • diff_weights_desc – Diff alpha parameters memory descriptor.

  • diff_dst_desc – Diff destination memory descriptor.

  • hint_fwd_pd – Primitive descriptor for a PReLU forward propagation primitive. It is used as a hint for deciding which memory format to use.

  • attr – Primitive attributes to use. Attributes are optional and default to empty attributes.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns:

Source memory descriptor.

Returns:

A zero memory descriptor if the primitive does not have a source parameter.

memory::desc diff_src_desc() const#

Returns a diff source memory descriptor.

Returns:

Diff source memory descriptor.

Returns:

A zero memory descriptor if the primitive does not have a diff source memory with.

memory::desc diff_dst_desc() const#

Returns a diff destination memory descriptor.

Returns:

Diff destination memory descriptor.

Returns:

A zero memory descriptor if the primitive does not have a diff destination parameter.

prop_kind get_prop_kind() const#

Returns a propagation kind.

Returns:

A propagation kind.

Returns:

dnnl::prop_kind::undef if the primitive does not have a propagation parameter.