Elementwise

Elementwise#

The elementwise primitive applies an operation to every element of the tensor. Variable names follow the standard Conventions.

\[\dst(\overline{x}) = Operation(\src(\overline{x})),\]

for \(\overline{x} = (x_0, \ldots, x_n)\).

Forward#

The following forward operations are supported. Here \(s\) and \(d\) denote \(\src\) and \(\dst\), tensor values respectively.

Elementwise algorithm	Forward formula
`eltwise_abs`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ -s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_clip`, `eltwise_clip_use_dst_for_bwd`	\(d = \begin{cases} \beta & \text{if}\ s > \beta \geq \alpha \\ s & \text{if}\ \alpha < s \leq \beta \\ \alpha & \text{if}\ s \leq \alpha \end{cases}\)
`eltwise_elu`, `eltwise_elu_use_dst_for_bwd`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha (e^s - 1) & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_exp`, `eltwise_exp_use_dst_for_bwd`	\(d = e^s\)
`eltwise_gelu_erf`	\(d = 0.5 s (1 + erf[\frac{s}{\sqrt{2}}])\)
`eltwise_gelu_tanh`	\(d = 0.5 s (1 + tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)])\)
`eltwise_hardsigmoid`	\(d = \text{max}(0, \text{min}(1, \alpha s + \beta))\)
`eltwise_hardswish`	\(d = s \cdot hardsigmoid(s)\)
`eltwise_linear`	\(d = \alpha s + \beta\)
`eltwise_log`	\(d = \log_{e}{s}\)
`eltwise_logistic`, `eltwise_logistic_use_dst_for_bwd`	\(d = \frac{1}{1+e^{-s}}\)
`eltwise_mish`	\(d = s \cdot \tanh{(\log_{e}(1+e^s))}\)
`eltwise_pow`	\(d = \alpha s^{\beta}\)
`eltwise_relu`, `eltwise_relu_use_dst_for_bwd`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_round`	\(d = round(s)\)
`eltwise_soft_relu`	\(d =\frac{1}{\alpha} \log_{e}(1+e^{\alpha s})\)
`eltwise_sqrt`, `eltwise_sqrt_use_dst_for_bwd`	\(d = \sqrt{s}\)
`eltwise_square`	\(d = s^2\)
`eltwise_swish`	\(d = \frac{s}{1+e^{-\alpha s}}\)
`eltwise_tanh`, `eltwise_tanh_use_dst_for_bwd`	\(d = \tanh{s}\)

Backward#

The backward propagation computes \(\diffsrc(\overline{s})\), based on \(\diffdst(\overline{s})\) and \(\src(\overline{s})\). However, some operations support a computation using \(\dst(\overline{s})\) memory produced during forward propagation. Refer to the table above for a list of operations supporting destination as input memory and the corresponding formulas.

The following backward operations are supported. Here \(s\), \(d\), \(ds\) and \(dd\) denote \(\src\), \(\dst\), \(\diffsrc\), and a \(\diffdst\) tensor values respectively.

Elementwise algorithm	Backward formula
`eltwise_abs`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ -dd & \text{if}\ s < 0 \\ 0 & \text{if}\ s = 0 \end{cases}\)
`eltwise_clip`	\(ds = \begin{cases} dd & \text{if}\ \alpha < s < \beta \\ 0 & \text{otherwise}\ \end{cases}\)
`eltwise_clip_use_dst_for_bwd`	\(ds = \begin{cases} dd & \text{if}\ \alpha < d < \beta \\ 0 & \text{otherwise}\ \end{cases}\)
`eltwise_elu`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ dd \cdot \alpha e^s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_elu_use_dst_for_bwd`	\(ds = \begin{cases} dd & \text{if}\ d > 0 \\ dd \cdot (d + \alpha) & \text{if}\ d \leq 0 \end{cases}\) only if \(\alpha \geq 0\)
`eltwise_exp`	\(ds = dd \cdot e^s\)
`eltwise_exp_use_dst_for_bwd`	\(ds = dd \cdot d\)
`eltwise_gelu_erf`	\(ds = dd \cdot \left(0.5 + 0.5 \, \mathrm{erf}\left({\frac{s}{\sqrt{2}}}\right) + \frac{s}{\sqrt{2\pi}}e^{-0.5s^{2}}\right)\)
`eltwise_gelu_tanh`	\(\begin{array}{rl} ds = & dd \\ & \cdot 0.5 (1 + \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) \\ & \cdot (1 + \sqrt{\frac{2}{\pi}} (s + 0.134145 s^3) \\ & \cdot (1 - \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) ) \end{array}\)
`eltwise_hardsigmoid`	\(ds = \begin{cases} dd \cdot \alpha & \text{if}\ 0 < \alpha s + \beta < 1 \\ 0 & \text{otherwise}\ \end{cases}\)
`eltwise_hardswish`	\(ds = \begin{cases} dd \cdot (2 \alpha + \beta) & \text{if}\ 0 < \alpha s + \beta < 1 \\ dd & \text{if}\ \alpha \cdot s + \beta \geq 1 \\ 0 & \text{otherwise} \end{cases}\)
`eltwise_linear`	\(ds = \alpha \cdot dd\)
`eltwise_log`	\(ds = \frac{dd}{s}\)
`eltwise_logistic`	\(ds = \frac{dd}{1+e^{-s}} \cdot (1 - \frac{1}{1+e^{-s}})\)
`eltwise_mish`	\(ds = dd \cdot \frac{e^{s} \cdot \omega}{\delta^{2}}\) with \(\omega = e^{3s} + 4 \cdot e^{2s} + e^{s} \cdot (4 \cdot s + 6) + 4 \cdot (s + 1)\) and \(\delta = e^{2s} + 2 \cdot e^{s} + 2\)
`eltwise_logistic_use_dst_for_bwd`	\(ds = dd \cdot d \cdot (1 - d)\)
`eltwise_pow`	\(ds = dd \cdot \alpha \beta s^{\beta - 1}\)
`eltwise_relu`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ \alpha \cdot dd & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_relu_use_dst_for_bwd`	\(ds = \begin{cases} dd & \text{if}\ d > 0 \\ \alpha \cdot dd & \text{if}\ d \leq 0 \end{cases}\) only if \(alpha \geq 0\)
`eltwise_soft_relu`	\(ds = \frac{dd}{1 + e^{-\alpha s}}\)
`eltwise_sqrt`	\(ds = \frac{dd}{2\sqrt{s}}\)
`eltwise_sqrt_use_dst_for_bwd`	\(ds = \frac{dd}{2d}\)
`eltwise_square`	\(ds = dd \cdot 2 s\)
`eltwise_swish`	\(ds = \frac{dd}{1 + e^{-\alpha s}}(1 + \alpha s (1 - \frac{1}{1 + e^{-\alpha s}}))\)
`eltwise_tanh`	\(ds = dd \cdot (1 - \tanh^2{s})\)
`eltwise_tanh_use_dst_for_bwd`	\(ds = dd \cdot (1 - d^2)\)

Difference Between Forward Training and Forward Inference#

There is no difference between the #dnnl_forward_training and #dnnl_forward_inference propagation kinds.

Execution Arguments#

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.

Primitive input/output	Execution argument index
\(\src\)	`DNNL_ARG_SRC`
\(\dst\)	`DNNL_ARG_DST`
\(\diffsrc\)	`DNNL_ARG_DIFF_SRC`
\(\diffdst\)	`DNNL_ARG_DIFF_DST`

Operation Details#

The dnnl::eltwise_forward::primitive_desc and dnnl::eltwise_backward::primitive_desc constructors take both parameters \(\alpha\), and \(\beta\). These parameters are ignored if they are unused by the algorithm.
The memory format and data type for \(\src\) and \(\dst\) are assumed to be the same. The same holds for \(\diffsrc\) and \(\diffdst\).
Both forward and backward propagation support in-place operations, meaning that \(\src\) can be used as input and output for forward propagation, and \(\diffdst\) can be used as input and output for backward propagation. In case of an in-place operation, the original data will be overwritten. Note, however, that some algorithms for backward propagation require original \(\src\), hence the corresponding forward propagation should not be performed in-place for those algorithms. Algorithms that use \(\dst\) for backward propagation can be safely done in-place.
For some operations it might be beneficial to compute backward propagation based on \(\dst(\overline{s})\), rather than on \(\src(\overline{s})\), for improved performance.

Note

For operations supporting destination memory as input, \(\dst\) can be used instead of \(\src\) when backward propagation is computed. This enables several performance optimizations (see the tips below).

Data Type Support#

The eltwise primitive should support the following combinations of data types.

Note

Here we abbreviate data types names for readability. For example, dnnl::memory::data_type::f32 is abbreviated to f32.

Propagation	Source / Destination	Intermediate data type
forward / backward	`f32`, `bf16`	`f32`
forward	`f16`	`f16`
forward	`s32` / `s8` / `u8`	`f32`

Here the intermediate data type means that the values coming in are first converted to the intermediate data type, then the operation is applied, and finally the result is converted to the output data type.

Data Representation#

The eltwise primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions.

Post-ops and Attributes#

Type	Operation	Description	Restrictions
Post-op	`Binary`	Applies a binary operation to the result

API#

struct eltwise_forward : public dnnl::primitive #

Elementwise unary operation forward propagation primitive.

Public Functions

eltwise_forward()#: Default constructor. Produces an empty object.

eltwise_forward(const primitive_desc &pd)#

Constructs an eltwise forward propagation primitive.

Parameters:: pd – Primitive descriptor for an eltwise forward propagation primitive.

struct primitive_desc : public dnnl::primitive_desc #

Primitive descriptor for an elementwise forward propagation primitive.

Public Functions

primitive_desc() = default#: Default constructor. Produces an empty object.

primitive_desc(const engine &aengine, prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &src_desc, const memory::desc &dst_desc, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive.

Parameters:

aengine – Engine to use.
aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.
aalgorithm – Elementwise algorithm kind.
src_desc – Source memory descriptor.
dst_desc – Destination memory descriptor.
attr – Primitive attributes to use. Attributes are optional and default to empty attributes.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

primitive_desc(const engine &aengine, prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &src_desc, const memory::desc &dst_desc, float alpha, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive with an alpha parameter.

Parameters:

aengine – Engine to use.
aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.
aalgorithm – Elementwise algorithm kind.
src_desc – Source memory descriptor.
dst_desc – Destination memory descriptor.
alpha – The alpha parameter for the elementwise operation. Specific meaning depends on the algorithm.
attr – Primitive attributes to use. Attributes are optional and default to empty attributes.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

primitive_desc(const engine &aengine, prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &src_desc, const memory::desc &dst_desc, float alpha, float beta, const primitive_attr &attr = default_attr(), bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive with an alpha and beta parameters.

Parameters:

aengine – Engine to use.
aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.
aalgorithm – Elementwise algorithm kind.
src_desc – Source memory descriptor.
dst_desc – Destination memory descriptor.
alpha – The alpha parameter for the elementwise operation. Specific meaning depends on the algorithm.
beta – The beta parameter for the elementwise operation. Specific meaning depends on the algorithm.
attr – Primitive attributes to use. Attributes are optional and default to empty attributes.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns:: Source memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a source parameter.

memory::desc dst_desc() const#

Returns a destination memory descriptor.

Returns:: Destination memory descriptor.
Returns:: A zero memory descriptor if the primitive does not have a destination parameter.

dnnl::algorithm get_algorithm() const#

Returns an algorithm kind.

Returns:: An algorithm kind.
Returns:: dnnl::algorithm::undef if the primitive does not have an algorithm parameter.

dnnl::prop_kind get_prop_kind() const#

Returns a propagation kind.

Returns:: A propagation kind.
Returns:: dnnl::prop_kind::undef if the primitive does not have a propagation parameter.

float get_alpha() const#

Returns an alpha.

Returns:: An alpha.
Returns:: Zero if the primitive does not have an alpha parameter.

float get_beta() const#

Returns a beta.

Returns:: A beta.
Returns:: Zero if the primitive does not have a beta parameter.

struct eltwise_backward : public dnnl::primitive #

Elementwise unary operation backward propagation primitive.

Elementwise

Contents

Elementwise#

Forward#

Backward#

Difference Between Forward Training and Forward Inference#

Execution Arguments#

Operation Details#

Data Type Support#

Data Representation#

Post-ops and Attributes#

API#