Elementwise#

The elementwise primitive applies an operation to every element of the tensor. Variable names follow the standard Conventions.

\[\dst(\overline{x}) = Operation(\src(\overline{x})),\]

for \(\overline{x} = (x_0, \ldots, x_n)\).

Forward#

The following forward operations are supported. Here \(s\) and \(d\) denote \(\src\) and \(\dst\), tensor values respectively.

Elementwise algorithm	Forward formula
`eltwise_abs`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ -s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_bounded_relu`	\(d = \begin{cases} \alpha & \text{if}\ s > \alpha \geq 0 \\ s & \text{if}\ 0 < s \leq \alpha \\ 0 & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_clip`	\(d = \begin{cases} \beta & \text{if}\ s > \beta \geq \alpha \\ s & \text{if}\ \alpha < s \leq \beta \\ \alpha & \text{if}\ s \leq \alpha \end{cases}\)
`eltwise_elu`, `eltwise_elu_use_dst_for_bwd`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha (e^s - 1) & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_exp`, `eltwise_exp_use_dst_for_bwd`	\(d = e^s\)
`eltwise_gelu_erf`	\(d = 0.5 s (1 + erf[\frac{s}{\sqrt{2}}])\)
`eltwise_gelu_tanh`	\(d = 0.5 s (1 + tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)])\)
`eltwise_linear`	\(d = \alpha s + \beta\)
`eltwise_log`	\(d = \log_{e}{s}\)
`eltwise_logistic`, `eltwise_logistic_use_dst_for_bwd`	\(d = \frac{1}{1+e^{-s}}\)
`eltwise_pow`	\(d = \alpha s^{\beta}\)
`eltwise_relu`, `eltwise_relu_use_dst_for_bwd`	\(d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_round`	\(d = round(s)\)
`eltwise_soft_relu`	\(d = \log_{e}(1+e^s)\)
`eltwise_sqrt`, `eltwise_sqrt_use_dst_for_bwd`	\(d = \sqrt{s}\)
`eltwise_square`	\(d = s^2\)
`eltwise_swish`	\(d = \frac{s}{1+e^{-\alpha s}}\)
`eltwise_tanh`, `eltwise_tanh_use_dst_for_bwd`	\(d = \tanh{s}\)

Backward#

The backward propagation computes \(\diffsrc(\overline{s})\), based on \(\diffdst(\overline{s})\) and \(\src(\overline{s})\). However, some operations support a computation using \(\dst(\overline{s})\) memory produced during forward propagation. Refer to the table above for a list of operations supporting destination as input memory and the corresponding formulas.

The following backward operations are supported. Here \(s\), \(d\), \(ds\) and \(dd\) denote \(\src\), \(\dst\), \(\diffsrc\), and a \(\diffdst\) tensor values respectively.

Elementwise algorithm	Backward formula
`eltwise_abs`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ -dd & \text{if}\ s < 0 \\ 0 & \text{if}\ s = 0 \end{cases}\)
`eltwise_bounded_relu`	\(ds = \begin{cases} dd & \text{if}\ 0 < s \leq \alpha, \\ 0 & \text{otherwise}\ \end{cases}\)
`eltwise_clip`	\(ds = \begin{cases} dd & \text{if}\ \alpha < s \leq \beta \\ 0 & \text{otherwise}\ \end{cases}\)
`eltwise_elu`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ dd \cdot \alpha e^s & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_elu_use_dst_for_bwd`	\(ds = \begin{cases} dd & \text{if}\ d > 0 \\ dd \cdot (d + \alpha) & \text{if}\ d \leq 0 \end{cases}\) only if \(\alpha \geq 0\)
`eltwise_exp`	\(ds = dd \cdot e^s\)
`eltwise_exp_use_dst_for_bwd`	\(ds = dd \cdot d\)
`eltwise_gelu_erf`	\(ds = dd \cdot \left(0.5 + 0.5 \, \mathrm{erf}\left({\frac{s}{\sqrt{2}}}\right) + \frac{s}{\sqrt{2\pi}}e^{-0.5s^{2}}\right)\)
`eltwise_gelu_tanh`	\(\begin{array}{rl} ds = & dd \\ & \cdot 0.5 (1 + \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) \\ & \cdot (1 + \sqrt{\frac{2}{\pi}} (s + 0.134145 s^3) \\ & \cdot (1 - \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) ) \end{array}\)
`eltwise_linear`	\(ds = \alpha \cdot dd\)
`eltwise_log`	\(ds = \frac{dd}{s}\)
`eltwise_logistic`	\(ds = \frac{dd}{1+e^{-s}} \cdot (1 - \frac{1}{1+e^{-s}})\)
`eltwise_logistic_use_dst_for_bwd`	\(ds = dd \cdot d \cdot (1 - d)\)
`eltwise_pow`	\(ds = dd \cdot \alpha \beta s^{\beta - 1}\)
`eltwise_relu`	\(ds = \begin{cases} dd & \text{if}\ s > 0 \\ \alpha \cdot dd & \text{if}\ s \leq 0 \end{cases}\)
`eltwise_relu_use_dst_for_bwd`	\(ds = \begin{cases} dd & \text{if}\ d > 0 \\ \alpha \cdot dd & \text{if}\ d \leq 0 \end{cases}\) only if \(alpha \geq 0\)
`eltwise_soft_relu`	\(ds = \frac{dd}{1 + e^{-s}}\)
`eltwise_sqrt`	\(ds = \frac{dd}{2\sqrt{s}}\)
`eltwise_sqrt_use_dst_for_bwd`	\(ds = \frac{dd}{2d}\)
`eltwise_square`	\(ds = dd \cdot 2 s\)
`eltwise_swish`	\(ds = \frac{dd}{1 + e^{-\alpha s}}(1 + \alpha s (1 - \frac{1}{1 + e^{-\alpha s}}))\)
`eltwise_tanh`	\(ds = dd \cdot (1 - \tanh^2{s})\)
`eltwise_tanh_use_dst_for_bwd`	\(ds = dd \cdot (1 - d^2)\)

Difference Between Forward Training and Forward Inference#

There is no difference between the #dnnl_forward_training and #dnnl_forward_inference propagation kinds.

Execution Arguments#

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.

Primitive input/output	Execution argument index
\(\src\)	`DNNL_ARG_SRC`
\(\dst\)	`DNNL_ARG_DST`
\(\diffsrc\)	`DNNL_ARG_DIFF_SRC`
\(\diffdst\)	`DNNL_ARG_DIFF_DST`

Operation Details#

The dnnl::eltwise_forward::desc::desc() and dnnl::eltwise_backward::desc::desc() constructors take both parameters \(\alpha\), and \(\beta\). These parameters are ignored if they are unused by the algorithm.
The memory format and data type for \(\src\) and \(\dst\) are assumed to be the same, and in the API are typically denoted as data (for example dnnl::eltwise_forward::desc::desc() has a data_desc argument). The same holds for \(\diffsrc\) and \(\diffdst\). The corresponding memory descriptors are denoted as diff_data_desc.
Both forward and backward propagation support in-place operations, meaning that \(\src\) can be used as input and output for forward propagation, and \(\diffdst\) can be used as input and output for backward propagation. In case of an in-place operation, the original data will be overwritten. Note, however, that some algorithms for backward propagation require original \(\src\), hence the corresponding forward propagation should not be performed in-place for those algorithms. Algorithms that use \(\dst\) for backward propagation can be safely done in-place.
For some operations it might be beneficial to compute backward propagation based on \(\dst(\overline{s})\), rather than on \(\src(\overline{s})\), for improved performance.

Note

For operations supporting destination memory as input, \(\dst\) can be used instead of \(\src\) when backward propagation is computed. This enables several performance optimizations (see the tips below).

Data Type Support#

The eltwise primitive should support the following combinations of data types.

Note

Here we abbreviate data types names for readability. For example, dnnl::memory::data_type::f32 is abbreviated to f32.

Propagation	Source / Destination	Intermediate data type
forward / backward	`f32`, `bf16`	`f32`
forward	`f16`	`f16`
forward	`s32` / `s8` / `u8`	`f32`

Here the intermediate data type means that the values coming in are first converted to the intermediate data type, then the operation is applied, and finally the result is converted to the output data type.

Data Representation#

The eltwise primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions.

Post-ops and Attributes#

The eltwise primitive does not have to support any post-ops or attributes.

API#

struct eltwise_forward : public dnnl::primitive #

Elementwise unary operation forward propagation primitive.

Public Functions

eltwise_forward()#: Default constructor. Produces an empty object.

eltwise_forward(const primitive_desc &pd)#

Constructs an eltwise forward propagation primitive.

Parameters: pd – Primitive descriptor for an eltwise forward propagation primitive.

struct desc#

Descriptor for an elementwise forward propagation primitive.

Public Functions

desc(prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &data_desc, float alpha = 0, float beta = 0)#

Constructs a descriptor for an elementwise forward propagation primitive.

Parameters

aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.
aalgorithm – Elementwise algorithm kind.
data_desc – Source and destination memory descriptors.
alpha – The alpha parameter for the elementwise operation. Specific meaning depends on the algorithm.
beta – The beta parameter for the elementwise operation. Specific meaning depends on the algorithm.

struct primitive_desc : public dnnl::primitive_desc #

Primitive descriptor for an elementwise forward propagation primitive.

Public Functions

primitive_desc()#: Default constructor. Produces an empty object.

primitive_desc(const desc &adesc, const engine &aengine, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive.

Parameters

adesc – Descriptor for an elementwise forward propagation primitive.
aengine – Engine to use.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

primitive_desc(const desc &adesc, const primitive_attr &attr, const engine &aengine, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive.

Parameters

adesc – Descriptor for an elementwise forward propagation primitive.
aengine – Engine to use.
attr – Primitive attributes to use.
allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns: Source memory descriptor.
Returns: A zero memory descriptor if the primitive does not have a source parameter.

memory::desc dst_desc() const#

Returns a destination memory descriptor.

Returns: Destination memory descriptor.
Returns: A zero memory descriptor if the primitive does not have a destination parameter.

struct eltwise_backward : public dnnl::primitive #

Elementwise unary operation backward propagation primitive.

oneAPI Specification 1.2-provisional-rev-1 documentation

Elementwise

Contents

Elementwise#

Forward#

Backward#

Difference Between Forward Training and Forward Inference#

Execution Arguments#

Operation Details#

Data Type Support#

Data Representation#

Post-ops and Attributes#

API#