Elementwise#

The elementwise primitive applies an operation to every element of the tensor. Variable names follow the standard Conventions.

\dst(x)=Operation(\src(x)),

for x=(x0,,xn).

Forward#

The following forward operations are supported. Here s and d denote \src and \dst, tensor values respectively.

Elementwise algorithm

Forward formula

eltwise_abs

d={sif s>0sif s0

eltwise_bounded_relu

d={αif s>α0sif 0<sα0if s0

eltwise_clip

d={βif s>βαsif α<sβαif sα

eltwise_elu, eltwise_elu_use_dst_for_bwd

d={sif s>0α(es1)if s0

eltwise_exp, eltwise_exp_use_dst_for_bwd

d=es

eltwise_gelu_erf

d=0.5s(1+erf[s2])

eltwise_gelu_tanh

d=0.5s(1+tanh[2π(s+0.044715s3)])

eltwise_linear

d=αs+β

eltwise_log

d=loges

eltwise_logistic, eltwise_logistic_use_dst_for_bwd

d=11+es

eltwise_pow

d=αsβ

eltwise_relu, eltwise_relu_use_dst_for_bwd

d={sif s>0αsif s0

eltwise_round

d=round(s)

eltwise_soft_relu

d=loge(1+es)

eltwise_sqrt, eltwise_sqrt_use_dst_for_bwd

d=s

eltwise_square

d=s2

eltwise_swish

d=s1+eαs

eltwise_tanh, eltwise_tanh_use_dst_for_bwd

d=tanhs

Backward#

The backward propagation computes \diffsrc(s), based on \diffdst(s) and \src(s). However, some operations support a computation using \dst(s) memory produced during forward propagation. Refer to the table above for a list of operations supporting destination as input memory and the corresponding formulas.

The following backward operations are supported. Here s, d, ds and dd denote \src, \dst, \diffsrc, and a \diffdst tensor values respectively.

Elementwise algorithm

Backward formula

eltwise_abs

ds={ddif s>0ddif s<00if s=0

eltwise_bounded_relu

ds={ddif 0<sα,0otherwise 

eltwise_clip

ds={ddif α<sβ0otherwise 

eltwise_elu

ds={ddif s>0ddαesif s0

eltwise_elu_use_dst_for_bwd

ds={ddif d>0dd(d+α)if d0 only if α0

eltwise_exp

ds=ddes

eltwise_exp_use_dst_for_bwd

ds=ddd

eltwise_gelu_erf

ds=dd(0.5+0.5erf(s2)+s2πe0.5s2)

eltwise_gelu_tanh

ds=dd0.5(1+tanh[2π(s+0.044715s3)])(1+2π(s+0.134145s3)(1tanh[2π(s+0.044715s3)]))

eltwise_linear

ds=αdd

eltwise_log

ds=dds

eltwise_logistic

ds=dd1+es(111+es)

eltwise_logistic_use_dst_for_bwd

ds=ddd(1d)

eltwise_pow

ds=ddαβsβ1

eltwise_relu

ds={ddif s>0αddif s0

eltwise_relu_use_dst_for_bwd

ds={ddif d>0αddif d0 only if alpha0

eltwise_soft_relu

ds=dd1+es

eltwise_sqrt

ds=dd2s

eltwise_sqrt_use_dst_for_bwd

ds=dd2d

eltwise_square

ds=dd2s

eltwise_swish

ds=dd1+eαs(1+αs(111+eαs))

eltwise_tanh

ds=dd(1tanh2s)

eltwise_tanh_use_dst_for_bwd

ds=dd(1d2)

Difference Between Forward Training and Forward Inference#

There is no difference between the #dnnl_forward_training and #dnnl_forward_inference propagation kinds.

Execution Arguments#

When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table.

Primitive input/output

Execution argument index

\src

DNNL_ARG_SRC

\dst

DNNL_ARG_DST

\diffsrc

DNNL_ARG_DIFF_SRC

\diffdst

DNNL_ARG_DIFF_DST

Operation Details#

  1. The dnnl::eltwise_forward::desc::desc() and dnnl::eltwise_backward::desc::desc() constructors take both parameters α, and β. These parameters are ignored if they are unused by the algorithm.

  2. The memory format and data type for \src and \dst are assumed to be the same, and in the API are typically denoted as data (for example dnnl::eltwise_forward::desc::desc() has a data_desc argument). The same holds for \diffsrc and \diffdst. The corresponding memory descriptors are denoted as diff_data_desc.

  3. Both forward and backward propagation support in-place operations, meaning that \src can be used as input and output for forward propagation, and \diffdst can be used as input and output for backward propagation. In case of an in-place operation, the original data will be overwritten. Note, however, that some algorithms for backward propagation require original \src, hence the corresponding forward propagation should not be performed in-place for those algorithms. Algorithms that use \dst for backward propagation can be safely done in-place.

  4. For some operations it might be beneficial to compute backward propagation based on \dst(s), rather than on \src(s), for improved performance.

Note

For operations supporting destination memory as input, \dst can be used instead of \src when backward propagation is computed. This enables several performance optimizations (see the tips below).

Data Type Support#

The eltwise primitive should support the following combinations of data types.

Note

Here we abbreviate data types names for readability. For example, dnnl::memory::data_type::f32 is abbreviated to f32.

Propagation

Source / Destination

Intermediate data type

forward / backward

f32, bf16

f32

forward

f16

f16

forward

s32 / s8 / u8

f32

Here the intermediate data type means that the values coming in are first converted to the intermediate data type, then the operation is applied, and finally the result is converted to the output data type.

Data Representation#

The eltwise primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions.

Post-ops and Attributes#

The eltwise primitive does not have to support any post-ops or attributes.

API#

struct eltwise_forward : public dnnl::primitive#

Elementwise unary operation forward propagation primitive.

Public Functions

eltwise_forward()#

Default constructor. Produces an empty object.

eltwise_forward(const primitive_desc &pd)#

Constructs an eltwise forward propagation primitive.

Parameters

pd – Primitive descriptor for an eltwise forward propagation primitive.

struct desc#

Descriptor for an elementwise forward propagation primitive.

Public Functions

desc(prop_kind aprop_kind, algorithm aalgorithm, const memory::desc &data_desc, float alpha = 0, float beta = 0)#

Constructs a descriptor for an elementwise forward propagation primitive.

Parameters
  • aprop_kind – Propagation kind. Possible values are dnnl::prop_kind::forward_training, and dnnl::prop_kind::forward_inference.

  • aalgorithm – Elementwise algorithm kind.

  • data_desc – Source and destination memory descriptors.

  • alpha – The alpha parameter for the elementwise operation. Specific meaning depends on the algorithm.

  • beta – The beta parameter for the elementwise operation. Specific meaning depends on the algorithm.

struct primitive_desc : public dnnl::primitive_desc#

Primitive descriptor for an elementwise forward propagation primitive.

Public Functions

primitive_desc()#

Default constructor. Produces an empty object.

primitive_desc(const desc &adesc, const engine &aengine, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive.

Parameters
  • adesc – Descriptor for an elementwise forward propagation primitive.

  • aengine – Engine to use.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

primitive_desc(const desc &adesc, const primitive_attr &attr, const engine &aengine, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise forward propagation primitive.

Parameters
  • adesc – Descriptor for an elementwise forward propagation primitive.

  • aengine – Engine to use.

  • attr – Primitive attributes to use.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns

Source memory descriptor.

Returns

A zero memory descriptor if the primitive does not have a source parameter.

memory::desc dst_desc() const#

Returns a destination memory descriptor.

Returns

Destination memory descriptor.

Returns

A zero memory descriptor if the primitive does not have a destination parameter.

struct eltwise_backward : public dnnl::primitive#

Elementwise unary operation backward propagation primitive.

See also

eltwise_forward

Public Functions

eltwise_backward()#

Default constructor. Produces an empty object.

eltwise_backward(const primitive_desc &pd)#

Constructs an eltwise backward propagation primitive.

Parameters

pd – Primitive descriptor for an eltwise backward propagation primitive.

struct desc#

Descriptor for an elementwise backward propagation primitive.

Public Functions

desc(algorithm aalgorithm, const memory::desc &diff_data_desc, const memory::desc &data_desc, float alpha = 0, float beta = 0)#

Constructs a descriptor for an elementwise backward propagation primitive.

Parameters
  • aalgorithm – Elementwise algorithm kind.

  • diff_data_desc – Diff source and destination memory descriptors.

  • data_desc – Source memory descriptor.

  • alpha – The alpha parameter for the elementwise operation. Specific meaning depends on the algorithm.

  • beta – The beta parameter for the elementwise operation. Specific meaning depends on the algorithm.

struct primitive_desc : public dnnl::primitive_desc#

Primitive descriptor for eltwise backward propagation.

Public Functions

primitive_desc()#

Default constructor. Produces an empty object.

primitive_desc(const desc &adesc, const engine &aengine, const eltwise_forward::primitive_desc &hint_fwd_pd, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise backward propagation primitive.

Parameters
  • adesc – Descriptor for an elementwise backward propagation primitive.

  • aengine – Engine to use.

  • hint_fwd_pd – Primitive descriptor for an elementwise forward propagation primitive. It is used as a hint for deciding which memory format to use.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

primitive_desc(const desc &adesc, const primitive_attr &attr, const engine &aengine, const eltwise_forward::primitive_desc &hint_fwd_pd, bool allow_empty = false)#

Constructs a primitive descriptor for an elementwise backward propagation primitive.

Parameters
  • adesc – Descriptor for an elementwise backward propagation primitive.

  • attr – Primitive attributes to use.

  • aengine – Engine to use.

  • hint_fwd_pd – Primitive descriptor for an elementwise forward propagation primitive. It is used as a hint for deciding which memory format to use.

  • allow_empty – A flag signifying whether construction is allowed to fail without throwing an exception. In this case an empty object will be produced. This flag is optional and defaults to false.

memory::desc src_desc() const#

Returns a source memory descriptor.

Returns

Source memory descriptor.

Returns

A zero memory descriptor if the primitive does not have a source parameter.

memory::desc diff_src_desc() const#

Returns a diff source memory descriptor.

Returns

Diff source memory descriptor.

Returns

A zero memory descriptor if the primitive does not have a diff source memory with.

memory::desc diff_dst_desc() const#

Returns a diff destination memory descriptor.

Returns

Diff destination memory descriptor.

Returns

A zero memory descriptor if the primitive does not have a diff destination parameter.