LeakyReLU

LeakyReLU operation is a type of activation function based on ReLU. It has a small slope for negative values with which LeakyReLU can produce small, non-zero, and constant gradients with respect to the negative values. The slope is also called the coefficient of leakage.

Unlike PReLU, the coefficient \(\alpha\) is constant and defined before training.

LeakyReLU operation applies following formula on every element of src tensor (the variable names follow the standard Conventions):

\[\begin{split}\dst = \begin{cases} \src & \text{if}\ \src \ge 0 \\ \alpha \src & \text{if}\ \src < 0 \end{cases}\end{split}\]

Operation Attributes

Attribute Name	Description	Value Type	Supported Values	Required or Optional
alpha	Alpha is the coefficient of leakage.	f32	Arbitrary f32 value but usually a small positive value.	Required

Execution Arguments

The inputs and outputs must be provided according to the below index order when constructing an operation.

Inputs

Index	Argument Name	Required or Optional
0	src	Required

Outputs

Index	Argument Name	Required or Optional
0	dst	Required

Supported Data Types

LeakyReLU operation supports the following data type combinations.

Src	Dst
f32	f32
bf16	bf16
f16	f16