🚀 Feature

There is a new activation function ACON (CVPR 2021) that unifies ReLU and Swish.
ACON is simple but very effective, code is here: https://github.com/nmaac/acon/blob/main/acon.py#L19

The improvements are very significant:

Motivation

Pitch

I would like to suggest replacing SiLU with ACON directly because SiLU (Swish) is used in your project, its general and effective form ACON may also show improvements.

Alternatives

It also has an enhanced version meta-ACON that uses a small network to learn beta explicitly, which may influence the speed a bit.

Additional context

Code and paper.