support for selu activation function

[BorisVSchmid]

Ran into an interesting discussion on hackers' news on an activation function that seems to work well, even with deeper fully connected networks. See comparison and paper here, and discussion at reddit and HN.

https://github.com/shaohua0116/Activation-Visualization-Histogram

https://www.reddit.com/r/MachineLearning/comments/6g5tg1/r_selfnormalizing_neural_networks_improved_elu/
https://news.ycombinator.com/item?id=14527686

[cnuernber]

Great find, thanks!

[gigasquid]

Since I have context doing swish. I can take a look at tackling this next.

[gigasquid]

Here is the paper https://arxiv.org/pdf/1706.02515.pdf

[cnuernber]

You may need a couple more tensor functions to make that happen, mainly the 'moving' class of functions:

https://www.mathworks.com/help/matlab/ref/movmean.html
https://www.mathworks.com/help/matlab/ref/movsum.html

I think this may be on the harder (and more interesting) scale of things so please speak up if the toolset isn't there yet so we can address that.

[gigasquid]

@cnuernber On the face of it, it seems like a simple activation function and derivative. But I could easily be missing something.

From the paper the activation function is

and the derivative is

where alpha = 1.6732632423543772848170429916717
and lambda = 1.0507009873554804934193349852946

Here is the implementation in DeepLearning4J
https://github.com/deeplearning4j/libnd4j/pull/401/files

Where do you see the moving mean and sum come in?

[gigasquid]

@cnuernber I see now, it's a the selu dropout. I'll definitely have to dig into that a bit more 😄

[cnuernber]

That wasn't it, I misread the paper and thought the weights would have a different update step which is why I thought this would be really hard.

[gigasquid]

Ah ok. I'll get a PR together for the selu activation first. I'll then take a look at adding selu dropout in another PR, since I think that is a bit more involved.