# Elementwise functions for deep learning This post is a wiki. Anyone with karma >750 is welcome to improve it.

Let's say I want to define a softmax function over a vector input.

In Python/Numpy that would look something like this

X = np.array([x1, x2, x3])
numerators = np.exp(X)
denominator = np.sum(numerators)
softmax_probs = numerators / denominators # [e^x1 / sum(e^x1, e^x2, e^x3), e^x2 / sum(e^x1, e^x2, e^x3),...


edit retag close merge delete

Sort by » oldest newest most voted

def softmax_probs(L):
R=list(map(exp, L))
S=sum(R)
return map(lambda u:u/S, R)
L=list(var("a, b, c"))
list(softmax_probs(L))
[e^a/(e^a + e^b + e^c), e^b/(e^a + e^b + e^c), e^c/(e^a + e^b + e^c)]


?

The point of the local variables is to avoid recomputing the same quantities more than once :

sage: %timeit list(map(lambda u:exp(u)/sum(map(exp, L)), L))
70.4 µs ± 942 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
sage: %timeit list(softmax_probs(L))
31.2 µs ± 699 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

more