how to define symbolic function on functions?

edit

The reason for the results is that callable symbolic expressions are defined by interpreting the input variables as symbolic variables, and calling will amount to substitution of values into those variables in the expression on the right-hand side. I agree this limitation is not obvious and often confusing for beginners.

Indeed, when f and g are symbolic variables, and x is a symbolic variable, then f(x) and g(x) will both evaluate to x, and integral(f(x)*g(x), (x,0,1)) hence evaluates to integral(x^2, (x,0,1)) which is indeed 1/3.

As explained in the other answer, what you want is a Python function instead.

Your expressions are ill defined. A python function does the job

def inner_product(f,g): return integral(f*g, (x,-pi,pi))

Thus

inner_product(sin(3*x),sin(3*x))

returns $\pi$.

See here for symbolic computation.