sage: var('a y')
(a, y)
sage: F = function('F',x)
sage: G = F.derivative(x)
sage: G
D[0](F)(x)
sage: exp = x^2*G + a + x - x^2*y
sage: p(h) = h^2-h+1
sage: p(exp)
x^2*y - x^2*D[0](F)(x) + (x^2*y - x^2*D[0](F)(x) - a - x)^2 - a - x + 1
sage: p(exp).expand()
x^4*y^2 - 2*x^4*y*D[0](F)(x) + x^4*D[0](F)(x)^2 - 2*a*x^2*y + 2*a*x^2*D[0](F)(x) - 2*x^3*y + 2*x^3*D[0](F)(x) + x^2*y - x^2*D[0](F)(x) + a^2 + 2*a*x + x^2 - a - x + 1
Downside: these aren't really differential operators or D-module thingies, just derivatives of a dummy function. There is this functionality, but:
sage: from sage.tensor.differential_form_element import d
sage: exp1 = x^2*d + a + x - x^2*y
TypeError: unsupported operand parent(s) for '*': 'Symbolic Ring' and '<type 'function'>'
so I don't think it's immediately usable. See also this Trac ticket about implementing more general pseudo-differential operators.
I'm trying to understand exactly what you mean. Wouldn't your answer still (possibly) involve d/dx? For instance, wouldn't (d/dx + x^5)^2 = d^2/(d^2 x) + 5x^4 + x^5 d/dx + x^10? Or am I misunderstanding what you mean? Also, I don't understand the purpose of the "1" you put after p(x^2d/dx + a + x - x^2y). (Since you again put a 1 in a similar position in your comment to kcrisman's answer below, I assume it's not a typo.)
Sorry I wasn't clear. Try a simpler example. p is a polynomial in 1 variable. p(d/dx + x^2) is a differential operator. p(d/dx + x^2) 1 is that operator acting on the constant function. The result is a polynomial in x. I'd like to calculate that polynomial in Sage. I have a workaround, using Fourier transforms, that Sage can do via Macsyma, but I suspect an algebraic method would scale better.
Here's an example. Let p(y)=y^2. Then p(d/dx + x^2) = (d/dx)^2 + 2 x^2 d/dx + 2 x + x^4. Acting on 1, this operator gives 2 x + x^4.