# Can I use @parallel for class/instance methods?

I would like to be able to evaluate some methods in parallel. For example:

class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)

sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required


Note that @parallel does work fine if I define meth1 outside of the class, and introspection T.meth1? indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?

edit retag close merge delete

Sort by » oldest newest most voted

I've posted an initial patch at ticket 11461 which fixes the problem that Niles reported. The key is to use the descriptor protocol ( __get__ ) to "wrap" the correct function depending on how the function is accessed. This will also work with classmethods and staticmethods provided that the parallel decorator is applied after either of those two. For example,

class Foo(object):
@parallel(2)
@classmethod
def bar(cls, n):
return str(cls)*n

more

This isn't quite an answer, but hopefully will help.

Niles, look at the traceback more carefully.

    114                 return self.p_iter(f, (normalize_input(a) for a in args[0]))
115             else:
--> 116                 return f(*args, **kwds)
117         return g
118


Clearly it's gotten the parallel, as you say. But the whole thing is

# Construct the wrapper parallel version of the function we're wrapping.
# We may rework this so g is a class instance, which has the plus that
# we can query g for how it works, etc.
def g(*args, **kwds):
if len(args) > 0 and isinstance(args[0], (list, types.GeneratorType)):
return self.p_iter(f, (normalize_input(a) for a in args[0]))
else:
return f(*args, **kwds)
return g


If I insert a print statement (print args) before the if/else as well as print statements about which branch I take, I get

sage: T.meth1(L[0])
(<class '__main__.PTest'>, 1000)
second branch
168
sage: r = T.meth1(L)
(<class '__main__.PTest'>, [1000, 2000, 3000, 4000, 5000])
second branch
---------------------------------------------------------
TypeError


Does this help? It's never even reaching the point of the iterator, because the first item in args is not iterable, but the class itself. I don't know how to fix this, except by adding a case in the if statement, but I have no idea whether that would break anything else.

more

Thanks Karl! That did help a lot, and I think I have a workaround. First, to be clear, I think the basic answer to my question is:

"No; a method always implicitly prepends its arguments with self (the class instance it is bound to), and @parallel only works properly if the first argument is a list or tuple. Therefore the two are fundamentally incompatible."

Of course maybe someone clever can think of a way to improve @parallel. I spent some time on this, but I couldn't come up with a reliable way to test whether the first argument was the class instance to which the method was bound.

UPDATE: This is now ticket 11461 :)

But I did think of a reasonable workaround: Write a method for the class which will produce the parallelizable function. This function can still take self as its first argument and thereby access any other attributes in the class; the only difference will be that the user will have to explicitly give the class instance as the first argument. Here's a demo:

class PTest(SageObject):
attr = 'red'
def meth0(self,n):
return n
def generate_parallel_method(self):
@parallel
def meth1(self,n):
"long and complicated class method"
sleep(2)
return [prime_pi(n),self.attr]
return meth1


And here's how it works:

sage: N = PTest()
sage: L = [(N,x) for x in [1000,2000,3000,4000,5000]]
sage: pmeth = N.generate_parallel_method()

sage: pmeth(100)
Traceback (most recent call last)
...
TypeError: meth1() takes exactly 2 arguments (1 given)
sage: pmeth(N,100)
[25, 'red']

sage: r = pmeth(L)
sage: for t in r:
....:     print "%s --> %s"%(t[0],t[1])
....:
((<class '__main__.PTest'>, 2000), {}) --> [303, 'red']
((<class '__main__.PTest'>, 1000), {}) --> [168, 'red']
((<class '__main__.PTest'>, 3000), {}) --> [430, 'red']
((<class '__main__.PTest'>, 4000), {}) --> [550, 'red']
((<class '__main__.PTest'>, 5000), {}) --> [669, 'red']

more

Good work. I still think that someone clever could do it, as you say. For instance, one could check whether the first thing is a class instance and the rest is a list of the appropriate type, in a try/except block ... Maybe you should ask this on sage-devel. It would certainly be a natural improvement.

( 2011-06-09 12:11:20 +0200 )edit

Here are some ideas for making the @parallel decorator usable on methods.

I did not check the code. But usually, a decorator takes as its argument a function f. Now, it is easy to check whether f is a plain function or a method: Use ismethod() or ismethoddescriptor() from the inspect module.

On top of that, one may even think of making @parallel usable on wrapped methods, such as

@parallel
@cached_method
def my_method(self, L,**args):


In that case, the function put into the @parallel decorator would be an instance of a CachedMethodCaller. Those things could be tested with the function isclassinstance() from sage.misc.sageinspect. Perhaps that particular example does not make sense, but it may still be a case to be taken into account.

more

I'm not sure this will work -- I tried a similar approach checking for the .__self__ attribute and using it to determine if the first argument was equal to the class instance. But the code failed, saying that f did not have a .__self__ attribute. I know that methods do have this attribute, so I think what's being passed to Parallel is some kind of unbound version of the method . . . and now I'm really lost. In any case, I guess we should take further discussion to the new ticket :) http://trac.sagemath.org/sage_trac/ticket/11461

( 2011-06-10 14:31:31 +0200 )edit