Ask Your Question

Can I use @parallel for class/instance methods?

asked 2011-06-09 06:48:34 +0200

niles gravatar image

updated 2011-06-09 11:47:58 +0200

I would like to be able to evaluate some methods in parallel. For example:

class PTest(SageObject):
    def meth0(self,n):
        return n
    def meth1(self,n):
        "long and complicated method"
        return prime_pi(n)

sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
sage: r = T.meth1(L)
TypeError: an integer is required

Note that @parallel does work fine if I define meth1 outside of the class, and introspection T.meth1? indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?

edit retag flag offensive close merge delete

4 Answers

Sort by ยป oldest newest most voted

answered 2011-06-11 20:16:26 +0200

Mike Hansen gravatar image

I've posted an initial patch at ticket 11461 which fixes the problem that Niles reported. The key is to use the descriptor protocol ( __get__ ) to "wrap" the correct function depending on how the function is accessed. This will also work with classmethods and staticmethods provided that the parallel decorator is applied after either of those two. For example,

class Foo(object):
    def bar(cls, n):
        return str(cls)*n
edit flag offensive delete link more

answered 2011-06-09 10:16:30 +0200

kcrisman gravatar image

updated 2011-06-09 10:19:11 +0200

This isn't quite an answer, but hopefully will help.

Niles, look at the traceback more carefully.

    114                 return self.p_iter(f, (normalize_input(a) for a in args[0]))
    115             else:
--> 116                 return f(*args, **kwds)
    117         return g

Clearly it's gotten the parallel, as you say. But the whole thing is

# Construct the wrapper parallel version of the function we're wrapping.
# We may rework this so g is a class instance, which has the plus that
# we can query g for how it works, etc.
def g(*args, **kwds):
    if len(args) > 0 and isinstance(args[0], (list, types.GeneratorType)):
        return self.p_iter(f, (normalize_input(a) for a in args[0]))
        return f(*args, **kwds)
return g

If I insert a print statement (print args) before the if/else as well as print statements about which branch I take, I get

sage: T.meth1(L[0])
(<class '__main__.PTest'>, 1000)
second branch
sage: r = T.meth1(L)
(<class '__main__.PTest'>, [1000, 2000, 3000, 4000, 5000])
second branch

Does this help? It's never even reaching the point of the iterator, because the first item in args is not iterable, but the class itself. I don't know how to fix this, except by adding a case in the if statement, but I have no idea whether that would break anything else.

edit flag offensive delete link more

answered 2011-06-09 11:46:26 +0200

niles gravatar image

updated 2011-06-10 14:27:20 +0200

Thanks Karl! That did help a lot, and I think I have a workaround. First, to be clear, I think the basic answer to my question is:

"No; a method always implicitly prepends its arguments with self (the class instance it is bound to), and @parallel only works properly if the first argument is a list or tuple. Therefore the two are fundamentally incompatible."

Of course maybe someone clever can think of a way to improve @parallel. I spent some time on this, but I couldn't come up with a reliable way to test whether the first argument was the class instance to which the method was bound.

UPDATE: This is now ticket 11461 :)

But I did think of a reasonable workaround: Write a method for the class which will produce the parallelizable function. This function can still take self as its first argument and thereby access any other attributes in the class; the only difference will be that the user will have to explicitly give the class instance as the first argument. Here's a demo:

class PTest(SageObject):
    attr = 'red'
    def meth0(self,n):
        return n
    def generate_parallel_method(self):
        def meth1(self,n):
            "long and complicated class method"
            return [prime_pi(n),self.attr]
        return meth1

And here's how it works:

sage: N = PTest()
sage: L = [(N,x) for x in [1000,2000,3000,4000,5000]]
sage: pmeth = N.generate_parallel_method()

sage: pmeth(100)
Traceback (most recent call last)
TypeError: meth1() takes exactly 2 arguments (1 given)
sage: pmeth(N,100)
[25, 'red']

sage: r = pmeth(L)
sage: for t in r:
....:     print "%s --> %s"%(t[0],t[1])
((<class '__main__.PTest'>, 2000), {}) --> [303, 'red']
((<class '__main__.PTest'>, 1000), {}) --> [168, 'red']
((<class '__main__.PTest'>, 3000), {}) --> [430, 'red']
((<class '__main__.PTest'>, 4000), {}) --> [550, 'red']
((<class '__main__.PTest'>, 5000), {}) --> [669, 'red']
edit flag offensive delete link more


Good work. I still think that someone clever could do it, as you say. For instance, one could check whether the first thing is a class instance and the rest is a list of the appropriate type, in a try/except block ... Maybe you should ask this on sage-devel. It would certainly be a natural improvement.

kcrisman gravatar imagekcrisman ( 2011-06-09 12:11:20 +0200 )edit

answered 2011-06-10 02:59:32 +0200

Simon King gravatar image

Here are some ideas for making the @parallel decorator usable on methods.

I did not check the code. But usually, a decorator takes as its argument a function f. Now, it is easy to check whether f is a plain function or a method: Use ismethod() or ismethoddescriptor() from the inspect module.

On top of that, one may even think of making @parallel usable on wrapped methods, such as

def my_method(self, L,**args):

In that case, the function put into the @parallel decorator would be an instance of a CachedMethodCaller. Those things could be tested with the function isclassinstance() from sage.misc.sageinspect. Perhaps that particular example does not make sense, but it may still be a case to be taken into account.

edit flag offensive delete link more


I'm not sure this will work -- I tried a similar approach checking for the `.__self__` attribute and using it to determine if the first argument was equal to the class instance. But the code failed, saying that `f` did not have a `.__self__` attribute. I know that methods do have this attribute, so I think what's being passed to Parallel is some kind of unbound version of the method . . . and now I'm really lost. In any case, I guess we should take further discussion to the new ticket :)

niles gravatar imageniles ( 2011-06-10 14:31:31 +0200 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools


Asked: 2011-06-09 06:48:34 +0200

Seen: 737 times

Last updated: Jun 11 '11