ASKSAGE: Sage Q&A Forum - RSS feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Sat, 11 Jun 2011 20:16:26 +0200Can I use @parallel for class/instance methods?https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/I would like to be able to evaluate some methods in parallel. For example:
class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)
sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required
Note that `@parallel` does work fine if I define `meth1` outside of the class, and introspection `T.meth1?` indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?
Thu, 09 Jun 2011 06:48:34 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/Answer by Simon King for <p>I would like to be able to evaluate some methods in parallel. For example:</p>
<pre><code>class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)
sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required
</code></pre>
<p>Note that <code>@parallel</code> does work fine if I define <code>meth1</code> outside of the class, and introspection <code>T.meth1?</code> indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?</p>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12429#post-id-12429Here are some ideas for making the `@parallel` decorator usable on methods.
I did not check the code. But usually, a decorator takes as its argument a function f. Now, it is easy to check whether f is a plain function or a method: Use `ismethod()` or `ismethoddescriptor()` from the `inspect` module.
On top of that, one may even think of making `@parallel` usable on *wrapped* methods, such as
@parallel
@cached_method
def my_method(self, L,**args):
In that case, the function put into the @parallel decorator would be an instance of a CachedMethodCaller. Those things could be tested with the function `isclassinstance()` from `sage.misc.sageinspect`. Perhaps that particular example does not make sense, but it may still be a case to be taken into account.Fri, 10 Jun 2011 02:59:32 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12429#post-id-12429Comment by niles for <p>Here are some ideas for making the <code>@parallel</code> decorator usable on methods.</p>
<p>I did not check the code. But usually, a decorator takes as its argument a function f. Now, it is easy to check whether f is a plain function or a method: Use <code>ismethod()</code> or <code>ismethoddescriptor()</code> from the <code>inspect</code> module.</p>
<p>On top of that, one may even think of making <code>@parallel</code> usable on <em>wrapped</em> methods, such as</p>
<pre><code>@parallel
@cached_method
def my_method(self, L,**args):
</code></pre>
<p>In that case, the function put into the @parallel decorator would be an instance of a CachedMethodCaller. Those things could be tested with the function <code>isclassinstance()</code> from <code>sage.misc.sageinspect</code>. Perhaps that particular example does not make sense, but it may still be a case to be taken into account.</p>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?comment=21596#post-id-21596I'm not sure this will work -- I tried a similar approach checking for the `.__self__` attribute and using it to determine if the first argument was equal to the class instance. But the code failed, saying that `f` did not have a `.__self__` attribute. I know that methods do have this attribute, so I think what's being passed to Parallel is some kind of unbound version of the method . . . and now I'm really lost. In any case, I guess we should take further discussion to the new ticket :) http://trac.sagemath.org/sage_trac/ticket/11461Fri, 10 Jun 2011 14:31:31 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?comment=21596#post-id-21596Answer by kcrisman for <p>I would like to be able to evaluate some methods in parallel. For example:</p>
<pre><code>class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)
sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required
</code></pre>
<p>Note that <code>@parallel</code> does work fine if I define <code>meth1</code> outside of the class, and introspection <code>T.meth1?</code> indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?</p>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12426#post-id-12426This isn't quite an answer, but hopefully will help.
Niles, look at the traceback more carefully.
114 return self.p_iter(f, (normalize_input(a) for a in args[0]))
115 else:
--> 116 return f(*args, **kwds)
117 return g
118
Clearly it's gotten the parallel, as you say. But the whole thing is
# Construct the wrapper parallel version of the function we're wrapping.
# We may rework this so g is a class instance, which has the plus that
# we can query g for how it works, etc.
def g(*args, **kwds):
if len(args) > 0 and isinstance(args[0], (list, types.GeneratorType)):
return self.p_iter(f, (normalize_input(a) for a in args[0]))
else:
return f(*args, **kwds)
return g
If I insert a print statement (print args) before the if/else as well as print statements about which branch I take, I get
sage: T.meth1(L[0])
(<class '__main__.PTest'>, 1000)
second branch
168
sage: r = T.meth1(L)
(<class '__main__.PTest'>, [1000, 2000, 3000, 4000, 5000])
second branch
---------------------------------------------------------
TypeError
Does this help? It's never even reaching the point of the iterator, because the first item in args is not iterable, but the class itself. I don't know how to fix this, except by adding a case in the if statement, but I have no idea whether that would break anything else.Thu, 09 Jun 2011 10:16:30 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12426#post-id-12426Answer by niles for <p>I would like to be able to evaluate some methods in parallel. For example:</p>
<pre><code>class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)
sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required
</code></pre>
<p>Note that <code>@parallel</code> does work fine if I define <code>meth1</code> outside of the class, and introspection <code>T.meth1?</code> indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?</p>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12427#post-id-12427Thanks Karl! That did help a lot, and I think I have a workaround. First, to be clear, I think the basic answer to my question is:
*"No; a method always implicitly prepends its arguments with `self` (the class instance it is bound to), and `@parallel` only works properly if the first argument is a list or tuple. Therefore the two are fundamentally incompatible."*
Of course maybe someone clever can think of a way to improve `@parallel`. I spent some time on this, but I couldn't come up with a reliable way to test whether the first argument was the class instance to which the method was bound.
UPDATE: This is now ticket 11461 :)
<br>
But I did think of a reasonable workaround: Write a method for the class which will produce the parallelizable function. This function can still take `self` as its first argument and thereby access any other attributes in the class; the only difference will be that the user will have to explicitly give the class instance as the first argument. Here's a demo:
class PTest(SageObject):
attr = 'red'
def meth0(self,n):
return n
def generate_parallel_method(self):
@parallel
def meth1(self,n):
"long and complicated class method"
sleep(2)
return [prime_pi(n),self.attr]
return meth1
And here's how it works:
sage: N = PTest()
sage: L = [(N,x) for x in [1000,2000,3000,4000,5000]]
sage: pmeth = N.generate_parallel_method()
sage: pmeth(100)
Traceback (most recent call last)
...
TypeError: meth1() takes exactly 2 arguments (1 given)
sage: pmeth(N,100)
[25, 'red']
sage: r = pmeth(L)
sage: for t in r:
....: print "%s --> %s"%(t[0],t[1])
....:
((<class '__main__.PTest'>, 2000), {}) --> [303, 'red']
((<class '__main__.PTest'>, 1000), {}) --> [168, 'red']
((<class '__main__.PTest'>, 3000), {}) --> [430, 'red']
((<class '__main__.PTest'>, 4000), {}) --> [550, 'red']
((<class '__main__.PTest'>, 5000), {}) --> [669, 'red']
Thu, 09 Jun 2011 11:46:26 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12427#post-id-12427Comment by kcrisman for <p>Thanks Karl! That did help a lot, and I think I have a workaround. First, to be clear, I think the basic answer to my question is: </p>
<p><em>"No; a method always implicitly prepends its arguments with <code>self</code> (the class instance it is bound to), and <code>@parallel</code> only works properly if the first argument is a list or tuple. Therefore the two are fundamentally incompatible."</em></p>
<p>Of course maybe someone clever can think of a way to improve <code>@parallel</code>. I spent some time on this, but I couldn't come up with a reliable way to test whether the first argument was the class instance to which the method was bound.</p>
<p>UPDATE: This is now ticket 11461 :) </p>
<p><br/></p>
<p>But I did think of a reasonable workaround: Write a method for the class which will produce the parallelizable function. This function can still take <code>self</code> as its first argument and thereby access any other attributes in the class; the only difference will be that the user will have to explicitly give the class instance as the first argument. Here's a demo:</p>
<pre><code>class PTest(SageObject):
attr = 'red'
def meth0(self,n):
return n
def generate_parallel_method(self):
@parallel
def meth1(self,n):
"long and complicated class method"
sleep(2)
return [prime_pi(n),self.attr]
return meth1
</code></pre>
<p>And here's how it works:</p>
<pre><code>sage: N = PTest()
sage: L = [(N,x) for x in [1000,2000,3000,4000,5000]]
sage: pmeth = N.generate_parallel_method()
sage: pmeth(100)
Traceback (most recent call last)
...
TypeError: meth1() takes exactly 2 arguments (1 given)
sage: pmeth(N,100)
[25, 'red']
sage: r = pmeth(L)
sage: for t in r:
....: print "%s --> %s"%(t[0],t[1])
....:
((<class '__main__.PTest'>, 2000), {}) --> [303, 'red']
((<class '__main__.PTest'>, 1000), {}) --> [168, 'red']
((<class '__main__.PTest'>, 3000), {}) --> [430, 'red']
((<class '__main__.PTest'>, 4000), {}) --> [550, 'red']
((<class '__main__.PTest'>, 5000), {}) --> [669, 'red']
</code></pre>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?comment=21605#post-id-21605Good work. I still think that someone clever could do it, as you say. For instance, one could check whether the first thing is a class instance and the rest is a list of the appropriate type, in a try/except block ... Maybe you should ask this on sage-devel. It would certainly be a natural improvement.Thu, 09 Jun 2011 12:11:20 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?comment=21605#post-id-21605Answer by Mike Hansen for <p>I would like to be able to evaluate some methods in parallel. For example:</p>
<pre><code>class PTest(SageObject):
def meth0(self,n):
return n
@parallel
def meth1(self,n):
"long and complicated method"
sleep(2)
return prime_pi(n)
sage: L = [1000,2000,3000,4000,5000]
sage: T = PTest()
sage: T.meth1(L[0])
168
sage: r = T.meth1(L)
...
TypeError: an integer is required
</code></pre>
<p>Note that <code>@parallel</code> does work fine if I define <code>meth1</code> outside of the class, and introspection <code>T.meth1?</code> indicates that the decorator has been applied to it. So I don't understand what else I should do; any ideas?</p>
https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12431#post-id-12431I've posted an initial patch at ticket 11461 which fixes the problem that Niles reported. The key is to use the descriptor protocol ( `__get__` ) to "wrap" the correct function depending on how the function is accessed. This will also work with classmethods and staticmethods provided that the parallel decorator is applied after either of those two. For example,
class Foo(object):
@parallel(2)
@classmethod
def bar(cls, n):
return str(cls)*nSat, 11 Jun 2011 20:16:26 +0200https://ask.sagemath.org/question/8154/can-i-use-parallel-for-classinstance-methods/?answer=12431#post-id-12431