Leon Bottou
2014-08-11 01:20:16 UTC
The attached cython program uses an extension class to represent a unit of
work. The with parallel block temporarily gets the gil to allocate the object,
then release the gil and performs the task with a for i in prange(...)
statement. My expectation was to have w recognized as a thread private
variable.
with nogil, parallel():
with gil:
w = Worker(n) # should be thread private
with nogil:
for i in prange(0,m):
r += w.run() # should be reduction
w = None # is this needed?
Cythonize (0.20.2) works without error but produces an incorrect C file.
hello.c: In function â__pyx_pf_5hello_runâ:
hello.c:2193:42: error: expected identifier before â)â token
hello.c: At top level:
The erroneous line is:
#pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5,
__pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2)
private(__pyx_filename, __pyx_lineno, __pyx_clineno)
shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value,
__pyx_parallel_exc_tb)
where you can see that the first private() clause has no argument. The
variable __pyx_v_w is not declared as private either as I would expect.
I believe that the problem comes from line 7720 in Cython/Compiler/Node.py
if self.privates:
privates = [e.cname for e in self.privates
if not e.type.is_pyobject]
code.put('private(%s)' % ', '.join(privates))
And I further believe that the clause "if not e.type.is_pyobject" has been
added because nothing would decrements the reference count of the thread
private worker object when leaving the parallel block.
My quick fix would be to remove this clause and make sure that my program
contains the line "w = None" before leaving the thread. But I realize that
this is not sufficient for you.
Note that the temporary python objects generated by the call to the Worker
construction are correctly recognized as thread private and their reference
count is correctly decremented when they are no longer needed. The problem
here is the clash between the python scoping rules and the semantics of thread
private variables. This is one of these cases where I would have liked to be
able to write
with nogil, parallel():
with gil:
cdef w = Worker(n) # block-scoped cdef
with nogil:
for i in prange(0,m):
r += w.run()
with an understanding that the scope of the cdef variable is limited to the
block where the cdef appears. But when you try this, cython tells you that
cdefs are not legal there.
work. The with parallel block temporarily gets the gil to allocate the object,
then release the gil and performs the task with a for i in prange(...)
statement. My expectation was to have w recognized as a thread private
variable.
with nogil, parallel():
with gil:
w = Worker(n) # should be thread private
with nogil:
for i in prange(0,m):
r += w.run() # should be reduction
w = None # is this needed?
Cythonize (0.20.2) works without error but produces an incorrect C file.
hello.c: In function â__pyx_pf_5hello_runâ:
hello.c:2193:42: error: expected identifier before â)â token
hello.c: At top level:
The erroneous line is:
#pragma omp parallel private() reduction(+:__pyx_v_r) private(__pyx_t_5,
__pyx_t_4, __pyx_t_3) firstprivate(__pyx_t_1, __pyx_t_2)
private(__pyx_filename, __pyx_lineno, __pyx_clineno)
shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value,
__pyx_parallel_exc_tb)
where you can see that the first private() clause has no argument. The
variable __pyx_v_w is not declared as private either as I would expect.
I believe that the problem comes from line 7720 in Cython/Compiler/Node.py
if self.privates:
privates = [e.cname for e in self.privates
if not e.type.is_pyobject]
code.put('private(%s)' % ', '.join(privates))
And I further believe that the clause "if not e.type.is_pyobject" has been
added because nothing would decrements the reference count of the thread
private worker object when leaving the parallel block.
My quick fix would be to remove this clause and make sure that my program
contains the line "w = None" before leaving the thread. But I realize that
this is not sufficient for you.
Note that the temporary python objects generated by the call to the Worker
construction are correctly recognized as thread private and their reference
count is correctly decremented when they are no longer needed. The problem
here is the clash between the python scoping rules and the semantics of thread
private variables. This is one of these cases where I would have liked to be
able to write
with nogil, parallel():
with gil:
cdef w = Worker(n) # block-scoped cdef
with nogil:
for i in prange(0,m):
r += w.run()
with an understanding that the scope of the cdef variable is limited to the
block where the cdef appears. But when you try this, cython tells you that
cdefs are not legal there.