[Cython] Compiler crash in RemoveUnreachableCode

Discussion:

Andriy Kornatskyy

2014-01-19 20:00:14 UTC

The cython compiler crash report below. Steps to reproduce:

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install wheezy.http

The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.

Thanks.

Andriy Kornatskyy

File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
Cython.Compiler.Errors.CompilerCrash:
Error compiling Cython file:
------------------------------------------------------------
...

"""
"""

__version__ = '0.1.129'
^
------------------------------------------------------------

src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode

ModuleNode.body = StatListNode(__init__.py:5:14)

Compiler crash traceback from this point on:
File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode
if not self.current_directives['remove_unreachable']:
TypeError: 'NoneType' object has no attribute '__getitem__'

Stefan Behnel

2014-01-20 20:19:07 UTC

Permalink

Hi,

thanks for the report, I can reproduce this.

Post by Andriy Kornatskyy
1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install wheezy.http
The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies.

Yes, that's rather surprising.

Post by Andriy Kornatskyy
File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437)
File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655)
------------------------------------------------------------
...
"""
"""
__version__ = '0.1.129'
^
------------------------------------------------------------
src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode
ModuleNode.body = StatListNode(__init__.py:5:14)
File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275)
File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode
TypeError: 'NoneType' object has no attribute '__getitem__'

I only get this when Cython is compiled. When I run it from the source
tree, it works.

Debugging into Visitor.py's CythonTransform shows that this check fails:

if isinstance(node, ModuleNode.ModuleNode):
self.current_directives = node.directives

"node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode.

My guess is that there's something wrong with the import mechanism that
leads to the module being imported twice. Relative versus absolute imports,
most likely. I also see a couple of weird modules in sys.modules, e.g.
"Cython.Compiler.re", so there are some more bits going generally wrong here.

I also tried it in Py3, but that even gives me a hard crash. Not really better.

I might be able to take a deeper look this weekend, but wouldn't mind if
others beat me to it.

Stefan

Andriy Kornatskyy

2014-01-20 21:16:07 UTC

Permalink

Stefan,

Thank you for the comments back.

May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.

Here is another bug (details below):

1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install lxml wheezy.core

It seems to have an issue while trying to install 2 or more libs at once.

Andriy Kornatskyy

...
Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg
Processing dependencies for lxml
Finished processing dependencies for lxml
Searching for wheezy.core
Reading https://pypi.python.org/simple/wheezy.core/
Best match: wheezy.core 0.1.129
Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d
Processing wheezy.core-0.1.129.tar.gz
Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg
Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1
Traceback (most recent call last):
...
File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation
self.generate_c_code(env, options, result)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code
rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback)
File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)

File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__
stream = StringIO()
TypeError: 'NoneType' object is not callable

Post by Stefan Behnel
Hi,
thanks for the report, I can reproduce this.

Yes, that's rather surprising.

I only get this when Cython is compiled. When I run it from the source
tree, it works.
self.current_directives = node.directives
"node" actually *is* an instance of ModuleNode, just not of *that* ModuleNode.
My guess is that there's something wrong with the import mechanism that
leads to the module being imported twice. Relative versus absolute imports,
most likely. I also see a couple of weird modules in sys.modules, e.g.
"Cython.Compiler.re", so there are some more bits going generally wrong here.
I also tried it in Py3, but that even gives me a hard crash. Not really better.
I might be able to take a deeper look this weekend, but wouldn't mind if
others beat me to it.
Stefan
_______________________________________________
cython-devel mailing list
https://mail.python.org/mailman/listinfo/cython-devel

Stefan Behnel

2014-01-21 08:16:46 UTC

Permalink

Post by Andriy Kornatskyy
May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.

No, I think it's a problem with importing in Cython compiled modules. ISTM
that some extension modules got "reloaded", i.e. their module dict got
cleared and they got reinitialised, thus recreating all global objects that
other modules had already imported (and still keep a reference to).

Post by Andriy Kornatskyy
1. virtualenv env
2. env/bin/easy_install cython
3. env/bin/easy_install lxml wheezy.core
It seems to have an issue while trying to install 2 or more libs at once.
...
Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg
Processing dependencies for lxml
Finished processing dependencies for lxml
Searching for wheezy.core
Reading https://pypi.python.org/simple/wheezy.core/
Best match: wheezy.core 0.1.129
Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d
Processing wheezy.core-0.1.129.tar.gz
Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg
Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1
...
File “env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation
self.generate_c_code(env, options, result)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code
rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback)
File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697)
File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__
stream = StringIO()
TypeError: 'NoneType' object is not callable

Looks like the same thing.

Stefan

Andriy Kornatskyy

2014-02-22 09:35:58 UTC

Permalink

Stefan,

Any update to the issue reported in this message thread?

If you can provide a link to the issue in cython bug tracker, so I can see progress, I would greatly appreciate that.

Thanks.

Andriy Kornatskyy

Post by Stefan Behnel

Post by Andriy Kornatskyy
May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools.

Looks like the same thing.
Stefan
_______________________________________________
cython-devel mailing list
https://mail.python.org/mailman/listinfo/cython-devel

Stefan Behnel

2014-02-22 19:40:47 UTC

Permalink

Post by Andriy Kornatskyy

Post by Stefan Behnel
I think it's a problem with importing in Cython compiled modules. ISTM
that some extension modules got "reloaded", i.e. their module dict got
cleared and they got reinitialised, thus recreating all global objects that
other modules had already imported (and still keep a reference to).

Looks like the same thing.

Any update to the issue reported in this message thread?

No, not really. The current setup is just way too vast to debug. I don't
even know how easy_install runs the installation of a single package, nor
how it's possible that two packages interfere in such a way. I'd expect it
to run subprocesses, so why any interference at all? It might be that the
main process loads the setup.py scripts in some way to read the meta data,
thus executes their imports, and then tries to unload the modules before it
loads the next setup.py script. That would be rather bad for any extension
modules imported by both. So bad that's I'd consider it a bug in setuptools.

Let's assume that reloading really is what's happening here. Given that
this basically doesn't work for extension modules, definitely not in Py2.x
and barely in Py3.x, we might be able to apply a barrier to the module init
function that prevents it from running a second time, and just return the
already created module somehow. But that's blank guessing and I really
can't tell for sure. It would help if someone could invest some time into
properly analysing this further.

Stefan

Stefan Behnel

2014-09-06 20:18:27 UTC

Permalink

I tried "pip install wheezy.http" instead of "easy_install wheezy.http" and
it works as expected. This sounds like a good work-around to me.

Stefan

Andriy Kornatskyy

2014-09-07 07:17:18 UTC

Permalink

Post by Stefan Behnel
I tried "pip install wheezy.http" instead of "easy_install wheezy.http" and
it works as expected. This sounds like a good work-around to me.

Yes, confirmed. Thanks.

Andriy