[Cython] Debugging a segfault in a cython generated module

Discussion:

Brian Granger

2008-04-19 03:12:18 UTC

Python (though if it's a cdef or cpdef method it uses vtables rather
than dictionaries). The only methods that do not get overridden are
__cinit__ and __dealloc__, and this is because we need to guarantee
that these methods get called, no matter what subclasses do.
__cinit__ is guaranteed to be called *exactly* once for an object,
and if successful __dealloc__ will be called *exactly* once on
deallocation. This makes these two methods ideal places to do memory
management associated with an object.
- Robert

I am realizing that this is also a problem. I was creating a memory
leak by having both the subclass and superclass do allocations in
their own __cinit__. The trick you propose:

cdef class Foo:

cdef __cinit__(self):
self._actual_cinit(self)
cdef _actual_cinit(self):
foo_ptr = doTheAllocation()

cdef class Bar(Foo):

cdef _actual_cinit(self):
bar_ptr = doTheAllocation()

When I try this pattern Foo's _actual_cinit *always gets called.* I
looked at the generated C code and it look like this:

__pyx_1 = ((struct __pyx_vtabstruct_6txbase_Foo *)((struct
__pyx_obj_6txbase_Foo *)__pyx_v_self)->__pyx_vtab)->_actualCinit()

When I made _actualCinit(), I got an attribute error (which makes
sense as the Python side of the class has not been built yet.)

Why is the vtable not finding the right _actualCinit implementation?
Is this a bug, or is this to be expected because the class has not
been fully built.

Also, why can't I just do the allocations in the __init__ method?
That has a much more flexible and explicit approach when dealing with
subclasses.

Thanks

Brian

Thanks
Brian

Thanks
Brian
On Fri, Apr 18, 2008 at 3:00 PM, Michael.Abshoff

Hi Brian,

Yes, at least to see if it can spot the problem.
To get started: Build your own python. Make sure to compile it
with
"--without-pymalloc" and also set the compile flags to "-O0 -
g". Build
valgrind 3.3.0 [or install it from binary packages] and run
your code
under it. If it segfaults valgrind [it happens to me somewhat
regularly,
but then usually something evil is happening in libSingular ;]
go and
check out the 3.4svn version.

Thanks, I will try this out. I have never used valgrind, so
this is
very helpful.

Ok, to run it do
valgrind --tool=memcheck --trace-children=yes --leak-
resolution=high
--log-file=foo ./path/to/python
Then do your thing, quit python. At that point there should be
foo.$PID
in the cwd. For a C++ specific example where we did dumb things at
dealloc time from Sage look at
http://trac.sagemath.org/sage_trac/ticket/1573
Also: searching for delete[] in Sage's trac will turn up a
number of
other examples. If you get stuck feel free to post the example
here and
once I take a look at the code I may be able to help you out.

Let me know if you have any trouble.

Thanks
Brian

In the end I would assume people won't mind if some easy cases
would get
added to the Cython wiki in form of a tutorial, i.e. if you do
FOO it
blows up but BAR fixes it. I have meant to do this for a long,
long time
in the Sage wiki, but never got around to it. But it should
probably be
in the Cython wiki for obvious reasons.
Thought?
Cheers,
Michael
_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Robert Bradshaw

2008-04-18 23:46:35 UTC

Permalink

Yes.

self._dealloc_cpp_pointers()
functionThatCallsDelete(self.foo_ptr)
...
...
functionThatCallsDelete(self.bar_ptr)
...

Will the method lookup for self._dealloc_cpp_pointers actually find
the one in the right class though? I would think that in class Foo,
self is known to be a Foo, therefore Foo._dealloc_cpp_pointers() will
always be called? I will try this, but maybe it would work if the
method were not cdefd. Also, will this type of thing work with
__cinit__?

No, inheritance for _dealloc_cpp_pointers would work just like in
Python (though if it's a cdef or cpdef method it uses vtables rather
than dictionaries). The only methods that do not get overridden are
__cinit__ and __dealloc__, and this is because we need to guarantee
that these methods get called, no matter what subclasses do.
__cinit__ is guaranteed to be called *exactly* once for an object,
and if successful __dealloc__ will be called *exactly* once on
deallocation. This makes these two methods ideal places to do memory
management associated with an object.

- Robert

Thanks
Brian

Post by Brian Granger
Thanks
Brian
On Fri, Apr 18, 2008 at 3:00 PM, Michael.Abshoff

Hi Brian,

Thanks, I will try this out. I have never used valgrind, so
this is
very helpful.

Let me know if you have any trouble.

Thanks
Brian

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Brian Granger

2008-04-18 23:39:49 UTC

Permalink

Yes.

self._dealloc_cpp_pointers()
functionThatCallsDelete(self.foo_ptr)
...
...
functionThatCallsDelete(self.bar_ptr)
...

Post by Brian Granger
Thanks
Brian
On Fri, Apr 18, 2008 at 3:00 PM, Michael.Abshoff

Hi Brian,

Thanks, I will try this out. I have never used valgrind, so this is
very helpful.

Ok, to run it do
valgrind --tool=memcheck --trace-children=yes --leak-resolution=high
--log-file=foo ./path/to/python
Then do your thing, quit python. At that point there should be
foo.$PID
in the cwd. For a C++ specific example where we did dumb things at
dealloc time from Sage look at
http://trac.sagemath.org/sage_trac/ticket/1573
Also: searching for delete[] in Sage's trac will turn up a number of
other examples. If you get stuck feel free to post the example
here and
once I take a look at the code I may be able to help you out.

Let me know if you have any trouble.

Thanks
Brian

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Robert Bradshaw

2008-04-18 23:07:20 UTC

Permalink

OK, I still haven't run valgrind (still compiling the c++ library on
Linux) but I have used gdb to figure out more of what is going on.
This leads me to a more general question about __cinit__ and
I have 2 c++ class that one of which is a subclass of the other. I am
cdef Foo_c *foo_ptr
functionThatCallsDelete(self.foo_ptr)
cdef Bar_c *bar_ptr
self.bar_ptr = newBar()
self.foo_ptr = <Foo_c *>self.bar_ptr
functionThatCallsDelete(self.bar_ptr)
The nice thing about this approach is that all the methods on Foo,
also appear on Bar, but the pointer type is right. This (almost)
makes for a really nice way of having c++ subclasses reflected in
cython....
But, what I find is that the __dealloc__ method of both Bar and Foo
are being called. Because their underlying pointers (bar_ptr and
foo_ptr) point to the same thing, the second segfaults with a double
How do people handle these things in cython? Can I prevent
__dealloc__ from being called twice?

No, __dealloc__ is guaranteed to be called, otherwise this could very
easily lead to memory leaks. I'm no C++ expert, but I'm not
understanding why __dealloc__ needs to be defined twice. Won't the
delete that's called on self.foo_ptr delete self.bar_ptr just fine
(since they're really pointers to the same object?)

If that doesn't work, try

cdef class Foo:
def __dealloc__(self):
self._dealloc_cpp_pointers()
cdef _dealloc_cpp_pointers(self):
functionThatCallsDelete(self.foo_ptr)
...

cdef class Bar(Foo):
...
cdef _dealloc_cpp_pointers(self):
functionThatCallsDelete(self.bar_ptr)
...

Thanks
Brian
On Fri, Apr 18, 2008 at 3:00 PM, Michael.Abshoff

Hi Brian,

Thanks, I will try this out. I have never used valgrind, so this is
very helpful.

Ok, to run it do
valgrind --tool=memcheck --trace-children=yes --leak-resolution=high
--log-file=foo ./path/to/python
Then do your thing, quit python. At that point there should be
foo.$PID
in the cwd. For a C++ specific example where we did dumb things at
dealloc time from Sage look at
http://trac.sagemath.org/sage_trac/ticket/1573
Also: searching for delete[] in Sage's trac will turn up a number of
other examples. If you get stuck feel free to post the example
here and
once I take a look at the code I may be able to help you out.

Let me know if you have any trouble.

Thanks
Brian

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Brian Granger

2008-04-18 22:56:41 UTC

Permalink

OK, I still haven't run valgrind (still compiling the c++ library on
Linux) but I have used gdb to figure out more of what is going on.
This leads me to a more general question about __cinit__ and
__dealloc__:

I have 2 c++ class that one of which is a subclass of the other. I am
mirroring this hierarchy in cython by creating two cdef'd classes:

cdef class Foo:

cdef Foo_c *foo_ptr

def __dealloc__(self):
functionThatCallsDelete(self.foo_ptr)

cdef class Bar(Foo):

cdef Bar_c *bar_ptr

def __cinit__(self):
self.bar_ptr = newBar()
self.foo_ptr = <Foo_c *>self.bar_ptr

def __dealloc__(self):
functionThatCallsDelete(self.bar_ptr)

The nice thing about this approach is that all the methods on Foo,
also appear on Bar, but the pointer type is right. This (almost)
makes for a really nice way of having c++ subclasses reflected in
cython....

But, what I find is that the __dealloc__ method of both Bar and Foo
are being called. Because their underlying pointers (bar_ptr and
foo_ptr) point to the same thing, the second segfaults with a double
delete problem. So question:

How do people handle these things in cython? Can I prevent
__dealloc__ from being called twice?

Thanks

Brian

On Fri, Apr 18, 2008 at 3:00 PM, Michael.Abshoff

Hi Brian,

Yes, at least to see if it can spot the problem.
To get started: Build your own python. Make sure to compile it with
"--without-pymalloc" and also set the compile flags to "-O0 -g". Build
valgrind 3.3.0 [or install it from binary packages] and run your code
under it. If it segfaults valgrind [it happens to me somewhat regularly,
but then usually something evil is happening in libSingular ;] go and
check out the 3.4svn version.

Thanks, I will try this out. I have never used valgrind, so this is
very helpful.

Ok, to run it do
valgrind --tool=memcheck --trace-children=yes --leak-resolution=high
--log-file=foo ./path/to/python
Then do your thing, quit python. At that point there should be foo.$PID
in the cwd. For a C++ specific example where we did dumb things at
dealloc time from Sage look at
http://trac.sagemath.org/sage_trac/ticket/1573
Also: searching for delete[] in Sage's trac will turn up a number of
other examples. If you get stuck feel free to post the example here and
once I take a look at the code I may be able to help you out.

Let me know if you have any trouble.

Thanks
Brian

In the end I would assume people won't mind if some easy cases would get
added to the Cython wiki in form of a tutorial, i.e. if you do FOO it
blows up but BAR fixes it. I have meant to do this for a long, long time
in the Sage wiki, but never got around to it. But it should probably be
in the Cython wiki for obvious reasons.
Thought?
Cheers,
Michael
_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Michael.Abshoff

2008-04-18 21:00:53 UTC

Permalink

Brian Granger wrote:

Hi Brian,

Yes, at least to see if it can spot the problem.
To get started: Build your own python. Make sure to compile it with
"--without-pymalloc" and also set the compile flags to "-O0 -g". Build
valgrind 3.3.0 [or install it from binary packages] and run your code
under it. If it segfaults valgrind [it happens to me somewhat regularly,
but then usually something evil is happening in libSingular ;] go and
check out the 3.4svn version.

Thanks, I will try this out. I have never used valgrind, so this is
very helpful.

Ok, to run it do

valgrind --tool=memcheck --trace-children=yes --leak-resolution=high
--log-file=foo ./path/to/python

Then do your thing, quit python. At that point there should be foo.$PID
in the cwd. For a C++ specific example where we did dumb things at
dealloc time from Sage look at

http://trac.sagemath.org/sage_trac/ticket/1573

Also: searching for delete[] in Sage's trac will turn up a number of
other examples. If you get stuck feel free to post the example here and
once I take a look at the code I may be able to help you out.

Let me know if you have any trouble.

Thanks
Brian

In the end I would assume people won't mind if some easy cases would get
added to the Cython wiki in form of a tutorial, i.e. if you do FOO it
blows up but BAR fixes it. I have meant to do this for a long, long time
in the Sage wiki, but never got around to it. But it should probably be
in the Cython wiki for obvious reasons.

Thought?

Cheers,

Michael

Brian Granger

2008-04-18 21:07:03 UTC

Permalink

Yes, at least to see if it can spot the problem.
To get started: Build your own python. Make sure to compile it with
"--without-pymalloc" and also set the compile flags to "-O0 -g". Build
valgrind 3.3.0 [or install it from binary packages] and run your code
under it. If it segfaults valgrind [it happens to me somewhat regularly,
but then usually something evil is happening in libSingular ;] go and
check out the 3.4svn version.

Thanks, I will try this out. I have never used valgrind, so this is
very helpful.

Let me know if you have any trouble.

Thanks

Brian

When people have seen this type of thing with cython, what types of
things have been the culprit? (ie, where should I start looking).

This is likey a __dealloc__ problem. Valgrind helps there since it
detects mismatched free and delete for example.

Thanks,
Brian

Cheers,
Michael

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Michael.Abshoff

2008-04-18 20:13:51 UTC

Permalink

Hi,

Hi Brian,

I have wrapped some templated c++ code using Cython. The c++ code
uses templates and I am using the "evil nasty" tricks that cython
allows to handle this. Everything was working fine (beautifully in
all actuality)....until I started getting seg faults, bus errors and
Python(4538) malloc: *** error for object 0x311140: Non-aligned
pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
These problems show up in very simple case, such as when importing a
module, instantiating a class and Python exiting.
I am running the Python 2.5.1 that comes with Leopard. From look back
at sage-devel list, it looks like this could be a problem with the
2.5.1. that comes with Leopard. I am going to try things on a Linux

I am not really aware of any specific issue with the Python on Leopard.

Does any one have general thoughts/experience is debugging these types
of things in Cython?
Is valgrind a good option? Any hints on getting started with cython+valgrind?

Yes, at least to see if it can spot the problem.

To get started: Build your own python. Make sure to compile it with
"--without-pymalloc" and also set the compile flags to "-O0 -g". Build
valgrind 3.3.0 [or install it from binary packages] and run your code
under it. If it segfaults valgrind [it happens to me somewhat regularly,
but then usually something evil is happening in libSingular ;] go and
check out the 3.4svn version.

Let me know if you have any trouble.

When people have seen this type of thing with cython, what types of
things have been the culprit? (ie, where should I start looking).

This is likey a __dealloc__ problem. Valgrind helps there since it
detects mismatched free and delete for example.

Thanks,
Brian

Cheers,

Michael

_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Robert Bradshaw

2008-04-18 20:35:10 UTC

Permalink

Hi,
I have wrapped some templated c++ code using Cython. The c++ code
uses templates and I am using the "evil nasty" tricks that cython
allows to handle this. Everything was working fine (beautifully in
all actuality)....until I started getting seg faults, bus errors and
Python(4538) malloc: *** error for object 0x311140: Non-aligned
pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
These problems show up in very simple case, such as when importing a
module, instantiating a class and Python exiting.
I am running the Python 2.5.1 that comes with Leopard. From look back
at sage-devel list, it looks like this could be a problem with the
2.5.1. that comes with Leopard. I am going to try things on a Linux
Does any one have general thoughts/experience is debugging these types
of things in Cython?
Is valgrind a good option? Any hints on getting started with cython
+valgrind?
When people have seen this type of thing with cython, what types of
things have been the culprit? (ie, where should I start looking).

I haven't personally wrapped much C++ code, but I know there is a
fair amount wrapped in sage. (look for cpp, and the corresponding pyx
files in http://hg.sagemath.org/sage-main/file/cc1e12a492fc/sage/ for
lots of examples).

Though I have not personally tried it, people do use cython+valgrind.
I use gdb mostly (and print statements). One problem with Cython code
is that is it weaves in and out of the python interpreter, the C and
Python stacktraces are of varying degrees of utility.

- Robert

Brian Granger

2008-04-18 20:20:09 UTC

Permalink

Hi,

I have wrapped some templated c++ code using Cython. The c++ code
uses templates and I am using the "evil nasty" tricks that cython
allows to handle this. Everything was working fine (beautifully in
all actuality)....until I started getting seg faults, bus errors and
messages like:

Python(4538) malloc: *** error for object 0x311140: Non-aligned
pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug

These problems show up in very simple case, such as when importing a
module, instantiating a class and Python exiting.
I am running the Python 2.5.1 that comes with Leopard. From look back
at sage-devel list, it looks like this could be a problem with the
2.5.1. that comes with Leopard. I am going to try things on a Linux
box to see if that helps. But I have some more general questions:

Does any one have general thoughts/experience is debugging these types
of things in Cython?

Is valgrind a good option? Any hints on getting started with cython+valgrind?

When people have seen this type of thing with cython, what types of
things have been the culprit? (ie, where should I start looking).

Thanks,

Brian

Stefan Behnel

2008-04-19 09:01:32 UTC

Permalink

Is valgrind a good option? Any hints on getting started with cython+valgrind?

I've never seen a better tool than valgrind for C code debugging, including
all of its tools. I use memcheck and callgrind most when debugging and optimising.

AFAIR from their web page, valgrind does not run on MacOS, though. Did that
change?

There's nothing special in using memcheck, I usually call it with the command
line that you can find in lxml's Makefile. You also need the suppression file
for the Python interpreter.

http://codespeak.net/svn/lxml/trunk/Makefile
http://codespeak.net/svn/lxml/trunk/valgrind-python.supp

You just have to take care that you have debugging symbols in all libraries
that you want to debug, and you might want to disable excessive compiler
optimisations, like function inlining. They make the debug output harder to read.

Stefan

Michael.Abshoff

2008-04-19 15:36:47 UTC

Permalink

Post by Stefan Behnel

Is valgrind a good option? Any hints on getting started with cython+valgrind?

Apple has an internal port of valgrind to OSX. It was supposed to be
released with OSX 10.5, but so far it hasn't made it out of the door.
The reason in unclear [at least to me ;)], but there is still hope I guess.

Cheers,

Michael

Post by Stefan Behnel
There's nothing special in using memcheck, I usually call it with the command
line that you can find in lxml's Makefile. You also need the suppression file
for the Python interpreter.
http://codespeak.net/svn/lxml/trunk/Makefile
http://codespeak.net/svn/lxml/trunk/valgrind-python.supp
You just have to take care that you have debugging symbols in all libraries
that you want to debug, and you might want to disable excessive compiler
optimisations, like function inlining. They make the debug output harder to read.
Stefan
_______________________________________________
Cython-dev mailing list
http://codespeak.net/mailman/listinfo/cython-dev

Stefan Behnel

2008-04-19 09:04:56 UTC

Permalink

BTW, have you tried using a different Cython version first? Just to make sure
the problem is actually in your own code.

Stefan

Stefan Behnel

2008-04-19 09:22:06 UTC

Permalink

Hi,

Post by Brian Granger
I am realizing that this is also a problem. I was creating a memory
leak by having both the subclass and superclass do allocations in
self._actual_cinit(self)
foo_ptr = doTheAllocation()
bar_ptr = doTheAllocation()
When I try this pattern Foo's _actual_cinit *always gets called.* I
__pyx_1 = ((struct __pyx_vtabstruct_6txbase_Foo *)((struct
__pyx_obj_6txbase_Foo *)__pyx_v_self)->__pyx_vtab)->_actualCinit()
When I made _actualCinit(), I got an attribute error (which makes
sense as the Python side of the class has not been built yet.)
Why is the vtable not finding the right _actualCinit implementation?
Is this a bug, or is this to be expected because the class has not
been fully built.

Hmmm, this should work (but in case vtab is not initialised correctly at this
point, I would consider that a bug).

Post by Brian Granger
Also, why can't I just do the allocations in the __init__ method?
That has a much more flexible and explicit approach when dealing with
subclasses.

The call to __init__() is a Python thing. Imagine a subclass that overrides
__init__ and forgets to make a super() call.

Stefan