Tuple unpacking goodness

Todays pleasant surprise:

>>> a = {'a': (0, 1), 'b': (2, 3)}
>>> for k, (v1, v2) in a.iteritems():
...     print k, v1, v2
... 
a 0 1
b 2 3
>>>

Nice!

The new faces of Europe

I can't help but agree.

Finding memory leaks in extension modules

After reading Ned Batchelder's post on his experience hunting a memory leak (which turned out to be a reference counting error) it occurred to me that even tough I have a script to check memory usage I should also really be checking reference counts with sys.gettotalrefcount(). And indeed, after adding this to my script I found one reference count leak. I still have faith in my script as it was before really since the reference leak in question was not making me loose memory - subtle bugs eh?

But how do you check an extension module for memory leaks? This seems pretty undocumented so here my approach:

• First you really need a debug build of python, this helps a lot since you get to use sys.gettotalrefcount() and get more predictable memory behaviour. The most complete way to build this is something like this (the MAXFREELIST stuff adapted from this):

  s="_MAXFREELIST=0"
  ./configure --with-pydebug --without-pymalloc --prefix=/opt/pydebug \
  CPPFLAGS="-DPyDict$s -DPyTuple$s -DPyUnicode$s -DPySet$s -DPyCFunction$s -DPyList$s -DPyFrame$s -DPyMethod$s"
  make
  make install
  

• Now run the test suite using valgrind, this is troublesome but a very useful thing to do. The valgrind memory checker will help you identify problems pretty quickly. It can be confused about Python however, but you only care about your extension module so you need to filter most of this. Luckily the python distribution ships with a valgrind suppression file in Misc/valgrind-python.supp that you can use, it's not perfect but helps. This is how I invoke valgrind:

  $ /opt/pydebug/bin/python setup.py build
  $ valgrind --tool=memcheck \
      --suppression=~/python-trunk/Misc/valgrind-python.supp \
      --leak-check=full /opt/pydebug/bin/python -E -tt setup.py test
  ==8599== Memcheck, a memory error detector
  ==8599== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
  ==8599== Using Valgrind-3.5.0-Debian and LibVEX; rerun with -h for copyright info
  ==8599== Command: /opt/pydebug/bin/python -E -tt setup.py test
  ==8599== 
  ==8599== Conditional jump or move depends on uninitialised value(s)
  ==8599==    at 0x400A66E: _dl_relocate_object (do-rel.h:65)
  ==8599==    by 0x4012492: dl_open_worker (dl-open.c:402)
  ==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
  ==8599==    by 0x4011D0D: _dl_open (dl-open.c:616)
  ==8599==    by 0x405AC0E: dlopen_doit (dlopen.c:67)
  ==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
  ==8599==    by 0x405B0DB: _dlerror_run (dlerror.c:164)
  ==8599==    by 0x405AB40: dlopen@@GLIBC_2.1 (dlopen.c:88)
  ==8599==    by 0x8132727: _PyImport_GetDynLoadFunc (dynload_shlib.c:130)
  ==8599==    by 0x81199D9: _PyImport_LoadDynamicModule (importdl.c:42)
  ==8599==    by 0x81161FE: load_module (import.c:1828)
  ==8599==    by 0x8117FAF: import_submodule (import.c:2589)
  ...
  running test
  ...
  FAILED (failures=4, errors=2)
  ==8599== 
  ==8599== HEAP SUMMARY:
  ==8599==     in use at exit: 1,228,588 bytes in 13,293 blocks
  ==8599==   total heap usage: 280,726 allocs, 267,433 frees, 70,473,201 bytes allocated
  ==8599== 
  ==8599== LEAK SUMMARY:
  ==8599==    definitely lost: 0 bytes in 0 blocks
  ==8599==    indirectly lost: 0 bytes in 0 blocks
  ==8599==      possibly lost: 1,201,420 bytes in 13,014 blocks
  ==8599==    still reachable: 27,168 bytes in 279 blocks
  ==8599==         suppressed: 0 bytes in 0 blocks
  ==8599== Rerun with --leak-check=full to see details of leaked memory
  ==8599== 
  ==8599== For counts of detected and suppressed errors, rerun with: -v
  ==8599== Use --track-origins=yes to see where uninitialised values come from
  ==8599== ERROR SUMMARY: 75 errors from 5 contexts (suppressed: 19 from 6)
  

  Note that the output is very verbose, usually I actually start with --leak-check=summary. Firstly notice that valgrind gives a lot of warnings already before your extension module gets loaded, that's python's problems and not yours so skip over that. The stuff output after (and during) the output of the test suite is what interests you. Most importantly look at the definitely lost line if that's not zero the you have a leak. The possibly lost is just python's problem (which sadly might hide problems you created too). When you do have lost blocks valgrind will give you a stack trace to pinpoint it, but you'll have to swim trough lots of "possibly lost" stack traces of python to find it. Best is probably to grep for your source files in the output.

• Next you should create function you want to execute in a loop, this should be exercising the code you want to tests for leaks. If you're really thourough possibly the entire test suite wrapped up in a function call would be good.

  Wrap it all up in a script that checks the memory usage and reference counts on each loop and compares the start and end values. Getting memory usage might be tricky from python (or you can use PSI of course) so depending on your situation you might prefer to do this with an script from your operating system.

  For PSI this is the script I currently use. I clearly have it easy since I can be sure PSI will be available :-). The reason I don't automate this script further (you could turn it into a unittest) is that I prefer to manually look at the output. Both memory and reference counting are funny and will most likely grow a little bit anyway. By looking at the output I can easily spot if it keeps growing or stabilises, there is only a problem if it keeps growing with every iteration (don't be afraid to run with many many iterations from time to time). When automating this you probably end up allowing some margin and might miss small leaks.

Hopefully some of this was useful for someone.

New Python System Information release!

I've just released a new version of PSI! PSI is a cross-platform Python package providing real-time access to processes and other miscellaneous system information such as architecture, boottime and filesystems. Among the highlights of this release are:

• Improved handling of time: We now have our own object to represent time. This may seem silly at first but it actually makes it easier to use all the normal ways of representing time easily as well as provide the highest possible accuracy.
• Improved handling of process names and arguments: There is an entire wiki page dedicated to this, but basically this simplifies presenting sensible process names to users massively since some attributes will always have meaningful values.
• Restructured exceptions: Whenever accessing attributes you will get a subclass of AttributeError like it should be so now you can happily use getattr().
• New experimental methods on Process objects: You can now send signals to processes using the .kill() method and find their children by using the .children() method.
• New experimental mount module: You can get detailed information about all mounted filesystems using this new module. It provides mount information as well as usage.

Another notable improvement is the ability to read the arguments of a 64-bit process while running inside a 32-bit python process on Solaris. It's small and almost no-one will notice it but make it so much more consistent!

Release early, release often: fail

Now for the bad news: all this means the API has changed in a backwards incompatible way.

It was already pretty obvious shortly after the last release that this would happen and was the reason I was hoping to release a new version soon. But that didn't happen. Although the last version had a "beta" version number on it it's trove classifier still claimed to be "alpha" and in end we don't promise API stability till we hit 1.0. But it's still not nice. Once we hit 0.3 we will actually try not to introduce changes to the API if possible. We intend to help this by using FutureWarning for APIs we're not sure about yet. In the mean time let's see how the Process API holds out during this release, hopefully it will prove to be good and require no more changes.

Credits

As before, thanks to Chris Miles and Erick Tryzelaar for helping out.

Synchronous classes in Python

What I'd like to build is an object that when doing anything with it would first acquire a lock and release it when finished. It's a pattern I use fairly-regularly and I am getting bored of always manually defining a lock next to the other object and manually acquiring and releasing it. It's also error prone.

Problem is, I can't find how to do it! The __getattribute__ method is bypassed by implicit special methods (like len() invoking .__len__()). That sucks. And from the description of this by-passing there seems to be no way to get round it. For this one time where I thought I found a use for meta-classes they still don't do the trick...