For example:
http://ubuntuforums.org/showthread.php?t=345832
Title is "more memory for python script". Almost exactly what I was looking for (I searched for "give more memory to python").
If you look down the thread, user "pmasiar" started debugging the algorithm and also gave incorrect statement about memory management ("Python takes all the memory it needs."). It actually doesn't. On my Windows machine I saw it only take 2 GB and then failed with memory error. It didn't even try to use page file, let alone I had more than 2 GB available.
* * *
The right answer turned out to be: 32 bit Python doesn't attempt to use 64-bit memory APIs, so the memory is limited to 2 GB or 3 GB due to address space limitation:
http://www.eggheadcafe.com/software/aspnet/35757570/memoryerror-can-i-use-more.aspx
It's interesting that the task of finding out how much memory is available to Python, is hell damn hard task, because Python wants the developer to abstract away from physical representation of data in memory, so there's no sizeof(). Also, Python does tricks like he doesn't allocate extra memory for strings with the same contents. It probably keeps hashes of all strings and doesn't reserve a new string in case if it's the same. Since strings are immutable, it can do it. This was one of the things I've found out in my memory test.
Python doesn't seem to use this trick with newly generated long numbers: I took factorial implementation from http://metaleks.net/programming/the-evolution-of-a-python-programmer (thanks
from array import array # this is sexy! def factorial(x): res = 1 for i in xrange(2, x + 1): res *= i return res l = [] i = 0 while True: # each 10000 takes (27264-15784)*1024 memory for j in xrange(0, 10000): l.append(factorial(1000)) i = i + (27264-15784)*1024 print i/1024/1024,"mb allocated"I've just noticed that I don't use "array" in this implementation. Actually tried to use arrays of bytes but got inconsistent results RE memory footprint. But I saved the import statement for memory, it's gorgeous!