0002715: Crash during huge, lengthy world gen (out-of-memory)

Notes
(0010409) TwelveBaud (reporter) 2010-07-15 23:51	My second test, this time with the Large Address Aware setting hacked into the executable, appears to be going well. After 6063 years and 7.2 million events, Dwarf Fortress went over the 2GB limit and kept going. It looks like legends are what's driving memory usage, and that using /LARGEADDRESSAWARE should be safe. Although I'm running on an extremely weird configuration of Windows XP 64-bit for Itanium Systems, this issue is also present on Windows XP 64-bit for Extended Systems, and the 64-bit editions of Windows Vista and Windows 7. (The issue also exists on the 32-bit editions of all versions of Windows, but there's not much to do that can fix it there.) While Windows may have plenty of free memory to give out, after a while it has no place in the running Dwarf Fortress to put it. In 32-bit versions of Windows, the processor can keep track of 4GB of "memory space" at a time, 2GB reserved for the program, and 2GB reserved for Windows. Thus, most programs could only access 2GB of memory each. Several developers took advantage of the existance of the reserved space, and, if there was an error making an object, would use the error code in place of the object pointer, which worked because conveniently error codes exactly overlapped the reserved space. In 64-bit Windows, when running a 32-bit program, Windows is located entirely outside of the 4GB "memory space" -- since it has 256TB of memory space to play with -- but the program is still limited to 2GB to avoid breaking programs that use that technique. However, adding a special compiler flag tells Windows that the technique is not in use, which means the program can safely handle up to 4GB of memory space. This is still less than a native 64-bit program could access, but double what would normally be available. This is realllly really easy. Another option is to use "memory-mapped 'files'". This technique can be used to allocate much more than 2 or 4GB of space, and provides the program with a smaller, movable "view" of the space. Thus, while you can't access the whole thing at once, you can "scroll through" as much memory as you want, limited only to how much at a time in that small window. Raymond Chen has a brief article summarizing the technique: http://blogs.msdn.com/b/oldnewthing/archive/2004/08/10/211890.aspx [^] This is a little harder, since it requires a rewrite of how to access whatever you're accessing, but offers the greatest compatibility for the effort required. A final option is to compile a special 64-bit edition of Dwarf Fortress, which doubles the size of each and every pointer. This can only be done after making sure that pointers are never stuffed into longs/DWORDs, and longs/DWORDS are never converted into pointers, since it requires a long long or a QWORD to fully represent the size of a pointer in 64-bit code. It also will not run on 32-bit editions of Windows. However, this will give it room for up to sixteen terabytes of memory, more than can possibly be installed or swapped out. This... amounts to a complete rewrite. And probably will break save compatibility. Nevertheless, it is a possible option. Anyway, I've been letting the altered Dwarf Fortress chug away as I've been typing this, and it looks like "Nitom Anur", the Planet of Wonder, has 1,394,967 historical figures in 12,152,449 events in 10,000 years, still somehow in the Age of Myth, with a memory high water mark of 3,359 MB, and no problems whatsoever. ☺

(0010415) Cryten (reporter) 2010-07-16 06:16	This seems more like a suggestion then anything else. The only thing Toady could do with this report bugwise would be to either force you not to set large history dates (I believe culling history is there to allow this) or put in better error handling to catch the program going over memory limit.

(0018156) G-Flex (reporter) 2011-07-08 14:37	Cryten: If the program is crashing, that's a bug, no two ways about it. Disallowing extremely long histories, better error-handling, or some other method of dealing with this would certainly be an improvement over the program running out of address space and crashing.

Issue History
Date Modified	Username	Field	Change
2010-07-15 21:19	TwelveBaud	New Issue
2010-07-15 23:51	TwelveBaud	Note Added: 0010409
2010-07-16 04:08	Logical2u	Relationship added	child of 0000136
2010-07-16 06:16	Cryten	Note Added: 0010415
2011-07-08 14:37	G-Flex	Note Added: 0018156
2012-02-21 06:51	Footkerchief	Relationship replaced	related to 0000136
2014-09-11 21:12	chuzzum	Issue Monitored: chuzzum
2015-12-06 18:51	Dwarfu	Relationship added	related to 0009243