0006015: Reclaiming multiple times gradually slows the game down, leather starts to appear

(0022968)
hessuu (reporter)
2012-06-12 14:31
edited on: 2012-06-13 12:37

Savefile of the last test:

http://dffd.wimbli.com/file.php?id=6491 [^]

(0022974)
Quietust (reporter)
2012-06-12 18:24
edited on: 2012-06-12 18:25

The leather is coming from subterranean animal peoples, but unless you've got a gigantic embark, you should only be getting a dozen or less per reclaim...

(0022981)
ag (reporter)
2012-06-13 13:27
edited on: 2012-06-14 08:15

After reclaim in that save I got 8000 amphibian men in the caves, and 400000 spider webs with the removed+garbage_collect flags (the ones set when an item is removed, but not deallocated yet). The webs do disappear as expected if I wait about 50 frames.

Profiling shows 95% load coming from unit friend-or-foe checks.

(0022982)
ag (reporter)
2012-06-13 13:38
edited on: 2013-04-09 19:09

Here is a complete list of stuff there:

http://pastebin.com/ULEBUwEF [^]

After doing a few calculations, it seems that the numbers fit a hypothesis of some things doubling on every subsequent reclaim, with resulting exponential growth.

EDIT: this link no longer appears to work -Dwarfu

(0023907)
warmist (reporter)
2013-03-19 13:24

Same happens if you sleep in the same fort. Unit count grows almost exponentially.

(0023925)
Dwarfu (manager)
2013-04-09 19:08

Redmannine's save from 0006302:
http://www.2shared.com/file/T4b_gEaH/region1.html [^]

(0023927)
Redmannine (reporter)
2013-04-10 05:04
edited on: 2013-04-10 05:06

yeah mine is a very similar problem, but has gotten to the point I cannot get an adventurer to enter the fortress site without the game crashing or locking up.

Only noticed the 'leather explosion' after attempting another reclaim on the same site and everything was slowed down

(0023964)
Redmannine (reporter)
2013-05-10 10:03

I've installed DFhack to attempt to solve the problem with my fort, auto-dumping and deleting all scattered items that covered the map, as well as removing contaminates from the dwarves and remaining items. Despite doing this, the game still runs at an abysmally slow rate, the frame rate something around one frame every 30seconds if I'm lucky, most the time it seems that Dwarf Fortress will fail to respond.

(0024001)
caldfir (reporter)
2013-06-12 15:20

This issue appears in several ways, and seems to be related to the site re-generating items and creatures for sites, as it does with worldgen sites (like non-historical people in towns still being there when you come back, in a consistent way).

This is a problem since player sites remember EVERYTHING that is left on them, and as such on each successive load, brings back both the originals AND a generated copy of everything.

(0025470)
hessuu (reporter)
2014-07-09 20:51

Still present in 0.40.01.

(0026664)
Quietust (reporter)
2014-07-16 13:40

I tried getting this to happen in 0.40.03, but the animal people are simply vanishing altogether after the initial abandon (and don't show up again on reclaim).

(0027039)
hessuu (reporter)
2014-07-20 11:23

Quietust, are you getting the massive performance hit and the leather still? Because I sure did.

(0028575)
bralbaard (reporter)
2014-08-08 02:26

I can reproduce this in 40.06.

In 34.0x this problem was present in adventure mode as well, and in this case I reproduced it in adventure mode in 40.06, I guess the underlying problem is the same as for fortress mode.

I visited an abandoned -player created- fortress, and after sleeping there repeatedly (five or six times?) severe lag started to occur, and leather started to accumulate on the surface. Here is a save game with an unfortunate adventurer at the site, the game is hardly responsive due to lag. leather is present on the surface (but was never produced at the site)

http://dffd.wimbli.com/file.php?id=9326 [^]

This is a rather nasty bug that had a big impact on adventure succession games in 34.11, causing some games to be abandoned completely, and severely impacting others.

(0029612)
Toady One (administrator)
2014-08-25 11:25

The animal people camp items can't be easily salvaged without the full army item update. I'm just going to remove the items themselves from the camps for the time being and leave a note to make sure this sort of thing doesn't happen when they return.

Issue History
Date Modified	Username	Field	Change
2012-06-12 14:06	hessuu	New Issue
2012-06-12 14:07	hessuu	Issue Monitored: hessuu
2012-06-12 14:07	hessuu	Issue End Monitor: hessuu
2012-06-12 14:08	hessuu	Issue Monitored: hessuu
2012-06-12 14:31	hessuu	Note Added: 0022968
2012-06-12 16:42	Footkerchief	Relationship added	related to 0001181
2012-06-12 18:24	Quietust	Note Added: 0022974
2012-06-12 18:25	Quietust	Note Edited: 0022974	View Revisions
2012-06-13 12:34	hessuu	Tag Attached: reclaim
2012-06-13 12:34	hessuu	Tag Attached: fps
2012-06-13 12:34	hessuu	Tag Attached: leather
2012-06-13 12:37	hessuu	Note Edited: 0022968	View Revisions
2012-06-13 13:27	ag	Note Added: 0022981
2012-06-13 13:38	ag	Note Added: 0022982
2012-06-14 08:15	Footkerchief	Note Edited: 0022981	View Revisions
2013-03-19 13:23	warmist	Issue Monitored: warmist
2013-03-19 13:24	warmist	Note Added: 0023907
2013-04-09 19:06	Dwarfu	Relationship added	has duplicate 0006302
2013-04-09 19:08	Dwarfu	Note Added: 0023925
2013-04-09 19:09	Dwarfu	Note Edited: 0022982	View Revisions
2013-04-10 05:04	Redmannine	Note Added: 0023927
2013-04-10 05:06	Redmannine	Note Edited: 0023927	View Revisions
2013-04-10 05:07	Redmannine	Issue Monitored: Redmannine
2013-05-10 10:03	Redmannine	Note Added: 0023964
2013-05-10 10:06	Redmannine	Tag Attached: crash
2013-05-10 10:06	Redmannine	Tag Attached: Fortress
2013-05-10 10:06	Redmannine	Tag Attached: lag
2013-06-12 15:20	caldfir	Note Added: 0024001
2014-01-03 10:43	Footkerchief	Relationship added	has duplicate 0006400
2014-01-03 10:43	Footkerchief	Relationship added	related to 0005106
2014-03-26 00:06	ag	Issue Monitored: ag
2014-03-26 01:49	Dwarfu	Assigned To	=> Dwarfu
2014-03-26 01:49	Dwarfu	Status	new => acknowledged
2014-05-28 03:11	Knight Otu	Relationship added	has duplicate 0006491
2014-07-09 20:51	hessuu	Note Added: 0025470
2014-07-16 13:40	Quietust	Note Added: 0026664
2014-07-20 11:23	hessuu	Note Added: 0027039
2014-08-08 02:26	bralbaard	Note Added: 0028575
2014-08-08 09:05	Footkerchief	Status	acknowledged => confirmed
2014-08-08 16:03	Footkerchief	Relationship added	related to 0006549
2014-08-25 11:25	Toady One	Note Added: 0029612
2014-08-25 11:25	Toady One	Status	confirmed => resolved
2014-08-25 11:25	Toady One	Fixed in Version	=> Next Version
2014-08-25 11:25	Toady One	Resolution	open => fixed
2014-08-25 11:25	Toady One	Assigned To	Dwarfu => Toady One

Notes
(0022968) hessuu (reporter) 2012-06-12 14:31 edited on: 2012-06-13 12:37	Savefile of the last test: http://dffd.wimbli.com/file.php?id=6491 [^]

(0022974) Quietust (reporter) 2012-06-12 18:24 edited on: 2012-06-12 18:25	The leather is coming from subterranean animal peoples, but unless you've got a gigantic embark, you should only be getting a dozen or less per reclaim...

(0022981) ag (reporter) 2012-06-13 13:27 edited on: 2012-06-14 08:15	After reclaim in that save I got 8000 amphibian men in the caves, and 400000 spider webs with the removed+garbage_collect flags (the ones set when an item is removed, but not deallocated yet). The webs do disappear as expected if I wait about 50 frames. Profiling shows 95% load coming from unit friend-or-foe checks.

(0022982) ag (reporter) 2012-06-13 13:38 edited on: 2013-04-09 19:09	Here is a complete list of stuff there: http://pastebin.com/ULEBUwEF [^] After doing a few calculations, it seems that the numbers fit a hypothesis of some things doubling on every subsequent reclaim, with resulting exponential growth. EDIT: this link no longer appears to work -Dwarfu

(0023907) warmist (reporter) 2013-03-19 13:24	Same happens if you sleep in the same fort. Unit count grows almost exponentially.

(0023925) Dwarfu (manager) 2013-04-09 19:08	Redmannine's save from 0006302: http://www.2shared.com/file/T4b_gEaH/region1.html [^]

(0023927) Redmannine (reporter) 2013-04-10 05:04 edited on: 2013-04-10 05:06	yeah mine is a very similar problem, but has gotten to the point I cannot get an adventurer to enter the fortress site without the game crashing or locking up. Only noticed the 'leather explosion' after attempting another reclaim on the same site and everything was slowed down

(0023964) Redmannine (reporter) 2013-05-10 10:03	I've installed DFhack to attempt to solve the problem with my fort, auto-dumping and deleting all scattered items that covered the map, as well as removing contaminates from the dwarves and remaining items. Despite doing this, the game still runs at an abysmally slow rate, the frame rate something around one frame every 30seconds if I'm lucky, most the time it seems that Dwarf Fortress will fail to respond.

(0024001) caldfir (reporter) 2013-06-12 15:20	This issue appears in several ways, and seems to be related to the site re-generating items and creatures for sites, as it does with worldgen sites (like non-historical people in towns still being there when you come back, in a consistent way). This is a problem since player sites remember EVERYTHING that is left on them, and as such on each successive load, brings back both the originals AND a generated copy of everything.

(0025470) hessuu (reporter) 2014-07-09 20:51	Still present in 0.40.01.

(0026664) Quietust (reporter) 2014-07-16 13:40	I tried getting this to happen in 0.40.03, but the animal people are simply vanishing altogether after the initial abandon (and don't show up again on reclaim).

(0027039) hessuu (reporter) 2014-07-20 11:23	Quietust, are you getting the massive performance hit and the leather still? Because I sure did.

(0028575) bralbaard (reporter) 2014-08-08 02:26	I can reproduce this in 40.06. In 34.0x this problem was present in adventure mode as well, and in this case I reproduced it in adventure mode in 40.06, I guess the underlying problem is the same as for fortress mode. I visited an abandoned -player created- fortress, and after sleeping there repeatedly (five or six times?) severe lag started to occur, and leather started to accumulate on the surface. Here is a save game with an unfortunate adventurer at the site, the game is hardly responsive due to lag. leather is present on the surface (but was never produced at the site) http://dffd.wimbli.com/file.php?id=9326 [^] This is a rather nasty bug that had a big impact on adventure succession games in 34.11, causing some games to be abandoned completely, and severely impacting others.

(0029612) Toady One (administrator) 2014-08-25 11:25	The animal people camp items can't be easily salvaged without the full army item update. I'm just going to remove the items themselves from the camps for the time being and leave a note to make sure this sort of thing doesn't happen when they return.