0007526: Dark towers contain thousands of goblins and trolls, causing lag

Notes
(0027082) Robsoie (reporter) 2014-07-20 17:21	Additional saved game , after i very very slowly and painfully managed to enter the fortress and explore it a bit to find where was the population , on the surface of the dark fortress only few trolls and few goblins, couldn't be what was the source of the slowdown, then after more exploring underground of that fortress i see a giant room with hundred of trolls there : http://dffd.wimbli.com/file.php?id=9094 [^]

(0027756) Robsoie (reporter) 2014-07-28 06:23 edited on: 2014-07-28 06:39	Still happening in 40.05 , i thought it would be fixed with the overpopulation fixes, but in that save : http://dffd.wimbli.com/file.php?id=9184 [^] I left the fortress and moved toward the dark fortress on the south east of the world, only to have the game being simply unplayable in term of slow and lag at that point. I opened that 40.05 pocket world with World Viewer and it told me at the dark fortress site : 1 Dwarf (probably the guy i retired there) 1 Mouse Monster 4951 goblins ! 5254 trolls ! No surprise unfortunately it is unplayable (and that's a pocket world)

(0027820) Robsoie (reporter) 2014-07-28 15:37	In 40.05 i experimented today a lot with MAX_STARTING_CIV_NUMBER MAX_POP_NUMBER MAX_SITE_POP_NUMBER in the entity_default.txt and TOTAL_CIV_NUMBER TOTAL_CIV_POPULATION SITE_CAP from worlgen.txt And from my experimentations, i ended noticing that the goblins' dark fortresses always end having, in each of them, more than 1000 individuals of goblins (and as much trolls) inside, while every other civs (dwarves, humans, elves) have much more reasonnable 100/150 individuals for their main bigger site. This when i set the exact same settings for each civs in the entity_default.txt Even with only 20 years of history and no war (from what i see in World Viewer) , there's always that huge amount of goblins+trolls in a dark fortress, overwhelming numbers that just make the game unplayable when you're adventuring close to the site.

(0027823) Dwarfu (manager) 2014-07-28 15:47	Thanks for the saves/follow-ups.

(0028518) Borschwanger (reporter) 2014-08-07 06:37 edited on: 2014-08-07 06:39	The three most populous sites on my current world are actually elves this time. Elves total 8-11k with various animal populations ranging from a few thousand up to 15300 just for Tigers+Jaguars+Gorillas. Several Dark Fortresses follow up just behind. Year 250. http://dffd.wimbli.com/file.php?id=9324 [^] Sunsivy, Oaksilvers, Conflictfogs, Wheeledmalice, Plaguefield, Hexfrost, Balanceplague, Realmsterror, Waddlehate, and Beachedseduced are worst. A couple more in the low thousands and then it drops off to below 1000 in mostly caves and then quickly down to under 500. Seems about 90-95% of sites have roughly 200 or less residents. In addition to the things Robsoie tried, I have also tried making worlds where Goblins and Elves were almost exclusively homosexual species by adding [orientation] tags to no effect (not in this world). Either it doesn't work like I imagine or I did it wrong (never did try 100% homosexual, though I probably should have...). Only mod is Phoebus and used some advanced worldgen parameters (savagery, good/evil, (semi)megabeasts, probably a few others).

(0028570) smjjames (reporter) 2014-08-07 22:06	Sorry for offtopic, but is [orientation] tags really a thing?

(0028571) Borschwanger (reporter) 2014-08-07 22:52	I dunno. It is in the wiki. http://dwarffortresswiki.org/index.php/DF2014:Creature_token#ORIENTATION [^] http://dwarffortresswiki.org/index.php/DF2014:Marriage [^]

(0028582) Footkerchief (manager) 2014-08-08 09:00	0007905 pertains to ORIENTATION. Please keep the discussion here on topic.

(0029610) Robsoie (reporter) 2014-08-25 07:37 edited on: 2014-08-25 07:43	world generated on 40.10 at only +/- 50 years , no raws edited out of the gob/elf/human/dwarves names files to remove accentuated characters save there : http://dffd.wimbli.com/file.php?id=9513 [^] An elf bowman adventurer left his companion waiting at the entrance of the goblin site, all was well, until i started to approach 2 goblins dark towers on my way to their dark fortress, at such point it slowdown heavily. But from the point of the save continue directly to the North , moving between the 2 dark towers, toward the dark fortress, it's going to be slower and slower into simply unplayable if you don't sprint (in which it's slow but at least you can move). according to legends viewer, the dark fortress should have 1310 Goblins 802 Trolls 798 Ogres 5 Goblin Outcasts 1 Lizard Demons 1 Elf Outcasts

(0029777) smjjames (reporter) 2014-08-29 12:19 edited on: 2014-08-29 12:25	I don't know if Toady One needs more saves to help solve this or not, but heres one of a tower site that is barely playable. I'm stubborn enough to just try and plow my way through it. http://dffd.wimbli.com/file.php?id=9558 [^] Nothing special about the save other than to demonstrate the laggyness. In the save, I'm just trying to wait for one of the rescued prisoners to snap out of it. Right now, I wish I could just pick up the kid as I move on or give him something to cheer him up and snap out of it because it's taking too long as is. According to Legendsviewer, there should be roughly: 10154 Goblins 5100 Trolls 131 Humans 78 Human Prisoners 1 Skunk Fiends I already killed some trolls and goblins, so it'll be a little less. Also, my computer is a pretty good one, so, with computers not that good, it's unplayable.

(0029820) smjjames (reporter) 2014-08-31 07:43	Need more savegames?

(0029913) samanato (reporter) 2014-09-03 08:48	This seems to be related to the goblins' lack of need to eat or drink. Putting [NO_EAT] and [NO_DRINK] on kobolds, for example, will swell up cave populations to thousands in a similar way as in dark fortresses.

(0029915) smjjames (reporter) 2014-09-03 09:15 edited on: 2014-09-03 09:44	They can still have thousands in a cave without those tags. I'll try removing those and see how things go. Okay, in the world I tested it in, the largest populated tower has 2267 Goblins and 1850 Trolls, about what you would get with some of the most populated human towns/cities. Most of them have goblin and troll populations that range between 1k to 2k each, seeming to average between 1,200 and 1,500. Theres also several with each troll and goblin population less than a thousand. Seems MUCH more reasonable to me. To note, I did make them mortal with the same lifespans as dwarves (elves are still immortal because elves), but that didn't affect the populations. Toady One, I'm sure you're reading this, what was the origional reason for putting the [NO_EAT] and [NO_DRINK] tags on goblins? Might not be needed anymore.

(0029916) Quietust (reporter) 2014-09-03 09:53 edited on: 2014-09-03 09:59	In the original save linked in the bug report itself, there are 174 units in play at first, and after walking south and triggering the lag, there are 461 units in play. If I walk even further south to load more of the fortress (across the river and over the next trench), it increases further to 611. smjjames's savegame has a whopping 1,606 units in play, which is clearly excessive.

(0029917) smjjames (reporter) 2014-09-03 10:07 edited on: 2014-09-03 10:39	In this new world I generated for adv mode, the three largest towers (and the only three to breach 3k) have populations of 3198 Goblins 2250 Trolls and 3578 Goblins 2550 Trolls and 5497 Goblins 3100 Trolls. Which is MUCH better than the 5k+, sometimes up to 10k for each population. Most towers have less than 2k for each population and some have less than 1k for each. Removing the [NO_EAT] and [NO_DRINK] tags help, but the way that the goblins are spread out, or rather, not spread out, causes lag problems and then theres the trolls doing crazy pathing which only adds to the issues. Edit: Over a thousand in play, wow. I'll check out one of the towers in this modified save and see how thing are.

(0029921) smjjames (reporter) 2014-09-03 15:46 edited on: 2014-09-03 17:30	Now testing a low pop tower with populations of 1275 Goblins 99 Humans 67 Human Prisoners 12 Dwarves 4 Dwarf Prisoners 1 Lemur Monsters This goblin civ doesn't have trolls at that site for some reason, other sites of that civ have trolls though. I split off a compressed file from the spot I did the first loaded unit count if you guys request it. Going eastward directly towards the spire. The first small tower I get to has 139 loaded units, 69 goblins, 36 crundles and the rest various cavern wildlife. No significant lag. Second count, near the river before the spire, loaded unit number is 122. Mild conversation spam related lag and a bit of pathing lag. Third count, approx center of spire, loaded unit number is 182, including cave wildlife. Slight amount of lag. Second test with a Dark Tower population of: 3578 Goblins 2550 Trolls 104 Humans 25 Human Prisoners 3 Kobolds (I gave kobolds normal speech, which is why they are there) 3 Dwarves 1 Llama Fiends First small tower reached and the lag is starting. Units loaded (or otherwise seen in DFhacks reveal+exterminate) is 396 with 226 goblins and 130 crundles, the rest, various cave critters plus a few humans. Second reading, a couple tiles outside the tower and the lag has ramped up, loaded unit count is 690, with 468 trolls and 159 goblins. And um, when I did a wait one hour to clear out the reveal data, DF crashed. I'll check if that was a DFhack related thing or something else. Also, I've been getting fairly regular brief freezes with DF as I move forward. Earlier I thought it was because of me using reveal all, but this time I didn't use it and I'm getting the brief lagspike pause. Also, I tested the wait one hour and while it didn't crash, it did briefly freeze, and I replicated what I did earlier when it crashed and it didn't that time, so, I don't know what that crash was about.

(0030216) smjjames (reporter) 2014-09-16 08:47	Still happens in a 40.12 save which is modded to remove [NOEAT] and {NODRINK] from goblins plus givng them the lifespan of dwarves. According to legendsviewer, the site pop is almost 1,600. http://dffd.wimbli.com/file.php?id=9715 [^]

(0032666) Witty (reporter) 2015-05-10 06:18	Still very much an issue in 40.24. http://dffd.bay12games.com/file.php?id=10842 [^] Vanilla goblins, only raw mod is allowing them to be adventure-playable. The site lag makes the region entirely unplayable

(0033331) Knight Otu (manager) 2015-12-03 03:22	Robsoie posted a save at 0009170.

(0034192) Boatsniper (reporter) 2015-12-25 11:57	Still occurs in 42.03, most prevalent within Dark Goblin Fortresses. I don't think it's normal for enormous Troll civilizations to exist in the dug out underground caverns (as in taking occupation titles, like weaver, administrator, animal trainer, etc.). To add onto the problem, the goblins and trolls will start killing each other once they get line of sight. It makes absolutely no sense. This issue is easily reproducible, modded or otherwise; generate a world with a high site cap and decent history, adventure your way into a Dark Goblin Pit/Fortress, and watch the game become disastrously unplayable.

(0035724) FantasticDorf (reporter) 2016-07-30 03:59 edited on: 2016-07-30 07:05	A quick check with a third party tool (legends reader found here: http://www.bay12forums.com/smf/index.php?topic=72702.msg6658295#msg6658295 [^] ) reveals that since 43.05 site populations are still exploding into enormous numbers. On my unmodded world there are globally 95200 goblins for a world with only 25347 historical figures to have ever lived, not to mention about ten thousand elves and 8000 humans. Only 24375 goblins can be supported in the 195 dark pits that have been built to accommodate the sprawl which stays within the rules of 125 per site. With in my most prominent tower, it has 10851 goblins and 2526 trolls & 2526 beak dogs. The oldest and first tower but just 1 tower out of a network of towers with large similar statistics, and this trends between the 3 goblin civilisations to reach the big 95200 global number. The wars that occur, such as my largest and latest war 'The War of Murders' has killed 23394 people, more than the 14491 historical figures that are supposed to be dead and the war still hasn't ended or either side feeling inclination to stop for more than 60 years (declared in 167, ongoing in 250) Auxiliary populations are crashing the game and making it strenuous on systems in adventure mode to approach sites because livestock is spinning out of control to keep up with the numbers in order to 'feed' those who even don't eat, all these people and the subsequent pathing causes as described lag spikes. Murders and war cannot keep on top of the goblins and i cannot fathom where or why all these unaccounted non-historical people are coming from. @boatsniper trolls have always (intentionally or not) been like that, since they are civilians but also livestock they get the best of both worlds with automatic job assignment and population scaling. However they cannot commit to a job because they are slow learners without 3rd party tool exploitation.

(0035728) Murphy (reporter) 2016-07-31 03:10	I thought they wouldn't ALL load at the same time? 1000+ goblins is the population pool, and individual non-historical goblins are pulled out when you visit the tower, and then they offload when you leave. You shouldn't actually ever see all 1000 present at the same time. Might not be the case with trolls, however, since they count as animals. I've noticed that the structure of dark fortress can link cavern layers together, causing water from the above layer to fall through the holes into the layer below. Could that be the real reason of the lag?

(0035735) LordKnows (reporter) 2016-08-02 03:10 edited on: 2016-08-02 10:16	My 43.05 world seems affected by a similar, but somewhat different issue of overpopulation, and I was advised to take my save here. http://dffd.bay12games.com/file.php?id=12322 [^] Notable about this is that this only happened AFTER world-gen had finished and I started playing fortress and adventurers. In 850 there were around 2500 fly-men in the world, now there are over half a million largely located in just two sites, Brassdark and Pointrose. I had two consecutive crashes on attempting to enter pointrose earlier, but just now was able to enter with only extreme performance issues (unsurprisingly, it's full of fly-men, 250 thousand apparently.) I don't have a backup save before the population started growing unfortunately, but do have the xml dump from 850 which I read with legendsviewer to see the huge difference over the two years. EDIT: I just noticed that the world and fly-man population had changed in the two weeks or so of in-game time that had passed since I last looked at the legends. Pointrose now only has 14629 fly men living there, which explains why it stopped crashing for me probably. It seems that fly-man outcasts, of which the vast majority of Pointrose consisted of have simply disappeared or something leaving only the ca. 15k that were "normal" fly-men in Pointrose, and brassdark still has hundreds of thousands... Really strange. Here is a backed up save before the outcasts disappeared, this is only two weeks before the save with a quarter-million less fly-man outcasts: http://dffd.bay12games.com/file.php?id=12324 [^]

(0037948) FantasticDorf (reporter) 2018-03-16 12:42	A exerpt from the public domain of Reddit to highlight how this bug's persistence into 44.06 - 7 and that it breaks dark tower raids (which are a primary target for children rescues) Reddit thread: ( https://www.reddit.com/r/dwarffortress/comments/83cu59/i_think_the_new_versions_off_site_pillaging_could/ [^] ) And subject image: ( https://i.redd.it/pkt14h002vk01.png [^] ) Any amount of squads cannot hope to face 2526 trolls in a single encounter even with the most trained and well equipped soldiers.

(0038130) FantasticDorf (reporter) 2018-04-05 09:22 edited on: 2018-04-05 14:06	A link to the use of [USE_EVIL_ANIMALS] has been established in my save, where overpopulation and a poor allocation of non-creature tags like [PET] make them buggy because use_evil_animals applies its own uses without respecting existing creature tags. Use_evil_animals in itself is a AI only allocator for trolls to exhibit non-creature raw roles, when they are simulated in fortress mode, the trolls are actually civilian peasants & 'other animals' contributing towards population without further tags tacked on like [PET] and don't function as pure livestock at all. Trolls in themselves only have numbers culled by attacks on the civilization, and would contribute to food (troll-cannibalism & beak dogs) on goblin sites while being immortal also extends their lifespan. Troll peasants when abstracted and added to other civs as the save demonstrates causes the populations to spiral out of control even if the entity is incapable of directly eating them or using their materials like dwarves. Save: http://dffd.bay12games.com/file.php?id=13651 [^] (fixed, i put in the link while it was still loading oops)

(0038927) FantasticDorf (reporter) 2018-11-09 23:49	Ive managed to further improve my methods and deliver results, it turns out that trolls were stressing the food populations by cannibalising each other which became apparent when i changed troll RAWS (in accordance with [ANIMAL] to prevent outsourcing, local populations only) A average fully grown Troll yields 30-40 meat from muscle alone, remove that muscalture's edibilty with something like a 'tough meat' material definition with no edible parts and the populations drop to expected levels when goblins (with a raw change) & trolls are eating. This means that all civs must be prioritising animals with the most meat per capita, including fat horses as part of their food simulation over other animals like pigs or cows. Addressing this could signficantly improve the stability of animal populations by making civs value them for something else or deliberately fatten up animals by diverting crops to feeding livestock encouraging them to eat grazers and be more selective.

(0039030) bluebird9281 (reporter) 2018-12-19 23:57	removing [NO_EAT][NO_DRINK] tag from the goblins might help.(a workaround) they have such a high population in their capitals cuz they don't eat/drink and the capitals ignore population cap!

(0039031) FantasticDorf (reporter) 2018-12-20 09:46	Sometimes my fixes don't always work and it still happens inevitably, anecdotally from running my recommendation for a long time and keeping a eye on the populations. Something more concrete might still be needed.

Issue History
Date Modified	Username	Field	Change
2014-07-20 17:12	Robsoie	New Issue
2014-07-20 17:21	Robsoie	Note Added: 0027082
2014-07-28 06:23	Robsoie	Note Added: 0027756
2014-07-28 06:24	Robsoie	Note Edited: 0027756	View Revisions
2014-07-28 06:39	Robsoie	Note Edited: 0027756	View Revisions
2014-07-28 15:37	Robsoie	Note Added: 0027820
2014-07-28 15:46	Dwarfu	Relationship added	related to 0007698
2014-07-28 15:47	Dwarfu	Note Added: 0027823
2014-07-28 15:47	Dwarfu	Assigned To	=> Dwarfu
2014-07-28 15:47	Dwarfu	Status	new => acknowledged
2014-07-28 15:48	Dwarfu	Relationship added	related to 0007547
2014-07-28 15:51	Dwarfu	Relationship added	related to 0007461
2014-08-06 18:03	Footkerchief	Summary	heavy slowdown case near dark fortress => Dark towers contain thousands of goblins and trolls, causing lag
2014-08-06 18:04	Footkerchief	Relationship added	has duplicate 0007578
2014-08-06 18:04	Footkerchief	Issue Monitored: alexis06
2014-08-06 18:04	Footkerchief	Relationship added	has duplicate 0007878
2014-08-06 21:07	Talvieno	Issue Monitored: Talvieno
2014-08-07 06:37	Borschwanger	Note Added: 0028518
2014-08-07 06:38	Borschwanger	Note Edited: 0028518	View Revisions
2014-08-07 06:39	Borschwanger	Note Edited: 0028518	View Revisions
2014-08-07 21:57	Footkerchief	Status	acknowledged => confirmed
2014-08-07 22:06	smjjames	Note Added: 0028570
2014-08-07 22:52	Borschwanger	Note Added: 0028571
2014-08-08 07:11	samanato	Issue Monitored: samanato
2014-08-08 09:00	Footkerchief	Note Added: 0028582
2014-08-25 07:37	Robsoie	Note Added: 0029610
2014-08-25 07:43	Robsoie	Note Edited: 0029610	View Revisions
2014-08-29 12:19	smjjames	Note Added: 0029777
2014-08-29 12:25	smjjames	Note Edited: 0029777	View Revisions
2014-08-29 12:25	smjjames	Note Edited: 0029777	View Revisions
2014-08-31 07:43	smjjames	Note Added: 0029820
2014-09-03 08:48	samanato	Note Added: 0029913
2014-09-03 09:15	smjjames	Note Added: 0029915
2014-09-03 09:36	smjjames	Note Edited: 0029915	View Revisions
2014-09-03 09:38	smjjames	Note Edited: 0029915	View Revisions
2014-09-03 09:44	smjjames	Note Edited: 0029915	View Revisions
2014-09-03 09:53	Quietust	Note Added: 0029916
2014-09-03 09:54	Quietust	Note Edited: 0029916	View Revisions
2014-09-03 09:59	Quietust	Note Edited: 0029916	View Revisions
2014-09-03 10:07	smjjames	Note Added: 0029917
2014-09-03 10:08	smjjames	Note Edited: 0029917	View Revisions
2014-09-03 10:35	4maskwolf	Issue Monitored: 4maskwolf
2014-09-03 10:39	smjjames	Note Edited: 0029917	View Revisions
2014-09-03 15:46	smjjames	Note Added: 0029921
2014-09-03 15:56	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 16:03	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 16:27	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 16:53	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 17:03	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 17:27	smjjames	Note Edited: 0029921	View Revisions
2014-09-03 17:30	smjjames	Note Edited: 0029921	View Revisions
2014-09-04 16:05	Saram-61-97-kon	Issue Monitored: Saram-61-97-kon
2014-09-16 08:47	smjjames	Note Added: 0030216
2014-09-22 09:50	Footkerchief	Relationship added	related to 0008334
2014-10-12 07:34	NekoHatsu	Issue Monitored: NekoHatsu
2015-05-10 06:18	Witty	Note Added: 0032666
2015-05-10 10:15	lethosor	Additional Information Updated	View Revisions
2015-05-10 10:16	lethosor	Sticky Issue	No => Yes
2015-12-03 03:19	Knight Otu	Relationship added	has duplicate 0009177
2015-12-03 03:20	Knight Otu	Relationship added	has duplicate 0009170
2015-12-03 03:20	Knight Otu	Relationship added	related to 0007009
2015-12-03 03:22	Knight Otu	Note Added: 0033331
2015-12-25 11:57	Boatsniper	Note Added: 0034192
2016-02-09 02:02	park66665	Issue Monitored: park66665
2016-05-08 17:08	Dwarfu	Relationship added	has duplicate 0009676
2016-07-28 19:11	lethosor	Relationship added	has duplicate 0009637
2016-07-28 19:12	lethosor	Additional Information Updated	View Revisions
2016-07-30 03:59	FantasticDorf	Note Added: 0035724
2016-07-30 07:05	FantasticDorf	Note Edited: 0035724	View Revisions
2016-07-31 03:10	Murphy	Note Added: 0035728
2016-08-02 03:10	LordKnows	Note Added: 0035735
2016-08-02 10:16	LordKnows	Note Edited: 0035735	View Revisions
2016-11-03 17:48	FantasticDorf	Note Added: 0036031
2017-01-28 14:14	FantasticDorf	Note Deleted: 0036031
2018-03-16 12:42	FantasticDorf	Note Added: 0037948
2018-04-05 09:22	FantasticDorf	Note Added: 0038130
2018-04-05 14:06	FantasticDorf	Note Edited: 0038130	View Revisions
2018-11-09 23:49	FantasticDorf	Note Added: 0038927
2018-12-19 23:57	bluebird9281	Note Added: 0039030
2018-12-20 09:46	FantasticDorf	Note Added: 0039031