Dwarf Fortress Bug Tracker - Dwarf Fortress
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0000332Dwarf FortressDwarf Mode -- Stockpilespublic2010-04-03 17:272010-07-26 07:17
Reporteroliver 
Assigned ToToady One 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version0.31.01 
Target VersionFixed in Version0.31.09 
Summary0000332: Raw files parsed in strict alphabetic order in Linux but not Windows, causing ID mismatches for saves moved between systems
DescriptionUsually, on embark, bituminous coal and lignite are considered to be an "Economic" stone, in terms of the stone stockpile categorization.

However, my current fort (about 3 years in) has suddenly decided that they should be in the "Other" category. The same fort was OK previously.

They are still red ("don't use as misc stone") in the 'z'->Stone screen.

This is possibly related to 0000219 as I seem to have lost the ability to make coke from those two stone types too.
Steps To Reproduce
Additional Information
TagsSave Included, SDL-only
Relationships
has duplicate 0000219resolved Footkerchief Smelting reaction ingredients/products change during gameplay, possibly Wine-specific 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2010-04-03 17:27oliverNew Issue
2010-04-26 07:22ZetNote Added: 0005113
2010-04-26 07:25DoctorZuberNote Added: 0005114
2010-04-26 10:31FootkerchiefNote Added: 0005138
2010-04-26 10:46FootkerchiefRelationship addedchild of 0000219
2010-04-26 15:52oliverNote Added: 0005196
2010-04-28 09:16CptFastbreakNote Added: 0005539
2010-04-28 09:16CptFastbreakNote Edited: 0005539bug_revision_view_page.php?bugnote_id=0005539#r2017
2010-04-28 09:28CptFastbreakIssue Monitored: CptFastbreak
2010-04-28 12:57FootkerchiefNote Added: 0005568
2010-04-28 16:13CptFastbreakNote Added: 0005603
2010-05-07 10:33FootkerchiefRelationship replacedrelated to 0000219
2010-05-16 23:52FootkerchiefSummaryCoal / lignite mysteriously move to "other" stone type => Bituminous coal / lignite mysteriously move to "other" stone type
2010-05-16 23:54FootkerchiefTag Attached: Save Included
2010-06-05 03:07EskarionNote Added: 0007711
2010-06-13 01:49Phancy PhysicistNote Added: 0008257
2010-06-13 23:59Phancy PhysicistNote Added: 0008391
2010-06-14 12:09jeti0Issue Monitored: jeti0
2010-06-14 12:14jeti0Note Added: 0008420
2010-06-28 01:49oliverNote Added: 0009206
2010-06-28 01:51oliverNote Edited: 0009206bug_revision_view_page.php?bugnote_id=0009206#r3529
2010-06-28 01:55oliverNote Edited: 0009206bug_revision_view_page.php?bugnote_id=0009206#r3530
2010-06-28 02:48oliverIssue Monitored: oliver
2010-06-29 14:53uggiNote Added: 0009325
2010-07-01 07:55uggiNote Added: 0009456
2010-07-01 08:07FootkerchiefNote Added: 0009458
2010-07-01 08:08FootkerchiefSummaryBituminous coal / lignite mysteriously move to "other" stone type => Bituminous coal / lignite mysteriously move to "other" stone type, due to raw files not being parsed in strict alphabetic order
2010-07-01 18:14oliverNote Added: 0009484
2010-07-01 20:35FootkerchiefRelationship replacedhas duplicate 0000219
2010-07-01 20:36FootkerchiefSummaryBituminous coal / lignite mysteriously move to "other" stone type, due to raw files not being parsed in strict alphabetic order => Raw files not parsed in strict alphabetic order in Linux, causing ID mismatches for saves moved between systems
2010-07-01 20:36FootkerchiefSticky IssueNo => Yes
2010-07-02 05:05oliverNote Added: 0009509
2010-07-02 05:24FootkerchiefSummaryRaw files not parsed in strict alphabetic order in Linux, causing ID mismatches for saves moved between systems => Raw files parsed in strict alphabetic order in Linux but not Windows, causing ID mismatches for saves moved between systems
2010-07-02 05:52m4st0rNote Added: 0009511
2010-07-02 06:52m4st0rNote Edited: 0009511bug_revision_view_page.php?bugnote_id=0009511#r3663
2010-07-02 08:31jeti0Issue End Monitor: jeti0
2010-07-02 09:43numBoneNote Added: 0009527
2010-07-02 19:24oliverNote Added: 0009551
2010-07-02 19:45FootkerchiefTag Attached: SDL-only
2010-07-02 19:47FootkerchiefIssue Monitored: Baughn
2010-07-02 19:47FootkerchiefNote Added: 0009554
2010-07-03 05:03BaughnNote Added: 0009568
2010-07-03 05:21Toady OneNote Added: 0009571
2010-07-03 05:21Toady OneAssigned To => Toady One
2010-07-03 05:21Toady OneStatusnew => acknowledged
2010-07-05 05:07uggiIssue Monitored: uggi
2010-07-08 03:59Toady OneNote Added: 0009774
2010-07-08 03:59Toady OneStatusacknowledged => resolved
2010-07-08 03:59Toady OneFixed in Version => 0.31.09
2010-07-08 03:59Toady OneResolutionopen => fixed
2010-07-09 21:06oliverIssue End Monitor: oliver
2010-07-12 12:53FootkerchiefSticky IssueYes => No
2010-07-26 07:17uggiIssue End Monitor: uggi

Notes
(0005113)
Zet   
2010-04-26 07:22   
I can confirm this. It has happened to me in two different forts now. Bituminous coal and lignite are red in the stones screen by default, but they don't list the "make coke" ability, and the smelters do not get the "make coke" menu option, even though loads of the resource are available.
(0005114)
DoctorZuber   
2010-04-26 07:25   
check (z) stone and see if it has changed to green (which it can do if you find a lot). I'm wondering if this also moves it in the stockpile now as well.
(0005138)
Footkerchief   
2010-04-26 10:31   
oliver and Zet: what OS are you using?
(0005196)
oliver   
2010-04-26 15:52   
In my case this was the same save that was transferred from XP to Linux/Wine and suffered from the disappearing reaction problem as described in 0000219, so it's quite likely related.
(0005539)
CptFastbreak   
2010-04-28 09:16   
Cross-posting this from 0000219, as I've been told that it's an occurence of this bug.

OS: Windows XP Professional 32Bit

I've had the same occurence with Bituminous Coal and Lignite:
- They're still red in the z stone screen.
- They don't list the 'make coke' ability.
- They are in 'ordinary stone' in the stockpile menu.
- It's possible that making coke still worked for some time as I only noticed this when my repeating orders got cancelled.
- Now I can add the jobs via manager, but it states 'Urist McSmelter cancels Make Coke from Lignite: needs rock' and cancels.
- I got a few reoccurring crashes afterwards that seemed to stop when I cancelled the manager coke orders, so ordering coke might eventually lead to crashes. I'm not positive on this one though.
- I can still make Pig Iron and Steel from charcoal coke.
- Saving and reloading did not help, smelter is not in a burrow, save was never transferred.
(0005568)
Footkerchief   
2010-04-28 12:57   
CptFastbreak: do you have anything interesting in your errorlog?

Also, it might be helpful to upload that save to http://dffd.wimbli.com/ [^]
(0005603)
CptFastbreak   
2010-04-28 16:13   
Here's the save file:
http://dffd.wimbli.com/file.php?id=2261 [^]

The errorlog doesn't contain anything related to the smelter or to coke, just a bunch of "path fail" and "loop path fail" mostly related to the construction of buildings and the fleeing from invaders.
(0007711)
Eskarion   
2010-06-05 03:07   
I encountered this bug after transfering a savegame made with 31.04 - legacy version to 31.05 - sdl-version (tried linux as well as win). Using the save with the 31.05 - legacy version works fine.
I also created a new world with the 31.05 sdl - version (win) and here the reactions listed in the z-stone menu are correct, although most of the ores now show the ability to make sharp blades while Obsidian isn't listed at all.
(0008257)
Phancy Physicist   
2010-06-13 01:49   
I also encounter this bug. I used wine before and loaded my save to linux and now I can't make coal. Interestingly, if I save and load into wine (new and old windows versions) the save game works fine. When I go back to linux and it is broken again. So i guess there is a linux-only bug here.
(0008391)
Phancy Physicist   
2010-06-13 23:59   
So one way to work a round this is to use links
"ln -s path_to_linux_df/data/save/ path_to_win_df/data/save"
then just open with wine queue up coal creation then exit open in linux and the coal creation will start up.

Not a great solution but works for now.
(0008420)
jeti0   
2010-06-14 12:14   
Actually, not only the coke creation tasks are missing from the smelters. All make <alloy> from <bars/ore> tasks are missing, too.

This happens when trying to use Wine generated world on native Linux DF and vice versa.
(0009206)
oliver   
2010-06-28 01:49   
(edited on: 2010-06-28 01:55)
I have managed to work out how to reproduce this. Someone may want to reopen 0000219 with this info, since it's more directly relevant there.

As suggested in a comment on 0000219, it is to do with the order in which the raws are loaded.

I reproduced this on Linux by the following steps:

* Create a new region, embark
* Verify that bituminous coal has the right reactions.
* Save the region.
* Run 'ls -lU | grep inorganic' in data/save/regionN/raw/objects. -U provides an "unsorted" list of files that will reflect the order that the OS-level directory-scanning APIs return. In my case I get this order:

inorganic_other.txt
inorganic_stone_layer.txt
inorganic_stone_mineral.txt
inorganic_stone_soil.txt
inorganic_metal.txt
inorganic_stone_gem.txt

* Do something to inorganic_stone_mineral.txt (which contains the bituminous coal definition) so it is in a different position in the order. It will be very OS and filesystem specific what you need to do here. In my case, on my Linux install, renaming it to "inorganic_test.txt" worked, producing this order:

inorganic_other.txt
inorganic_stone_layer.txt
inorganic_stone_soil.txt
inorganic_test.txt
inorganic_metal.txt
inorganic_stone_gem.txt

* Load the saved region, check reactions, note that bituminous coal is missing reactions and is considered an "other" stone in the stockpile lists.

* Save
* rename the raw file back to its original position
* Reload
* The reactions are fixed.

I'm guessing there is an ID stored internally somewhere in the region save that is just an index into the stone types loaded from the raws, and that index can end up pointing somewhere different depending on the order the raws are loaded. So moving between Wine/Linux/Windows etc, or just bad luck with what the filesystem does under the covers, has a chance of changing the ordering of the raws and causing chaos. Fun times!

Possible fix: sort the raw filenames before loading them, since that should be a stable order regardless of what the filesystem does. But that will break old saves :(

(Might be worth alerting Toady to this one, since it has the potential to cause all sorts of savegame problems - backups that don't work, can't transfer saves between different systems, savegames mysteriously going bad for no apparent reason, etc)
(0009325)
uggi   
2010-06-29 14:53   
I'm having this bug as well. I started the game under Wine. Later on I switched to Windows, mainly because the game ran so much faster under Windows. I moved the game files to Windows partition, and also upgraded the game a few times. I think I started with 31.03, running 31.08 now.

Then I noticed I can't make coke from bituminous coal. I can use the manager noble to add the job, but the dwarfs just cancel it, saying "..cancels Make coke from bituminous coal: Needs melanite."

I tried running the game under Wine again. I first copied the game files back to the Linux partition. Now the smelter shows "Make coke from bituminous coal" and "Make coke from lignite" options, which were not available under Windows.

However, as the game files have been moved from one partition to another and then copied back again, I'm not sure what the original order was when I started the game. The Linux partition is in ext3 format, and Windows partition is in NTFS format. This is the order inorganic* files are now listed in the Linux partition:
inorganic_stone_mineral.txt
inorganic_stone_layer.txt
inorganic_other.txt
inorganic_stone_soil.txt
inorganic_metal.txt
inorganic_stone_gem.txt

Now if I run the native Linux version with the exact same save files, linked from the Wine directory so they should be listed in the same order, I'm again unable to create coke.

Also if I try running the game straight from the Windows partition under Wine, I'm unable to create coke again. The inorganic* files are listed alphabetically in that partition.

In short: the only way to create coke now is to run the game under Wine from the Linux partition.
(0009456)
uggi   
2010-07-01 07:55   
If I rename those files, for example inorganic_stone_mineral.txt to inorganic_00_stone_mineral.txt (so it comes first in alphabetic order) I can then create coke in Windows too.

But, looking at the stone data in the z-stone-menu reveals that everything else is borked too. I could for example make bismuth bronze bars from cinnabar or brimstone. Everything in that screen changes depending on the order I rename those files.

So there's like ..umm.. 720 (?) different combinations how the inorganic* files can be ordered? I can't find the correct order, so I guess I'm better off starting a new game anyway.
(0009458)
Footkerchief   
2010-07-01 08:07   
oliver -- does that mean 0000219 can be reproduced without using Wine? If so, I'll reopen it. Thanks for investigating, by the way.
(0009484)
oliver   
2010-07-01 18:14   
Yes, I believe that both this bug and 0000219 have the same cause, a change in the load order of raws. That seems to be most commonly triggered by moving saves to/from Wine, but it could trigger in other ways too.
(0009509)
oliver   
2010-07-02 05:05   
Having dug into libgraphics a bit:

- POSIX systems (Linux/OSX) use glob(), which sorts results
- Win32 uses FindFirstFile()/FindNextFile(), which does not sort results
(0009511)
m4st0r   
2010-07-02 05:52   
(edited on: 2010-07-02 06:52)
I'm having strange reaction results without changing the OS. Linux only, no wine, but instead of bituminous coal magnetite is used and lignite reaction has vanished. Haven't changed the raw's either. Some days ago everything worked fine.
Chromite now has the "Make coke from Lignite" tag.

Could a kernel update change the sort results and produce this behavior? That's the only major system-change that happened in the last couple of days.
(0009527)
numBone   
2010-07-02 09:43   
I've been running in Windows from a shared drive on a Linux server. When I moved it to a shared drive on a Windows server, it seems to have caused this same problem. So it seems it still can be caused just by the servers having a different OS, even though I've been executing DF from the same workstation.
(0009551)
oliver   
2010-07-02 19:24   
There's a patch to libgraphics that makes the sorting of filenames explicit (and case-insensitive) at http://s3test.randomly.org/df/332/dflibgraphics-file-sorting.txt [^]

Only tested on Linux. I can't build/test on Win32 or OSX.
(0009554)
Footkerchief   
2010-07-02 19:47   
Reminder sent to: Baughn

Hey Baughn -- sounds like this is a fix you can add yourself. oliver posted a patch at 0000332:0009551
(0009568)
Baughn   
2010-07-03 05:03   
Toady says this shouldn't matter, but that something is obviously busted.

It'll get fixed at a deeper level than just sorting the files.
(0009571)
Toady One   
2010-07-03 05:21   
Yeah, I can get this to reproduce strictly within one version on Windows by changing the order of the files. However, the raw loader reports that the game is sorting them in with the proper, original order (the regions store the tokens of the individual objects, so file order should never matter). It seems there's some auxiliary information somewhere that's reacting badly to how the files are set up. My guess is that it is loading in the files, pre-calculating some reaction information, and then only doing the token sort afterward, but I have to check the order. If that's what it is, I just need to delay the pre-calc and everything should be fine all around. I can hope to preserve save compatibility (and get it working where it wasn't before), but no guarantees of course.
(0009774)
Toady One   
2010-07-08 03:59   
Going to mark this one of for 0.31.09. I fixed the erroneous pre-calcs I found, they would cause it, and I can no longer reproduce it after the fix. Hopefully it'll be good now.