0000332: Raw files parsed in strict alphabetic order in Linux but not Windows, causing ID mismatches for saves moved between systems

Notes
(0005113) Zet (reporter) 2010-04-26 07:22	I can confirm this. It has happened to me in two different forts now. Bituminous coal and lignite are red in the stones screen by default, but they don't list the "make coke" ability, and the smelters do not get the "make coke" menu option, even though loads of the resource are available.

(0005114) DoctorZuber (reporter) 2010-04-26 07:25	check (z) stone and see if it has changed to green (which it can do if you find a lot). I'm wondering if this also moves it in the stockpile now as well.

(0005138) Footkerchief (manager) 2010-04-26 10:31	oliver and Zet: what OS are you using?

(0005196) oliver (reporter) 2010-04-26 15:52	In my case this was the same save that was transferred from XP to Linux/Wine and suffered from the disappearing reaction problem as described in 0000219, so it's quite likely related.

(0005539) CptFastbreak (reporter) 2010-04-28 09:16 edited on: 2010-04-28 09:16	Cross-posting this from 0000219, as I've been told that it's an occurence of this bug. OS: Windows XP Professional 32Bit I've had the same occurence with Bituminous Coal and Lignite: - They're still red in the z stone screen. - They don't list the 'make coke' ability. - They are in 'ordinary stone' in the stockpile menu. - It's possible that making coke still worked for some time as I only noticed this when my repeating orders got cancelled. - Now I can add the jobs via manager, but it states 'Urist McSmelter cancels Make Coke from Lignite: needs rock' and cancels. - I got a few reoccurring crashes afterwards that seemed to stop when I cancelled the manager coke orders, so ordering coke might eventually lead to crashes. I'm not positive on this one though. - I can still make Pig Iron and Steel from charcoal coke. - Saving and reloading did not help, smelter is not in a burrow, save was never transferred.

(0005568) Footkerchief (manager) 2010-04-28 12:57	CptFastbreak: do you have anything interesting in your errorlog? Also, it might be helpful to upload that save to http://dffd.wimbli.com/ [^]

(0005603) CptFastbreak (reporter) 2010-04-28 16:13	Here's the save file: http://dffd.wimbli.com/file.php?id=2261 [^] The errorlog doesn't contain anything related to the smelter or to coke, just a bunch of "path fail" and "loop path fail" mostly related to the construction of buildings and the fleeing from invaders.

(0007711) Eskarion (reporter) 2010-06-05 03:07	I encountered this bug after transfering a savegame made with 31.04 - legacy version to 31.05 - sdl-version (tried linux as well as win). Using the save with the 31.05 - legacy version works fine. I also created a new world with the 31.05 sdl - version (win) and here the reactions listed in the z-stone menu are correct, although most of the ores now show the ability to make sharp blades while Obsidian isn't listed at all.

(0008257) Phancy Physicist (reporter) 2010-06-13 01:49	I also encounter this bug. I used wine before and loaded my save to linux and now I can't make coal. Interestingly, if I save and load into wine (new and old windows versions) the save game works fine. When I go back to linux and it is broken again. So i guess there is a linux-only bug here.

(0008391) Phancy Physicist (reporter) 2010-06-13 23:59	So one way to work a round this is to use links "ln -s path_to_linux_df/data/save/ path_to_win_df/data/save" then just open with wine queue up coal creation then exit open in linux and the coal creation will start up. Not a great solution but works for now.

(0008420) jeti0 (reporter) 2010-06-14 12:14	Actually, not only the coke creation tasks are missing from the smelters. All make <alloy> from <bars/ore> tasks are missing, too. This happens when trying to use Wine generated world on native Linux DF and vice versa.

(0009206) oliver (reporter) 2010-06-28 01:49 edited on: 2010-06-28 01:55	I have managed to work out how to reproduce this. Someone may want to reopen 0000219 with this info, since it's more directly relevant there. As suggested in a comment on 0000219, it is to do with the order in which the raws are loaded. I reproduced this on Linux by the following steps: * Create a new region, embark * Verify that bituminous coal has the right reactions. * Save the region. * Run 'ls -lU \| grep inorganic' in data/save/regionN/raw/objects. -U provides an "unsorted" list of files that will reflect the order that the OS-level directory-scanning APIs return. In my case I get this order: inorganic_other.txt inorganic_stone_layer.txt inorganic_stone_mineral.txt inorganic_stone_soil.txt inorganic_metal.txt inorganic_stone_gem.txt * Do something to inorganic_stone_mineral.txt (which contains the bituminous coal definition) so it is in a different position in the order. It will be very OS and filesystem specific what you need to do here. In my case, on my Linux install, renaming it to "inorganic_test.txt" worked, producing this order: inorganic_other.txt inorganic_stone_layer.txt inorganic_stone_soil.txt inorganic_test.txt inorganic_metal.txt inorganic_stone_gem.txt * Load the saved region, check reactions, note that bituminous coal is missing reactions and is considered an "other" stone in the stockpile lists. * Save * rename the raw file back to its original position * Reload * The reactions are fixed. I'm guessing there is an ID stored internally somewhere in the region save that is just an index into the stone types loaded from the raws, and that index can end up pointing somewhere different depending on the order the raws are loaded. So moving between Wine/Linux/Windows etc, or just bad luck with what the filesystem does under the covers, has a chance of changing the ordering of the raws and causing chaos. Fun times! Possible fix: sort the raw filenames before loading them, since that should be a stable order regardless of what the filesystem does. But that will break old saves :( (Might be worth alerting Toady to this one, since it has the potential to cause all sorts of savegame problems - backups that don't work, can't transfer saves between different systems, savegames mysteriously going bad for no apparent reason, etc)

(0009325) uggi (reporter) 2010-06-29 14:53	I'm having this bug as well. I started the game under Wine. Later on I switched to Windows, mainly because the game ran so much faster under Windows. I moved the game files to Windows partition, and also upgraded the game a few times. I think I started with 31.03, running 31.08 now. Then I noticed I can't make coke from bituminous coal. I can use the manager noble to add the job, but the dwarfs just cancel it, saying "..cancels Make coke from bituminous coal: Needs melanite." I tried running the game under Wine again. I first copied the game files back to the Linux partition. Now the smelter shows "Make coke from bituminous coal" and "Make coke from lignite" options, which were not available under Windows. However, as the game files have been moved from one partition to another and then copied back again, I'm not sure what the original order was when I started the game. The Linux partition is in ext3 format, and Windows partition is in NTFS format. This is the order inorganic* files are now listed in the Linux partition: inorganic_stone_mineral.txt inorganic_stone_layer.txt inorganic_other.txt inorganic_stone_soil.txt inorganic_metal.txt inorganic_stone_gem.txt Now if I run the native Linux version with the exact same save files, linked from the Wine directory so they should be listed in the same order, I'm again unable to create coke. Also if I try running the game straight from the Windows partition under Wine, I'm unable to create coke again. The inorganic* files are listed alphabetically in that partition. In short: the only way to create coke now is to run the game under Wine from the Linux partition.

(0009456) uggi (reporter) 2010-07-01 07:55	If I rename those files, for example inorganic_stone_mineral.txt to inorganic_00_stone_mineral.txt (so it comes first in alphabetic order) I can then create coke in Windows too. But, looking at the stone data in the z-stone-menu reveals that everything else is borked too. I could for example make bismuth bronze bars from cinnabar or brimstone. Everything in that screen changes depending on the order I rename those files. So there's like ..umm.. 720 (?) different combinations how the inorganic* files can be ordered? I can't find the correct order, so I guess I'm better off starting a new game anyway.

(0009458) Footkerchief (manager) 2010-07-01 08:07	oliver -- does that mean 0000219 can be reproduced without using Wine? If so, I'll reopen it. Thanks for investigating, by the way.

(0009484) oliver (reporter) 2010-07-01 18:14	Yes, I believe that both this bug and 0000219 have the same cause, a change in the load order of raws. That seems to be most commonly triggered by moving saves to/from Wine, but it could trigger in other ways too.

(0009509) oliver (reporter) 2010-07-02 05:05	Having dug into libgraphics a bit: - POSIX systems (Linux/OSX) use glob(), which sorts results - Win32 uses FindFirstFile()/FindNextFile(), which does not sort results

(0009511) m4st0r (reporter) 2010-07-02 05:52 edited on: 2010-07-02 06:52	I'm having strange reaction results without changing the OS. Linux only, no wine, but instead of bituminous coal magnetite is used and lignite reaction has vanished. Haven't changed the raw's either. Some days ago everything worked fine. Chromite now has the "Make coke from Lignite" tag. Could a kernel update change the sort results and produce this behavior? That's the only major system-change that happened in the last couple of days.

(0009527) numBone (reporter) 2010-07-02 09:43	I've been running in Windows from a shared drive on a Linux server. When I moved it to a shared drive on a Windows server, it seems to have caused this same problem. So it seems it still can be caused just by the servers having a different OS, even though I've been executing DF from the same workstation.

(0009551) oliver (reporter) 2010-07-02 19:24	There's a patch to libgraphics that makes the sorting of filenames explicit (and case-insensitive) at http://s3test.randomly.org/df/332/dflibgraphics-file-sorting.txt [^] Only tested on Linux. I can't build/test on Win32 or OSX.

(0009554) Footkerchief (manager) 2010-07-02 19:47	Reminder sent to: Baughn Hey Baughn -- sounds like this is a fix you can add yourself. oliver posted a patch at 0000332:0009551

(0009568) Baughn (manager) 2010-07-03 05:03	Toady says this shouldn't matter, but that something is obviously busted. It'll get fixed at a deeper level than just sorting the files.

(0009571) Toady One (administrator) 2010-07-03 05:21	Yeah, I can get this to reproduce strictly within one version on Windows by changing the order of the files. However, the raw loader reports that the game is sorting them in with the proper, original order (the regions store the tokens of the individual objects, so file order should never matter). It seems there's some auxiliary information somewhere that's reacting badly to how the files are set up. My guess is that it is loading in the files, pre-calculating some reaction information, and then only doing the token sort afterward, but I have to check the order. If that's what it is, I just need to delay the pre-calc and everything should be fine all around. I can hope to preserve save compatibility (and get it working where it wasn't before), but no guarantees of course.

(0009774) Toady One (administrator) 2010-07-08 03:59	Going to mark this one of for 0.31.09. I fixed the erroneous pre-calcs I found, they would cause it, and I can no longer reproduce it after the fix. Hopefully it'll be good now.

Issue History
Date Modified	Username	Field	Change
2010-04-03 17:27	oliver	New Issue
2010-04-26 07:22	Zet	Note Added: 0005113
2010-04-26 07:25	DoctorZuber	Note Added: 0005114
2010-04-26 10:31	Footkerchief	Note Added: 0005138
2010-04-26 10:46	Footkerchief	Relationship added	child of 0000219
2010-04-26 15:52	oliver	Note Added: 0005196
2010-04-28 09:16	CptFastbreak	Note Added: 0005539
2010-04-28 09:16	CptFastbreak	Note Edited: 0005539	View Revisions
2010-04-28 09:28	CptFastbreak	Issue Monitored: CptFastbreak
2010-04-28 12:57	Footkerchief	Note Added: 0005568
2010-04-28 16:13	CptFastbreak	Note Added: 0005603
2010-05-07 10:33	Footkerchief	Relationship replaced	related to 0000219
2010-05-16 23:52	Footkerchief	Summary	Coal / lignite mysteriously move to "other" stone type => Bituminous coal / lignite mysteriously move to "other" stone type
2010-05-16 23:54	Footkerchief	Tag Attached: Save Included
2010-06-05 03:07	Eskarion	Note Added: 0007711
2010-06-13 01:49	Phancy Physicist	Note Added: 0008257
2010-06-13 23:59	Phancy Physicist	Note Added: 0008391
2010-06-14 12:09	jeti0	Issue Monitored: jeti0
2010-06-14 12:14	jeti0	Note Added: 0008420
2010-06-28 01:49	oliver	Note Added: 0009206
2010-06-28 01:51	oliver	Note Edited: 0009206	View Revisions
2010-06-28 01:55	oliver	Note Edited: 0009206	View Revisions
2010-06-28 02:48	oliver	Issue Monitored: oliver
2010-06-29 14:53	uggi	Note Added: 0009325
2010-07-01 07:55	uggi	Note Added: 0009456
2010-07-01 08:07	Footkerchief	Note Added: 0009458
2010-07-01 08:08	Footkerchief	Summary	Bituminous coal / lignite mysteriously move to "other" stone type => Bituminous coal / lignite mysteriously move to "other" stone type, due to raw files not being parsed in strict alphabetic order
2010-07-01 18:14	oliver	Note Added: 0009484
2010-07-01 20:35	Footkerchief	Relationship replaced	has duplicate 0000219
2010-07-01 20:36	Footkerchief	Summary	Bituminous coal / lignite mysteriously move to "other" stone type, due to raw files not being parsed in strict alphabetic order => Raw files not parsed in strict alphabetic order in Linux, causing ID mismatches for saves moved between systems
2010-07-01 20:36	Footkerchief	Sticky Issue	No => Yes
2010-07-02 05:05	oliver	Note Added: 0009509
2010-07-02 05:24	Footkerchief	Summary	Raw files not parsed in strict alphabetic order in Linux, causing ID mismatches for saves moved between systems => Raw files parsed in strict alphabetic order in Linux but not Windows, causing ID mismatches for saves moved between systems
2010-07-02 05:52	m4st0r	Note Added: 0009511
2010-07-02 06:52	m4st0r	Note Edited: 0009511	View Revisions
2010-07-02 08:31	jeti0	Issue End Monitor: jeti0
2010-07-02 09:43	numBone	Note Added: 0009527
2010-07-02 19:24	oliver	Note Added: 0009551
2010-07-02 19:45	Footkerchief	Tag Attached: SDL-only
2010-07-02 19:47	Footkerchief	Issue Monitored: Baughn
2010-07-02 19:47	Footkerchief	Note Added: 0009554
2010-07-03 05:03	Baughn	Note Added: 0009568
2010-07-03 05:21	Toady One	Note Added: 0009571
2010-07-03 05:21	Toady One	Assigned To	=> Toady One
2010-07-03 05:21	Toady One	Status	new => acknowledged
2010-07-05 05:07	uggi	Issue Monitored: uggi
2010-07-08 03:59	Toady One	Note Added: 0009774
2010-07-08 03:59	Toady One	Status	acknowledged => resolved
2010-07-08 03:59	Toady One	Fixed in Version	=> 0.31.09
2010-07-08 03:59	Toady One	Resolution	open => fixed
2010-07-09 21:06	oliver	Issue End Monitor: oliver
2010-07-12 12:53	Footkerchief	Sticky Issue	Yes => No
2010-07-26 07:17	uggi	Issue End Monitor: uggi