0006558: "Broken Unicode" warnings for some keybindings

Notes
(0025238) Khym Chanur (reporter) 2014-07-08 21:10	A list of characters which actually shows something: Broken unicode: [KEY:<81>] Broken unicode: [KEY:<8d>] Broken unicode: [KEY:<8f>] Broken unicode: [KEY:<90>] Broken unicode: [KEY:<9d>] Broken unicode: [KEY: ] Broken unicode: [KEY:¡] Broken unicode: [KEY:¢] Broken unicode: [KEY:£] Broken unicode: [KEY:¤] Broken unicode: [KEY:¥] Broken unicode: [KEY:¦] Broken unicode: [KEY:§] Broken unicode: [KEY:¨] Broken unicode: [KEY:©] Broken unicode: [KEY:ª] Broken unicode: [KEY:«] Broken unicode: [KEY:¬] Broken unicode: [KEY:] Broken unicode: [KEY:®] Broken unicode: [KEY:¯] Broken unicode: [KEY:°] Broken unicode: [KEY:±] Broken unicode: [KEY:²] Broken unicode: [KEY:³] Broken unicode: [KEY:´] Broken unicode: [KEY:µ] Broken unicode: [KEY:¶] Broken unicode: [KEY:·] Broken unicode: [KEY:¸] Broken unicode: [KEY:¹] Broken unicode: [KEY:º] Broken unicode: [KEY:»] Broken unicode: [KEY:¼] Broken unicode: [KEY:½] Broken unicode: [KEY:¾] Broken unicode: [KEY:¿] Broken unicode: [KEY:À] Broken unicode: [KEY:Á] Broken unicode: [KEY:Â] Broken unicode: [KEY:Ã] Broken unicode: [KEY:Ä] Broken unicode: [KEY:Å] Broken unicode: [KEY:Æ] Broken unicode: [KEY:Ç] Broken unicode: [KEY:È] Broken unicode: [KEY:É] Broken unicode: [KEY:Ê] Broken unicode: [KEY:Ë] Broken unicode: [KEY:Ì] Broken unicode: [KEY:Í] Broken unicode: [KEY:Î] Broken unicode: [KEY:Ï] Broken unicode: [KEY:Ð] Broken unicode: [KEY:Ñ] Broken unicode: [KEY:Ò] Broken unicode: [KEY:Ó] Broken unicode: [KEY:Ô] Broken unicode: [KEY:Õ] Broken unicode: [KEY:Ö] Broken unicode: [KEY:×] Broken unicode: [KEY:Ø] Broken unicode: [KEY:Ù] Broken unicode: [KEY:Ú] Broken unicode: [KEY:Û] Broken unicode: [KEY:Ü] Broken unicode: [KEY:Ý] Broken unicode: [KEY:Þ] Broken unicode: [KEY:ß] Broken unicode: [KEY:à] Broken unicode: [KEY:á] Broken unicode: [KEY:â] Broken unicode: [KEY:ã] Broken unicode: [KEY:ä] Broken unicode: [KEY:å] Broken unicode: [KEY:æ] Broken unicode: [KEY:ç] Broken unicode: [KEY:è] Broken unicode: [KEY:é] Broken unicode: [KEY:ê] Broken unicode: [KEY:ë] Broken unicode: [KEY:ì] Broken unicode: [KEY:í] Broken unicode: [KEY:î] Broken unicode: [KEY:ï] Broken unicode: [KEY:ð] Broken unicode: [KEY:ñ] Broken unicode: [KEY:ò] Broken unicode: [KEY:ó] Broken unicode: [KEY:ô] Broken unicode: [KEY:õ] Broken unicode: [KEY:ö] Broken unicode: [KEY:÷] Broken unicode: [KEY:ø] Broken unicode: [KEY:ù] Broken unicode: [KEY:ú] Broken unicode: [KEY:û] Broken unicode: [KEY:ü] Broken unicode: [KEY:ý] Broken unicode: [KEY:þ] Broken unicode: [KEY:ÿ]

(0027248) Quietust (reporter) 2014-07-22 14:48	These errors are technically also happening on Windows, though the messages don't seem to show up unless DFHack is installed (in which case they get dumped to stdout.log).

(0027255) lue (reporter) 2014-07-22 16:32	I finally got around to a small script that fixed the file, the results of which are pasted here: https://gist.github.com/lue/8fa0425f6bb806b18baf [^] . This is assuming, of course, that the keybindings are supposed to correlate with the first 256 codepoints of Unicode (instead of, say, CP 437), which appears to be the case. Interestingly, in addition to the broken unicode errors and the erroneous question marks, codepoint 123 was a } when it should've been a { . I wonder if anyone ran into confusing problems with that in-game :) .

(0027264) Footkerchief (manager) 2014-07-22 19:36	I'm confused, does this problem only happen when using DFHack, or does that just make the error messages visible?

(0027267) lethosor (manager) 2014-07-22 20:11 edited on: 2014-07-22 20:12	Happens with a clean installation on OS X and Linux (and Windows, according to comments above, although the messages only show up if DFHack is installed).

(0027273) lue (reporter) 2014-07-22 20:41	For me, the "Broken Unicode" messages are printed to stdout (_not_ stderr). I don't use any utilities. You can check out the interface.txt file in your installation of DF, and see that the STRING_A keybindings starting with STRING_A128 are either question marks (likely put there by some text editor handling unrecognized input) or raw byte values, as opposed to utf8-encoded codepoints. A123 and A124 were also set incorrectly, for whatever reason.

(0027956) tesoul (reporter) 2014-07-30 09:24	In linux, open the file interface.txt with gedit as ISO-8859-1 and save as UTF-8

(0027974) lue (reporter) 2014-07-30 12:24	tesoul: please note that resaving like that won't fix all the keybindings, just the ones where plain byte values managed to survive to the final file. The ones that were substituted with question marks, as well as the { character keybinding, would still be broken. Also, the file I posted is still valid for 0.40.05 (that is, running `diff` on the result of fixing 0.40.04's file and the result of fixing 0.40.05's led to no output)

Issue History
Date Modified	Username	Field	Change
2014-07-07 21:22	lue	New Issue
2014-07-08 21:10	Khym Chanur	Note Added: 0025238
2014-07-08 21:10	Khym Chanur	Issue Monitored: Khym Chanur
2014-07-22 14:48	Quietust	Note Added: 0027248
2014-07-22 16:32	lue	Note Added: 0027255
2014-07-22 19:36	Footkerchief	Note Added: 0027264
2014-07-22 20:11	lethosor	Note Added: 0027267
2014-07-22 20:11	lethosor	Assigned To	=> lethosor
2014-07-22 20:11	lethosor	Status	new => acknowledged
2014-07-22 20:11	lethosor	Note Edited: 0027267	View Revisions
2014-07-22 20:12	lethosor	Note Edited: 0027267	View Revisions
2014-07-22 20:41	lue	Note Added: 0027273
2014-07-25 05:35	Hello71	Issue Monitored: Hello71
2014-07-28 16:32	lethosor	Status	acknowledged => confirmed
2014-07-30 09:24	tesoul	Note Added: 0027956
2014-07-30 12:24	lue	Note Added: 0027974
2014-08-01 12:16	Toady One	Status	confirmed => resolved
2014-08-01 12:16	Toady One	Fixed in Version	=> Next Version
2014-08-01 12:16	Toady One	Resolution	open => fixed
2014-08-01 12:16	Toady One	Assigned To	lethosor => Toady One
2015-07-28 19:52	lethosor	Relationship added	related to 0003497