UTF-8 Headache --
This is a discussion on UTF-8 Headache -- within the PHP Language forums, part of the PHP Programming Forums category; I have a function that (by fluke or whatever) used to work perfectly and seems to have changed behaviour on ...
|
|||||||
| FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
01-10-2006
|
|||
|
|||
UTF-8 Headache --
I have a function that (by fluke or whatever) used to work perfectly
and seems to have changed behaviour on me. The function was meant to take a string and convert it from have characters with diacritics to there non-diacritic equivalent. For example Dürer would become Durer -- except all of a sudden its becoming DA?rer. This is a problem :) The function and some sample HTML are below -- any clues or hints would be appreciated. I do see my extended character represented by the two -- I understand what has kinda happened I just dont know how to deal with it ... <?php function kill_diacritic ($word_string) { global $dbtype; if (empty($word_string)) { return $word_string; } else { $string_length = strlen($word_string); for ($x=0;$x<$string_length;$x++) { $ascii = ord(substr($word_string,$x,1)); switch($ascii){ case 224: // à case 225: // á case 226: // â case 227: // ã case 228: // ä case 229: // å $tmp = "a"; break; case 231: // ç $tmp = "c"; break; case 232: // è case 233: // é case 234: // ê case 235: // ë $tmp = "e"; break; case 236: // ì case 237: // í case 238: // î case 239: // ï $tmp = "i"; break; case 241: // ñ $tmp = "n"; break; case 240: // ð case 242: // ò case 243: // ó case 244: // ô case 245: // õ case 246: // ö case 248: // ø $tmp = "o"; break; case 154: // š $tmp = "s"; break; case 249: // ù case 251: // û case 252: // ü $tmp = "u"; break; case 253: // ý $tmp = "y"; break; case 158: // ž $tmp = "z"; break; case 192: // À case 193: // Á case 194: // Â case 195: // Ã case 196: // Ä case 197: // Å $tmp = "A"; break; /* // Oracle represents Æ as a ?. Not sure what MySQL will // Do with this character. Pretty sure nobody will ever // search using it but its there regardless. case 198: // Æ $tmp = "?"; break; */ case 200: // È case 201: // É case 202: // Ê case 203: // Ë $tmp = "E"; break; case 208: // Ð $tmp = "D"; break; case 204: // Ì case 205: // Í case 206: // Î case 207: // Ï $tmp = "I"; break; case 209: // Ñ $tmp = "N"; break; case 210: // Ò case 211: // Ó case 212: // Ô case 213: // Õ case 214: // Ö case 216: // Ø $tmp = "O"; break; case 138: // Š $tmp = "S"; break; case 217: // Ù case 218: // Ú case 219: // Û case 220: // Ü $tmp = "U"; break; case 159: // Ÿ case 221: // Ý $tmp = "Y"; break; } // switch if (!empty($tmp) or $tmp=="_") { $word_string = str_replace(chr($ascii),$tmp,$word_string); $tmp=""; } } // for } return $word_string; } ?> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>UTF8 Testing</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <link rel='stylesheet' href='styles/default/stylesheet.css' type='text/css'> <script type="text/JavaScript" src="javascript.js"></script> </head> <body> <form method="GET" action="index.php"> <input type="text" name="s" size="10" value="<?php echo $_GET['s']; ?>"> <input type="submit" value="Search"> </form> <p> <?php echo $_GET['s']; ?> </p> <p> <?php echo kill_diacritic($_GET['s']); ?> </p> </body> </html> 
|
|
01-11-2006
|
|||
|
|||
|
Re: UTF-8 Headache --
James wrote:
> I have a function that (by fluke or whatever) used to work perfectly > and seems to have changed behaviour on me. The function was meant to > take a string and convert it from have characters with diacritics to > there non-diacritic equivalent. For example D?rer would become Durer > -- except all of a sudden its becoming DA?rer. This is a problem :) > The function and some sample HTML are below -- any clues or hints would > be appreciated. I do see my extended character represented by the two > -- I understand what has kinda happened I just dont know how to deal > with it ... UTF-8 is a variable-length encoding. Accented characters are stored with two bytes so your code is not going to work. Instead of looping through the string manually, use the strtr() with an array as the translation table. Figuring out what the letters are in UTF-8 encoding won't be fun though. In general it's best to avoid using Unicode unless you're actually doing multi-lingual stuff.
|
|
01-11-2006
|
|||
|
|||
|
Re: UTF-8 Headache --
James wrote:
> I have a function that (by fluke or whatever) used to work perfectly > and seems to have changed behaviour on me. The function was meant to > take a string and convert it from have characters with diacritics to > there non-diacritic equivalent. For example Dürer would become Durer > -- except all of a sudden its becoming DA?rer. This is a problem :) > The function and some sample HTML are below -- any clues or hints would > be appreciated. I do see my extended character represented by the two > -- I understand what has kinda happened I just dont know how to deal > with it ... > > <?php > > function kill_diacritic ($word_string) { <snip> Imagine this: function translate($from_portuguese) { if ($from_portuguese == 'bom dia') return 'good morning'; } and now do echo translate('bonjour'); You won't expect that to work, will you? :) For the same reason that you can't accept French when you expect Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not work: All single-byte utf-8 characters are < 128; when a utf-8 character starts with 128 or greater you need two or more bytes to identify the character specified. http://www.w3.org/TR/html4/interact/...accept-charset The default value for this attribute is the reserved string "UNKNOWN". User agents may interpret this value as the character encoding that was used to transmit the document containing this FORM element. So, your web page was sent with utf-8 encoding (did you also configure your server for utf-8, or did you simply add the meta tag?), but there's no indication for what character sets you accept in forms. Maybe not all browsers interpret "UNKNOWN" as what is specified in the META tag, or even in a Content-Type HTTP header. Further reading: http://www.w3.org/International/tuto...rial-char-enc/ -- Mail to my "From:" address is readable by all at http://www.dodgeit.com/ == ** ## !! ------------------------------------------------ !! ## ** == TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>) may bypass my spam filter. If it does, I may reply from another address!
|
|
01-11-2006
|
|||
|
|||
|
|OT| Re: UTF-8 Headache --
Pedro Graca wrote:
<snip> > For the same reason that you can't accept French when you expect > Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not > work: <snip> Good explanation:-) Glad that you're back here. -- <?php echo 'Just another PHP saint'; ?> Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/
|
|
01-11-2006
|
|||
|
|||
|
Re: |OT| Re: UTF-8 Headache --
R. Rajesh Jeba Anbiah wrote:
> Pedro Graca wrote: >> For the same reason that you can't accept French when you expect >> Portuguese, accepting UTF-8 when you're expecting iso-8859-1 will not >> work: > > Good explanation:-) Thank you. The rest of the post was a mess though :) > Glad that you're back here. Thank you very much. It's good to be back. -- Mail to my "From:" address is readable by all at http://www.dodgeit.com/ == ** ## !! ------------------------------------------------ !! ## ** == TEXT-ONLY mail to the whole "Reply-To:" address ("My Name" <my@address>) may bypass my spam filter. If it does, I may reply from another address!
|
Linear Mode
