[solved]i have an encoding error for only 1 letter ...

[liraz]

well for ex. when i try to search an hebrew word i get good results thou 1 single letter in hebrew changes to a mark insted of the letter .... why only that letter is a problem ?

is there a way i could fix that problem ?

i added a picture so u will get to see the problem yourself :



thank u very much for your help in advance.

[Argos]

Your character set should be the right one for Hebrew. In the HEAD of your template index.php there is a line that says something like:

<meta http-equiv="Content-Type" content="text/html; charset=XXXXX" />

At the XXXX there is a little code, that you set in the Settings of WB Admin. Find out what you need to use: http://www.google.nl/search?hl=nl&source=hp&q=hebrew+character+set&meta=&aq=f&oq=

[liraz]

yes ofc i know that it set to utf-8 ..... suppose to work isn't it ?

[Argos]

Ah, I see your subject now, you used the word "encoding" so I should have known you knew that  

I suspect the problem is somewhere in the /search/ files. Maybe you should have a look at those. For example in search_modext.php there is this code:

Code:

case 'hebrew':
$mod_text = charset_to_utf8($mod_text, 'ISO-8859-8');
break;

I'm no coder, but maybe you should look into these files.

[liraz]

ty i did looked at it .... although that encoding looks fine ....

the thing that makes me wonder is actually that it only have a problem with one letter .... all the rest get out of the search function just fine ...

also in the rest of the website i get that letter just fine ... so some where in the process of it the search is killing my letter "נ"

i saw a file called search_convert.php and some options in it that changes English letter to German letter ...

could that make effect on Hebrew though ?

ty very much for your help Argos

[Argos]

I'm sorry, I'm afraid I can't help you. I'm no coder and also not into that character set stuff. But I'm pretty sure it must be something in those 4 files. Maybe some trial and error changes will result in a fix. Or someone else has a clue...

[kweitzel]

Did you change Charset after you created some of the content? I know this issue with some German special chars if you change from ISO to UTF after you have generated the content.

Another option could be, that you copied something out of a document encoded in a different charset which now can not be "converted". Try replacing this one character within the page.

Usually these 2 issues come up within the content as well. Your initial posts suggests that it is only within the search, right?

cheers

Klaus

[liraz]

Did you change Charset after you created some of the content? I know this issue with some German special chars if you change from ISO to UTF after you have generated the content.

Another option could be, that you copied something out of a document encoded in a different charset which now can not be "converted". Try replacing this one character within the page.

Usually these 2 issues come up within the content as well. Your initial posts suggests that it is only within the search, right?

cheers

Klaus

well Klaus ty for your respond,

i know those 2 issues also that's not it

i haven't changed my encode of the site, never ...
and the thing is its only on search not at all on the rest of the site ...

so it have to do some thing with the search function ... i write the string u can see at the picture at the the top of the post ... directly to the input field ... and press on search ... now the search function take my value and store it to a variable .... and then echo it back both in the input field and the results with the wrong encoding for that letter only ....

well i tried it with all the letters and that 1 is the only 1 that make a mess of it

i hope i helped u guys to help me on that case

hope to see your answer soon

Liraz

edit : i made another shoot for u m8 this time my search string contains all the hebrew alphabetic letters and this is the onlt letter having a problam :

[thorn]

Hello,

please open file search/search_modext.php, locate the lines (around line 300 for wb2.8 ) containing

Code:

} else {
    $mod_text = entities_to_umlauts($mod_text, 'UTF-8');
}

and replace by

Code:

} else {
    //$mod_text = entities_to_umlauts($mod_text, 'UTF-8');
}

i.e. comment the line out.

Does this help?


thorn.

[liraz]

Hello,

please open file search/search_modext.php, locate the lines (around line 300 for wb2.8 ) containing

Code:

} else {
    $mod_text = entities_to_umlauts($mod_text, 'UTF-8');
}

and replace by

Code:

} else {
    //$mod_text = entities_to_umlauts($mod_text, 'UTF-8');
}

i.e. comment the line out.

Does this help?


thorn.

hello thorn,

ty for your help, i tried your suggestion though its not helping.
btw could that problem come from mySQL DB ? 

[Argos]

Could it be the bug described here: http://www.websitebaker2.org/forum/index.php/topic,14920.0.html

[liraz]

Could it be the bug described here: http://www.websitebaker2.org/forum/index.php/topic,14920.0.html

i hoped that would maybe fix my bug here ... but i'm sad to say that even after i changed the file i have that same problem ...

why would it show every single letter in hebrew just fine all but one single letter ... it doesn't make any sense at all ...

by the way after i will fix that nasty bug i will be glad to upload all the work i have done to make it go right to left and the hebrew translation files ofc

[Argos]

Damn!

If it is a letter that can be replaced by one that is somewhat similar, maybe you could add an automatic replacement in the search_convert.php file, like is done with a lot of German letters.

[Argos]

Here is a thread that may hold a solution:
http://www.websitebaker2.org/forum/index.php/topic,8883.0.html


By the way: do you use WB2.8? And does the problem occur in other templates as well?

[liraz]

mmmm i just found out the easy way to pass that bug ... i just changed all my site encode to a more native hebrew encoding ... well ofc it turned into jibrish but ... the search function got the string and returned it just fine ..

so this is one easy choice to any one who want he's site on other charset ..

the problem is i want it on UTF-8 hehe

now i saw something in the search_modext.php file that Argos mentioned before ...

Code:

case 'hebrew':
$mod_text = charset_to_utf8($mod_text, 'ISO-8859-8');
break;

well i looked over the UTF8 convert function at the functions-utf8.php file ... well too complicated for me hhh but i saw torn made that function ...

now i think something is not going well there by converting from the DB to UTF-8 ....

if any 1 know what is going on there i'll be glad to know hhh

edit : Argos yes i use WB 2.8 and WB 2.7 both have the same problem in every template available both hebrew and english

edit2 : well i read that post ... the thing is i'm just using the default allcss template ... and it has the meta :

Code:

<meta http-equiv="Content-Type" content="text/html;
           charset=<?php if(defined('DEFAULT_CHARSET')) { echo DEFAULT_CHARSET; } else { echo 'utf-8'; }?>" />

and all my translations r using utf-8 also ....  another thing to think about is that not all of the input comes out wrong .. only one letter ... is there any thing i can do to translate that letter to a code or some thing like we do to "/" for example ?

[liraz]

well i found that post in hebrew about the same problem on wordpress there its just been fixed so i google translated it for u here :

http://translate.google.co.il/translate?js=y&prev=_t&hl=iw&ie=UTF-8&layout=1&eotf=1&u=http%3A%2F%2Fmarksw.com%2Fwordpress%2F%3Fp%3D1785&sl=iw&tl=en

by referring to the letter "F" they mean the hebrew letter "נ" .... it translated it hehe

now if it helps i also found out that the referring value of that letter is :

F0 for ASCII
05E0 for Unicode

does that help us getting closer to the fix ?

[thorn]

Hello,

Argos point me in the right direction, thanks!

If you look carefully on your second image, you will realise that there is added some extra space between the Mem and the Samekh (aside the question mark). That is, the Nun (\xd7\xa0) is wrongly transformed to \xd7 (which is garbage in utf-8) and &nbsp; (which appears to be \xa0 in ISO8859-1, but NOT in UTF-8).
I don't know why this happens, yet.

Please inspect the HTML-source of that search-result page to figure out what exactly the result of that faulty transformation was.


EDIT: Please download this script http://help.websitebaker.org/media/common/wb_analyse_script.txt and put that text into a code section. It will perform some test that may help to figure out what's going on.


thorn.

[Argos]

Hi Thorn, I just tried that little test script out of curiousity (I'm not the question poster), and see what it does. Looks like a useful script, but it seems to be outdated. It talks about WB 2.6.7 

[liraz]

yes thorn u r exactly right ofc !

i read some thing about that just a few minutes ago, though i don't know any thing about that stuff

i uploaded the html source of the search result page in here.

but i did not understand what to do with that script u gave me .. sry could u explain pls in what code section should i put that ?

[Argos]

Liraz: create a new page and use type Code (instead of WYSIWYG). Put the code in that page, and see the output on your site.

[liraz]

got u now here is my result :

Code:

script-version: 2008.01.23

Checking mySQL
version : 5.1.32-community-log
version_comment : MySQL Community Server (GPL)
version_compile_machine : ia32
version_compile_os : Win32
Checking mySQL character-sets
character_set_client : latin1
character_set_connection : latin1
character_set_database : utf8
character_set_filesystem : binary
character_set_results : latin1
character_set_server : latin1
character_set_system : utf8
character_sets_dir : c:\wamp\bin\mysql\mysql5.1.32\share\charsets\
wb28 : CREATE DATABASE `wb28` /*!40100 DEFAULT CHARACTER SET utf8 */
settings : CREATE TABLE `settings` ( `setting_id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) NOT NULL DEFAULT '', `value` text NOT NULL, PRIMARY KEY (`setting_id`) ) ENGINE=MyISAM AUTO_INCREMENT=49 DEFAULT CHARSET=latin1
sql_mode :
Checking PHP
Version: 5.2.9-1
Server API: apache2handler
default_charset:
magic_quotes_gpc:
magic_quotes_runtime:
max_execution_time: 30
upload_max_filesize: 2M
post_max_size: 8M
file_uploads: 1
memory_limit: 128M
max_input_time: 60
loaded extensions: bcmath calendar com_dotnet ctype session filter ftp hash iconv json odbc pcre Reflection date libxml standard tokenizer zlib SimpleXML dom SPL wddx xml xmlreader xmlwriter apache2handler gd mbstring mysql mysqli PDO pdo_mysql pdo_sqlite
safe_mode:
disable_functions:
session_support: OK -- SID content:
session.use_cookies: 1
session.use_only_cookies: 0
error_reporting: 0
display_errors: 1
Checking WB:
WB_VERSION: 2.8
VERSION: 2.8
DEFAULT_CHARSET: utf-8
OPERATING_SYSTEM: linux
Checking Apache
Apache/2.2.11 (Win32) PHP/5.2.9-1

========================================

    * ERROR: This is not WebsiteBaker 2.6.7! Solution: update to WebsiteBaker 2.6.7
    * WARN: extension 'curl' missing -- you can't use 'Curl Module'. Solution: use a different module.
    * Notice: upload_max_filesize < 8MB. You can't upload large files or pictures. Solution: try to add a .htaccess-file with "php_value upload_max_filesize 8388608"

Note: Explanation of terms --
ERROR: To make use of websitebaker, you have to change the server-configuration!
WARN: You can use websitebaker, but special attention is needed.
NOTICE: You can use WebsiteBaker without harm, but there may some odd behavior

Max memory-usage while processing this script: 2MByte

edit : also added it as a text file.

[thorn]

Hello,

Code:

character_set_database : utf8

wb28 : CREATE DATABASE `wb28` /*!40100 DEFAULT CHARACTER SET utf8 */

settings : CREATE TABLE `settings` ( `setting_id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(255) NOT NULL DEFAULT '',
   `value` text NOT NULL, PRIMARY KEY (`setting_id`) ) ENGINE=MyISAM AUTO_INCREMENT=49 DEFAULT CHARSET=latin1

This is the only uncommon issue I noticed so far.
The database defaults to utf8, but the tables were created using latin1.
But I can't see how this should affect the search-results page only ...

BTW: The guys from IP.Board solved this Problem back in July 2009 http://community.invisionpower.com/index.php?app=tracker&showissue=16241 , unfortunately it's closed source.

Have to look closer at this issue tomorrow.


thorn. 

[liraz]

yes i found that post too. i also found some more stuff about it in hebrew though ...

its a known bug of php function Preg_replace () and that hebrew letter ...

i read some where that the hebrew letter "נ" have the same referring code as "/n" at utf-8 encode ...

if u will understand some thing out of the post i put on google translate for u they say there some problematic attributes for that function ...

ty very much for all your help thorn i will look forward to your answer

edit : btw the guys at wordpress and phpbb also faced that issue and fixed it ... if it helps ..

[thorn]

Hello,

we figured out that under special circumstances (Windows?, special PCRE-version?), the regex "\s"-character-class will match all appearing "\xa0", too. so, a regex like in

Code:

$str = preg_match('/\s+/',' ', $str);

to remove all unnecessary whitespace in a string, will corrupt some characters, e.g. the Hebrew "Nun": "\xd7\xa0" will become "\xd7 ".
 
BTW: This contradicts PCRE's documentation, which stated explicitly that U+00A0 is NOT included in the \s-character-class (but in \h).


Ticket filed: http://project.websitebaker2.org/ticket/915


thorn.

[liraz]

ty very much for all your help on this issue thorn and Argos

marked [solved]

liraz