JSON encoding error: Malformed UTF-8 characters, possibly incorrectly encoded

rf1234

This is the error I am getting.

I am using this version of Data Tables:
https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js

I use Editor with PHP, version is 2.0.7

The browser's console shows the following error:

The problem occurs when PHP Editor tries to json_encode the data object. Editor crashes and I get the first error message above.

I checked on SO how to fix this and found this:
https://stackoverflow.com/questions/46305169/php-json-encode-malformed-utf-8-characters-possibly-incorrectly-encoded

Hence I added this on "postGet" and tried both versions (looping through $data myself or let PHP do it):

->on( 'postGet', function ( $e, &$data, $id ) { 
//        foreach ($data AS $key => $val) {
//            $data[$key] = mb_convert_encoding( $val, "UTF-8", "UTF-8" );
//        }
    $data = mb_convert_encoding( $data, "UTF-8", "UTF-8" );
})
->process($_POST);    
$editor->json();

In my debugger the $data object looked 100% fine and the mb_convert_encoding did not produce any errors in either version.

$editor->json();

caused the crash.

How can I locate the problem? The console message doesn't allow me to locate anything: It is a location in a min-file. I wanted to the use the regular js file
https://cdn.datatables.net/1.11.5/js/jquery.dataTables.js
Surprisingly that produced errors in other parts of my code when executing. Strange.

This is all I get in the network tab as a server response. Pretty much useless unfortunately.

I also tried to repair the database table. But there was nothing wrong with it.

I am stuck now and this is very, very urgent for me to fix: I cannot onboard new clients before this is fixed! I would be very grateful if you could help me with this.

rf1234

By the way the exact same PHP Editor works perfectly fine when used in a different page. Hence I tend to believe that the error message is not meaningful. But of course I don't know...

rf1234

Just checked this in my debugger:

->on( 'postGet', function ( $e, &$data, $id ) { 
//        foreach ($data AS $key => $val) {
//            $data[$key] = mb_convert_encoding( $val, "UTF-8", "UTF-8" );
//        }
    $test = json_encode($data);
    $data = mb_convert_encoding( $data, "UTF-8", "UTF-8" );
})

json_encode($data) produces a valid result on both pages.

But:

$test = $editor->json(false, JSON_INVALID_UTF8_SUBSTITUTE);

produces a valid result only in one case!! In the other case the result is "false"!
I have absolutely no clue what causes this, but it is the cause of the problem.

@allan : I guess you will be the only one who can help me with this ... Thanks.

rf1234

->on( 'postGet', function ( $e, &$data, $id ) { 
    if ( ! isset($_SESSION['govCredUserId']) ) { //it is the admin page
        $_SESSION["ctr_inst_json"] = '{"data":' . json_encode($data) . '}';
    }
})
->process($_POST);
if ( isset($_SESSION['govCredUserId']) ) { //not the admin page
    $editor->json();
} else {
    echo $_SESSION["ctr_inst_json"];
    unset($_SESSION["ctr_inst_json"]);
}

This is my work around for the time being. It produces valid json that works on both of my pages. So something odd must happen in Editor between "postGet" and "->json()". The work around only works for reading, not for editing ... Then you get a crash because Editor doesn't seem to execute "postGet".

Hopefully someone can find out what it is.

rf1234

I found the problem: Unlike in the user table I only used the first initial of the firstname for rendering the options like this in the installation table:

Mjoin::inst( 'user' )
    ->link( 'ctr_installation.id', 'ctr_installation_has_principal.ctr_installation_id' )
    ->link( 'user.id', 'ctr_installation_has_principal.user_id' )
    ->order( 'user.lastname asc' )
    ->fields(
        Field::inst( 'id' )->set( false )                
            ->options( Options::inst()
                ->table('user')
                ->value('id')
                ->label( array('acad', 'lastname', 'firstname', 'email') )
                ->render( function ( $row ) {   
                    if ( $row['lastname'] <= '' ) {
                        return 'Incognito';
                    }
                    if ($row['acad'] > '') {
                        return $row['lastname'] . ', ' . $row['acad'] . ' '. mb_substr($row['firstname'],0,1) . '. (' . $row['email'] . ')';  
                    } else {
                        return $row['lastname'] . ', ' . ' '. mb_substr($row['firstname'],0,1) . '. (' . $row['email'] . ')';
                    }
                } )

Right now I am using "mb_substr". That returns a proper "ö" if you use the substring with a length of 1. With "substr" only garbage is returned because "ö" is a multi-byte character and a "substr" of 1 is nothing meaningful.

return $row['lastname'] . ', '. mb_substr($row['firstname'],0,1) . '. (' . $row['email'] . ')';

This was my hardest bug ever. Happy to be done now.

allan

Yeah - that's a cracker. Good to hear that you managed to track down the issue and get it resolved. Thanks for the updates.

Allan

rf1234

Thanks, Allan. My second last update was rubbish though ...

I didn't know that the $data - object does not return mjoined results on "postGet". Hence I had no issues json_encoding $data.

When I noticed the above I started looking at the mjoined tables: I subsequently deleted all new entries from the user table one by one - always checking whether it worked or not. Until I got to the "Umlaut"-record which was created yesterday afternoon.

Then I thought I just mustn't allow German Umlaute as first chars in "firstname" which turned out to be the wrong idea again... Eventually I found the multi-byte substring issue.

Now I am going to check all of my code to indentify similar cases. Don't want this to ever happen again

Long journey!