Performance to load big json file

panpansh

Hello to everyone,
My pages, table etc ... work fine ;) and thank you for this.
The problem is just the load time :(
I have a client side processing only. This is my restriction.
I have a big json file to complete a table : about 26Mo ... 200.000 rows ...

the table look like this :
[code]
var oTable2 = $('#Params').dataTable( {
"bLengthChange": false,
"iDisplayLength": 50,
"bDeferRender": true,
"bPaginate": true,
"sPaginationType": "full_numbers",
"sScrollY": $('#container').height()-235,
"bProcessing": true,
"bJQueryUI": true,
"sDom": '<"H"<"toolbar2">lfr>t<"F"ip>',
"sAjaxSource": "sources/Params.json",
"aoColumns": [
{ "mDataProp": "parentIdjson", "bSearchable": false },
{ "mDataProp": "idjson", "bSearchable": false },
{ "mDataProp": "paramType" },
{ "mDataProp": "paramID" },
{ "mDataProp": "value" }
] // "bVisible": false, => parentIdjson and idjson
});
$("div.toolbar2").html('Parameters');
oTable2.fnSetColumnVis( 0, false ); // REMOVE TO DISPLAY PARENTID
oTable2.fnSetColumnVis( 1, false ); // REMOVE TO DISPLAY ID
[/code]

The load time is about 16.000ms.
I am looking for a solution ... maybe cut json to x*1.000 rows and load on background or something like this but I don't know to insert it into table ... and ... I don't know if it's a good solution.
I'm interested if you have ideas.
Thanks

allan

200k rows? You might want to consider using server-side processing: http://datatables.net/faqs#speed . Simple fact is that sending 200k rows down the wire is going to take a bit of time, so server-side processing is, I would say, an attractive option.

Allan

panpansh

Ok thanks, you think is not possible to improve performance by using multiple json ? or other solution ?
If my tutor for my stage don't wan't a server side processing ? :)

allan

It is always possible to improve performance, but I'd be surprised if it can be improved to the point where the performance for 200k rows is acceptable. How big is your JSON file? That's going to take a fair bit of time down the wire, never mind anything else.

Allan

timtucker

Not sure if it's an option (or you're using IE to begin with), but if you're using IE 8 and have some control over the client browser you might want to look into Firefox or Chrome.

When did some performance testing in the past I found a pretty dramatic difference in speed between IE8 and Firefox when rendering large tables (multiple orders of magnitude for anything > 10k rows).

panpansh

allan : my json => about 26Mo... :(
in addition to my question, it would not come possible to avoid downloading the json by the browser ? because the json file is located next to the index.html on the client ... (this is a report generated on the client machine and open locally by the client).

timtucker : I use Firefox and Chrome ... and it's better with chrome ... I don't know why ...

allan

26 megabytes 8-). That's a big a** JSON file. My internet connection would take about 5 minutes to download that, never mind the amount of time needed by the browser to parser it, sort it etc.

I think trying to get that to work in the browser would be barking up the wrong tree entirely. Have you considered server-side processing? That will handle millions of rows no problem since all the work is done by the SQL engine on the server, and the client only downloads the rows that it needs.

Allan

panpansh

I think you misunderstood the point about the json file download.
Context: my html page is locally on the client computer directly with the json file. Theoretically I do not need to download the json file via the browser ... That would save me time ...
Question: Is it possible to read directly into my json file locally without the browser download the json ?

all is in local and the browser too ...

allan

Okay - so local helps - however, 26MB is a lot of data, and I don't think you'll ever get good performance out of the browser with that amount of data. As a very simply test you could read in the file and then try tot he sort the data in it as a simple array. That's the absolute baseline performance that you will get.

Allan