Processing Percentage Done / Progress Bar for Large AJAX Data Sets?

matthawke

I am using DataTables to tabulate and process some ~5000 rows from a JSON data set. On IE it's starting to get slow, and I don't want to put people off using a new system just because it takes a few seconds to pop up. And it is only a few seconds, but during that time there's no indication of the progress of the processing - all appearances would suggest that the page just hangs while processing.

What I would like to do is have the "Processing..." message display a percentage and/or a progress bar as it processes. (In fact, the default "Processing..." box style would probably look great with a progress bar filling the box behind the text.)

I couldn't find any assistance in the API, so I took a look at the code (identified _fnProcessingDisplay) and made a change to test my understanding, as follows (v1.5.4 at line 1862):

[code]
/* Got the data - add it to the table */
for ( var i=0 ; i

allan

Hi Matt,

Interesting idea. I think the reason that you are seeing things slow down significantly, is because you are doing a _lot_ of DOM interaction there. Depending on how the browser tried to optimise this, it might well being doing a lot of page reflows, which is very expensive time-wise.

A small optimisation would be to only update the DOM every 1% difference, rather than every difference (which is going to be must less that 1% if you have 5000 rows). However, I suspect that this probably won't be optimal either, as this isn't the only place that does some heavy processing when dealing with data. For example the sorting can be quite expensive on processing time, as can the sorting column highlighting (I'd suggest disabling this with bSortClasses:false btw - you might notice a significant decrease in processing time with this off).

As such, there isn't a single place in DataTables where you could just add in a progress indicator, but rather there would need to be some profiling done to try and get it to be suitable "smooth" in it's reporting of the progress.

If the bSortClasses doesn't help much, have you considered using server-side processing? That can significantly speed things up since a database engine (which is optimised for sorting/filtering etc) will be doing the heavy lifting.

Regards,
Allan

matthawke

Thanks Allan, these are all good suggestions. I had already noted the performance bump from disabling bSortClasses in other tables, and I have even disabled sorting entirely on this table to get as much speed as possible. (I still use DataTables for its filtering, pagination and row manipulation features.)

I did some more investigation and discovered that it is not the rendering which takes the majority of the time, but the JSON download and processing (the JSON data is ~750KB and rising). So, in order to display a proper progress bar, I'd need to get download processing feedback from the getJSON procedure, which just isn't possible AFAIK...

Just as I type this, it occurs to me that I could manually perform the getJSON request outside of the DataTables init function, but grab it in four or five smaller chunks. This way I could give a progress update on complete of each JSON chunk. Although there would be more XHR loads, there would be more feedback to the user at an early stage of the load. Once I've got the JSON data loaded into a variable, it seems it would be a simple matter of using fnAddData() each time...

Will feed back any further developments! :)