big tables ... local cache?

bwlang

I'm now in a place where i need to do server side mode and I'm not looking forward to it.

I wonder if anyone has thought about caching up the whole table data in a json blob, but only showing a window of data instead of going back and forth to the server for each window.

Seems like that might be a huge speed advantage...
Maybe it's even possible to consume the dom based table into a json blob and eliminate all but the currently visible bits.

Is this a crazy idea?

I think this has some advantages over the Gears/html5 local storage bits discussed here:
http://datatables.net/forums/comments.php?DiscussionID=36

allan

With server-side processing some tables can deal with 20 million odd rows, so a local cache for the full thing wouldn't work (that would just be Ajax sourced data - which DataTables does do). What you can do with server-side processing though, is to have pipelining: http://datatables.net/examples/server_side/pipeline.html which is a cache of the subset of the results.

Allan

bwlang

I'm running into speed problems on my measly 200 records...
http://www.neb.com/polymerases
It's worse with 2000 records
http://polbase.neb.com/references
I have an internal page with 6000 rows that's getting close to unusable.

Maybe for table of size 0-20000 rows it's reasonable to try to scrape the table and modify a set of dom object for paging, filtering and sorting.


Or maybe I'm just "doing it wrong".

allan

Looking at your page in Webkit's "resources" tab, it shows that it's taking 4 seconds just to get your page (the second link). The table then rendered reasonably quickly (although obviously IE will take a fair bit longer). That's a long download time for the page - are your SQL queries perhaps running a bit slow? Possibly worth profiling them.

For client-side processing, regardless of if you are using ajax sourced data or DOM sourced data, 2000 rows is typically the point at which server-side processing becomes attractive (mainly because of IE - 10000 rows if you can discount IE).

Allan

bwlang

No sql, this should be straight out of memcache - though i'm not pre-priming the cache yet - you probably just primed it for me ;)

I'm going to experiment a bit with this json idea. I'll let you know if i come up with anything interesting.

allan

Have you looked at this example: http://datatables.net/examples/data_sources/ajax.html ? Is that basically what you are talking about?

Allan