How to deal with gigantic data (think gmail with millions of msgs)

falcon

Say I have a stream of data, coming in through websockets. DataTables will display no more than 100 or so at a time, the rest will reside on the server. The data is real-time, so updates are coming in every few seconds.

How can I handle such a situation?

I'll only have to display a 100 or so messages, so I don't need (or want) to record any more in any client data structure or disk. When the client clicks 'next,' I need to request the right batch of of records from the server. This is not terribly difficult, in theory. What if the user filters on some words, then starts iterating through the results...a little more complex. Say the user adds a filter, now I need to handle incoming real-time messages and filter them as well.

What can DataTables do to help me in this situation. I can't use the pre-existing ajax functionality. The API looks very interesting. I can obviously add data to the table, is it my responsibility to remove data which can't be seen by the user (or does datatables figure that out?)? Is there a way for me to intercept the request "display records 1000 to 11000" so I can have my code request it from server and push the response into the table? Can I intercept filter or sort requests?

Look forward to any ideas others may have. I am basically trying to avoid building the whole thing on my own.
Thanks

theinternot

"Server-side processing - When dealing with large data sets (for example 20 million rows) the web-browser simply can't cope with the amount of processing that is required for DataTables. Instead it's a good idea to pass off the data crunching to a process specifically designed for that - namely an SQL database (or any other data source!). The server-side process will do all of the pagination, sorting, filtering etc, while DataTables will simply display the results and handle user interaction."