Client-side / Server-side hybrid render

GregP

Description of problem: A long time ago, we switched from server-side processing to client-side. DT.net is performant, even with tens of thousands of records! However, our back end has increasingly complex database queries, and we have finally hit a scalability issue requiring us to use server-side pagination.

Therein lies the problem. Our data comes in from a web worker on a "message" channel, which is in turn given to an emitter, based on the type of data. Let's say for example, I have "users" data, I will emit "newUsersAvailable" when back end broadcasts and update. A subscriber listens to "newUsersAvailable" and when it happens, can either initialize a DataTable (once) or update the table (subsequent). Gone are the days of Ajax polling! It's lovely in general when I have a whole data set. It comes in, I use API to add the rows, call a draw on it, suppress pagination update, and we're off to the races!

But now that I can get the back end to send me paginated payloads (including all the relevant params like totalRecords and start record and all that), I have no way to feed it to DataTables in a "client-side" way. It only cares about the data. Or am I wrong? Can I give it pagination/search/filter values and tell it to update the rendered view?

Ajax override function to the rescue, right? Unfortunately no. The emitter is completely async and completely without a way to verify that a given payload is a "response" to the update requested. In other words, I can't encapsulate a Promise inside of the Ajax override. There's no concept of "wait for the new data" because it's just data coming in, not "new" or "updated" data. Just data. As humans, WE know it's new. But there's no way to tell the Ajax function that. It just wants to resolve a promise I can't make.

I have seriously considered forking and adding a parameterized function that accepts a payload directly (rather than waiting for a Promise to resolve) and then goes into otherwise the same code path as what would happen when Ajax resolves. I don't know if it's that simple, but even if it is, not interested in voluntarily giving up Allan's updates!

I don't know much about the plugin architecture. Is it open enough that I could write a plugin that would do the above, rather than forking?

Thanks for reading the wall of text.

allan

Hi Greg,

Good to hear from you again!

But now that I can get the back end to send me paginated payloads (including all the relevant params like totalRecords and start record and all that), I have no way to feed it to DataTables in a "client-side" way. It only cares about the data. Or am I wrong? Can I give it pagination/search/filter values and tell it to update the rendered view?

I think you are going to have to elaborate on this one for me a bit. What do you mean by giving DataTables the pagination/search values? DataTables is the input for those parameter, the client requests what they want to see.

To clarify, does your backend have a way of answering a request for page X, and responding with only the records for that page? Or can it only do all records or individual updates?

Allan

GregP

Hi Allan! Good to hear from you, too. It's too bad you made such a rock solid product that you never hear from me! Regarding this specific dilemna, I have no idea how to keep the description terse. I keep trying to rewrite, and end up with 8 paragraphs again. Here's attempt 3 (previous 2 discarded):

The back end just sends out broadcasts, and the front end just receives them independently of DataTables. So how does DataTables use the data? Currently we have an event emitter that publishes that the data has arrived. There's a subscriber for the data, and it calls a handler that brings in DT-- it makes a new DataTable, if this is the first time, or it clears, adds, draws if this is a subsequent call. It has functioned fine up until now because as client-side rendering expects, we were initializing or updating with a full data set.

In the planned update, we can instead tell the back end to send out paginated/filtered data. The broadcaster even echoes back our request parameters such as start record, search term, sort column and order, etc. The problem is: client-side rendering doesn't know what to do with that stuff as far as I know (would love to be wrong!). It just renders as if the 25 records are the complete data set. Page 1 of 1, 25 total records.

Is there a way, with client-side rendering, to pass it/set the pagination/sort/filter values and render them? I fear the answer is "no". The whole reason server-side option exists is to handle that case!

But I also can't switch to server-side, because "ajax" needs to trigger the callback. It doesn't matter what else happens, but "callback" needs the new data so that its rendering life cycle can be complete. There's no callback, because there's no "fetch" per se. We don't ask "give me data", the update is triggered by an event external to DataTables.

---

My best idea for a workaround that I arrived at between my original question and now is to make a function that implements promise, and then poll on my in-document data for a change. This function is used inside the "ajax" property. When an appropriate change is detected (ie. to one of the pagination/sort/filtering values) I can resolve the promise, which can then trigger the callback. Feels a bit fiddly, but I don't hate it. It does add an extra layer of fragility, though.

Versus, imagine if I could do this:

myTable.completeUpdate(payload);

Where payload includes data, recordsTotal, recordsFiltered. User interactions are maybe another interesting problem to solve, but let's solve that one later.

Sharing code of what we're currently doing would be pointless at this stage, since it's still a design and architecture discussion. Seeing how we update the broadcasters wouldn't be useful. But I feel guilty for not sharing any since that's the usual way to go to get support, haha! Thanks for reading anyhow.

...I not only failed to be terse, but wrote something even longer than the original!

GregP

Yikes, I think I lost about 10 paragraphs of answer... probably for the best, maybe I can distill it into something less daunting (edit: I failed). Good to talk with you again, too! Too bad DT is so solid that you never have to hear from me. ;-)

My imaginary best case scenario:

myTable.completeUpdate(payload);

Where payload contains data, recordsTotal, recordsDisplayed

It's not really about giving DT pagination/search values, I guess. I misspoke. But what I really mean is being able to manually trigger updates to the "info" and pagination widgets based on updated and passed-in values.

But as far as I know, client-side doesn't have a concept of these things being passed in, because it was written with the understanding that the data is available as a complete set. It calculates all of that FOR you, so why would you need to pass them in? In fact, you anticipated the need to do something like this, and wrote... server-side processing!

So why not just use server-side processing? It's mostly about the "ajax" property that is used when server-side is true. I don't have a URL because my data isn't fetched from an HTTP endpoint. And I don't have a promise to resolve because the data just keeps arriving asynchronously. I can't "request" data inside of "ajax" in order for a promise to resolve.

---

The best I could come up with between my original question and now is that I go ahead and use server-side, including an "ajax" override. Inside the ajax override, "data" is collected (my new page/sort/filter params), which triggers a method that can update the independent broadcaster so that it starts sending out new data based on the parameters. Following this, I call a method, say waitForChange() which implements Promises. waitForChange in turn either polls or watches via proxy for a change on my in-document data. When a meaningful change is detected (ie. the things OTHER than data), I resolve the promise, the "ajax" override can be completed, and I think we're off to the races.

But

myTable.completeUpdate(payload).draw(false);

would be better. ;-)

OR even

completeUpdate(myTable, payload) which would alias adding new data, updating total and display records, and draw.

allan

Yikes, I think I lost about 10 paragraphs of answer...

It was just picked up by the spam filter. I've let it through now but can remove it if you prefer.

Let me get back to you on this tomorrow when I've got time to read this and wrap my head around it

Allan

GregP

Haha, it can stay, but it's somewhat redundant. I just tried about 4 times to explain it in different ways! One of those 4 sticking around isn't the end of the world. ;-)

If you think a demo or something would be helpful, drop me a message.

GregP

Just FYI, I did a POC with ajax:

ajax: function (data, callback) {
    fcSocket.editBroadcast(broadcastID, data);

    // Wait for promise to resolve with updated data
    utils
    .checkUserUpdate(3000)
    .then(data => {
        callback({
            draw: data.draw,
            recordsTotal: data.totalRecords,
            recordsFiltered: data.totalFilteredRecords,
            data: data.user,
        });
    })
    .catch(error => {
        console.error('Data update failed:', error);
    });
},

fcSocket.editBroadcast knows how to tell the back end to update based on user interactions, and broadcastID is higher up in the scope. utils.checkUserUpdate is a function that implements Promise and can be given a timeout in MS after which the promise rejects.

So it fires off the async update, which does not and cannot return anything directly into the "ajax" function (the source of my original head-scratching). After this happens, the back-end broadcaster updates subsequent payloads per the edit request. checkUserUpdate keeps peeking in on the variable that stores the data, and comparing an "initial" set of values with "incoming" until a change is noticed or it times out.

I need to have my colleague on back end update our payloads to make it work beyond just a POC, but it's promising.

allan

I'm still a little bit confused . Server-side and client-side processing are two very distinct things in how they handle the data. In server-side processing it will ask for just the rows for the currently displayed page - it isn't entirely clear to me that your backend will do that? Or does it do that, and the issue is "only" how to update the current page of data from the socket notification of a row update?

Allan

GregP

It's not the DataTable that requests our back end data. It comes in over a websocket, where any number of different subscribers use it. Since DT isn't the one requesting the data, there's no URL to supply, and a custom "ajax" function also doesn't do a request. We have to misuse "ajax" not to make a request, but to figure out a way to tell if new data is available "outside" of DT, and if so, bring it in.

Don't worry overmuch about how the data arrives, except that we cannot get DT to request it. Just imagine that we have a variable, "userData" that's an appropriately formatted "response". How could we allow DT to update when we broadcast that "userData" has been updated? And how can we tell DT that "userData" includes changes to totalRecords, start, search, sortCol, and other params normally reserved for the request phase?

As far as I can tell, there's not a way to do that, which leaves switching to server-side. But DT doesn't know how to get the data from our mechanism. Which left me with implementing the "checkUserUpdate" function that can eventually return new data. "checkUserUpdate" feels like a bit of a hack (especially since it uses a setInterval nested in a setTimeout, which is just yucky) when ideally I would be able to use my own mechanism for detecting a change (our emitter) and then simply tell DT to update accordingly.

I can update DT this way right now (with client-side), which is what we do because we have a full data set, but I can't continue doing that because I can't tell DT about changes to pagination, search, etc.

GregP

Keep in mind that I'm just going by documentation and samples. server-side needs ajax, and ajax seems to want to resolve a Promise or otherwise call "callback" with the new data. I can't just immediately fire the callback because new data is going to be in-flight.

If my explanation of my data flow and why I think DT can't handle it without workaround turns out to be wrong, the entire explanation might just be confusing because you might go, "Why not just use THESE options/api instead?" and I might go, "Oh."

kthorngren

Have you considered creating your own paging and search elements? I know it takes away from Datatables functionality but it might be easier than trying to force fit the source data into what Datatables supports. Datatables can still be used to display the data.

Kevin

GregP

Hey Kevin! I definitely thought of it, yes. :-) The ajax workaround posted above is what I think works out easiest even if there's a mental model disconnect. But I definitely thought of it!

allan

How could we allow DT to update when we broadcast that "userData" has been updated?

You'd listen for the change and then call draw() (assuming server-side processing is enabled).

And how can we tell DT that "userData" includes changes to totalRecords, start, search, sortCol, and other params normally reserved for the request phase?

When DataTables makes its request for data (ajax can be a function, so you could do some kind of local cache, it doesn't need to make an Ajax call) you'd return the updated data.

Fundamentally what I'm struggling with here is that I don't see how server-side processing can work with your implementation, if there is no mechanism for the table to say "I need records X to Y, with search Z applied". This is true regardless of if it is DataTables you are using or something else.

If you are streaming in updates from the socket and are using client-side processing, then you just update the rows with row().data(), which I guess is what you are doing now?

But is the issue you are trying to resolve the initial load of the data is significant?

If you want to lazy load pages of data, I don't see how you can get away from a mechanism to request specific pieces of data. Then if you are streaming updates, if a row with an update happens to be displayed, just redraw that page of data. That's what I did with CloudTables.

Sorry if I am making the water more muddy!

Allan

GregP

I'm not saying you're making the waters more muddy... but the problem is pretty clear and I'm not doing a good job explaining it. ;-)

We are indeed currently using clear, add row data, redraw to make things work when we have a full data set. But you can't clear, add row data, update pagination/filter information, and redraw. To make those parts of the UI change, you need to be in server-side mode.

The piece I think you're missing is that in the POC above, I do have a way to tell our back end, "I need records X to Y, with Z applied". We override the ajax function, and the first function that gets called, fcSocket.editBroadcast(broadcastID, data) communicates independently of DT to tell the back end that our needs for data have changed. Your own ajax implementation carries those values inside "data", and we can send them to our back end independently.

Then the next function implements Promise, but does some kludgy waiting for the in-memory data to change in order to resolve the Promise, versus actually "fetching" the data. Once the Promise does resolve, "callback" is invoked and the table redraws. That's how server-side works with the POC.

It's hard to explain that the data flow is different from most people's expected "request-response" paradigm. Honestly, the websocket push mechanism has been better in every way until we've run into this little hurdle, haha! And that's been with DataTables all along, which has been great.

In any event, the more I look at the POC, the more I think, "This isn't so bad." The only significant thing I will need to do to make it feel less kludgey is have our back end echo back our request parameters so that we can identify that the change has come into memory.

GregP

Or in other words, with client-side, I am currently able to pass in the data via API, and call draw. I want to be able to pass in the data PLUS the pagination/filtering/sorting params, and call draw. ;-)

The workaround to make that happen is to enable server-side, implement a slightly hacky ajax function, and call it a day I guess.

allan

I'm not saying you're making the waters more muddy...

Lol. However...! I'm unfortunately good at that sometimes. I've reread your comment on the POC code and yeah, that makes a lot of sense. Your workaround is indeed going to be the only way to make it happen with the DataTables code as it stands.

That said, at some point I am going to look into how to have the server-side tell the client-side about what it should be showing in terms of paging. This is required for infinite paging and for cases where you might request a page of data that no longer exists. The server would need a way to say "actually, instead of X to Y, here is A to B, draw that please".

I've not done any work on an implementation for that yet though.

Allan

GregP

Interestingly, yes, for many cases the server would need a way to say that, although for many of those cases, the server would actually just need to echo back what the client side requested.

For my particular use case, I think (would have to think it through again) it is enough to be able to say something like, "Hold onto those UI changes until data state gets updated" and then when the data comes in, it releases the UI changes so that everything stays in sync. That's what the ajax replacement POC is doing, more or less. The Promise doesn't get resolved until the data is resolved to my application's satisfaction, at which time the "ajax" gets completed and UI state can update.