After adding ko dashboard to datatables page, three extra draw events fire randomly on page return

WAdev_235

Link to test case: The application is used on an intranet behind a firewall and a link cannot be provided. The complexity of the application prevents the objects from being recreated in an online sandbox.
Debugger code (debug.datatables.net): This cannot be provided because data protected by privacy laws would be exposed.
Error messages shown: None
Description of problem: After adding a ko.js data model based dashboard to a page with a datatables object, the page sometimes (seemingly randomly) experiences three extra draw events when returning back to the datatables page. The state is not restored on the three extra draws, so that the sort by column and page size is reset.

Details:
DataTables 1.13.4, FixedColumns 4.2.2

The size of the data set for the table changes because the page being returned from changes the status of an item in the table so that it no longer appears. stateSave is enabled.

hard-coded parameters:
"pageLength": 10,
"pagingType": "full_numbers",
"processing": true,
"serverSide": true,
"stateSave": true,
"info": false,
"destroy": true,
"autoWidth": false,
"ordering": true,
"order": [[defaultSortedColumn, defaultSortedDirection]],

Regular operation with page size manually set to 25 and the page sorted by some column :

Error with page size manually set to 25 and the page sorted by some column:

(Logging is set to Verbose and the Violation message sometimes happens during regular operation also and the error does not happen.)

I have tried
- upgrading the dataTables from 1.13.4 to 1.13.10 along with the Bootstrap js files and jQuery js to latest versions
- upgrading the dataTables from 1.13.10 to 2.0.0
- changing the style sheets to the current versions
- changing the call to draw the dashboard from an .on call, to a .one call, to being part of the onDraw callback
- moving the dashboard out of the dataTables area on the page
- disabling code that adds a paging button before the pagination feature
- setting the knockout ko.options.deferUpdates flag to true before instantiating the dashboard knockout data model and then setting it back to false
- tried setting .extend({ deferred: true }); on the DashboardModel ko observable array
- setting up logging to record the contents of event and settings variables and record when draw is called

Has anyone experienced something like this before - 3 extra redraws seemingly randomly? It is not specific to a particular item in the table, nor to the number of returns back to the datatable preceding. (I'm hoping the 3 extra redraws means something to someone because it is always 3.) Is there anything else I can try for debugging/troubleshooting? With a breakpoint enabled in Visual Studio in the stateLoadCallback, I have not been able to get the problem to reoccur. Some sort of timing may be involved.

With the exception of some fix, to make the application function consistently, I think setting stateSave to false is the only option at this point.

I need to read more into the documentation, but my current understanding is that with stateSave enabled, the sort and item count in the datatable cannot be manually set. Then if stateSave is disabled so that the sort and item count could be set, the stateLoadCallback and stateSaveCallback that the page uses would not be fired.

kthorngren

I see initComplete fired a bunch of times. That suggests something is happening to cause Datatables to be initialized or reinitialized.

Error with page size manually set to 25 and the page sorted by some column:

Does this output of multiple initializations occur in a short period of time?

Without seeing any code its hard to say what to debug. It sounds like you will need to debug why the code to initialize Datatables is called 3 times in a short period of time. Not sure if this is in an event handler, function, $( document ).ready(..), etc.

I'm guessing the stateLoadCallback is performing a ajax request for the state and since there seems to be multiple close to simultaneous calls to init Datatables the ajax responses are still outstanding causing a race condition which ultimately causes Datatables to use the initialization options instead of the saved state.

Its possibly the violation following the stateLoadCallback is caused by a delayed response from the ajax request. Use the browser's network inspector to monitor this to see if there is a delay when the violation occurs.

Let us know what you find.

Kevin

kthorngren

If you are having slow ajax responses from the stateLoadCallback then possibly you can disable the stateSaveCallback and stateLoadCallbac and just use the default local storage for the saved states.

Kevin

WAdev_235

Thank you for replying.

The stateLoadCallback and stateSaveCallback only use sessionStorage, there are no AJAX calls there. The state managed by the stateLoadCallback and the stateSaveCallback is retained when the error occurs.

Then initComplete has
console.log("initComplete fired"); //I added this just for debugging
$("#mainContent").show();
$.unblockUI();

The multiple redraws happen in a burst. There are 4 draws total ( 1 regular and then 3 extra) in a very short period of time - it appears on screen as the datatable with data and then another draw that resets everything. When it does happen, it all happens as soon as the page loads.

This application has been around for years and it has two places on the page where there is a setTimeout() specified. I disabled both of those and the problem still occurs. The setTimeout() in the Violation console error is coming from one of the js libraries.

When the error occurs, the AJAX calls look like
Everything loaded before that is the .aspx page, the .js files, and the .css files.

There is the initial dataTables AJAX request for info, some icons load, the GetDashboardData AJAX call happens, more icons are loaded, then those two calls each happen three more times when the error occurs.

I added console messages for the dashboard data AJAX call for start, success, and failure. The failure case is not happening

Then when everything is working normally, it looks like

According to the console log, in the error case the console log deviates from the regular case beginning with the stateLoadCallback occurring three times in sequence. Then the rest of the processing related to each of the stateLoadCallback events happens in sequence and is not interleaved. Is there some other place, such as in the dataTables js files, that I can/should put debugging statements to gather more information? I have not looked at or read through any of those files before.

kthorngren

Is the WorkLoadTable.aspx the page that contains the Datatable? If so it looks like something is causing that page to be reloaded which is something Datatables won't do. If an error occurs within Datatables there will either be console messages or alert message. It doesn't reload the page or reinitialize itself when an error occurs.

I'm not familiar with ko.js but I wonder if its doing something to cause the page reload, since you mentioned the problem didn't occur before adding ko.js.

Kevin

WAdev_235

The WorkListTable page is just for obtaining DataTables data, and its code behind page is where some logic is contained.

"ajax": {
                    url: 'WorkListTable.aspx',
                    data: function (d) {
                        return {
                            "draw": d.draw,
                            "start": d.start,
                            "length": d.length,
                            "sortDir": d.order[0]['dir'],
                            "sortCol": d.order[0]['column'],
                            "whichTable": self.currentTableId(),
                            "isHistory": self.isHistory,
                             ... (there are more variables here)
                      };
                  }
          },

The DataTable resides on a page named Default and has three tabs. The page had ko.js on it already for various functions and has a ko View Model. These parts of the application have not changed.

The additional code for this effort has this

                    switch (self.currentTableId()) {
                        case "MyWorkListTable":
                            self.getDashboardData('MyWorkList', self.WorkPosId);
                            break;
                        case "ActiveWorkListTable":
                            self.getDashboardData('ActiveWorkList', self.searchForWorkPosDisplayValue());
                            break;
                        case "HistoricalWorkListTable":
                            self.getDashboardData("HistoricalWorkList", null);
                            break;
                    }

now in the drawCallback.

There is now also this code in the View Model (ko) for the page

        //Dashboard Section
        self.dashboardData = ko.observable(new DashboardModel());

        self.getDashboardData = function (worklist, workPos) {

            framework.logInfo("Get dashboard for " + worklist + " data started.");

            dataAccess.getDashboardData(worklist, workPos,
                //sucess
                function (data) {
                    self.fillDashboardWithData(data.ReturnValue);

                    framework.logInfo("Get dashboard data ended.");
                },
                //failure
                function (dt, exception) {
                    framework.logInfo("Get dashboard data failed.");
                    framework.showErrorMessage([dt.responseText]);
                    return;
                }
            );

        };

        self.fillDashboardWithData = function (data) {

            self.dashboardData().setDashboardData(data, self.searchForNAMEREMOVEDCreatedChkBoxValue());

        }

plus some other new code that is not called during this error event.

Then the Dashboard Model referred to is

class gridCell {
    constructor(name, detailLevel, greenNum, yellowNum, redNum, isData, NAMEREMOVEDCreated) {
    
        this.name = name;
        this.detailLevel = detailLevel; //P=program data, D=detailed, S=summary, E=empty
        this.onTimeCount = greenNum;
        this.nearLateCount = yellowNum;
        this.lateCount = redNum;
        this.isData = isData;
        

    }
    //method_1() { ... }
    //method_2() { ... }
    //method_3() { ... }
}

define("App/Model/DashboardModel",
    function (require) {

        var $ = require("lib/jquery/jquery");
        var ko = require("lib/knockout/knockout");
        var framework = require("App/Common/Framework");
        require("lib/knockout/knockout-postbox");

        return function DashboardModel(options) {
            var self = this;

            self.grids = ko.observableArray();

            self.setDashboardData = function (data) {

                self.grids.removeAll();

                var len = 0;

                if (data != null) {
                    len = data.length;
                }
                else {
                    return;
                }
                
                var numCells = 0;

                //Bootstrap row-col-5
                if (len <= 7) {
                    numCells = 10;
                }
                else {
                    var numRows = Math.ceil((len - 7) / 3);
                    numCells = 10 + numRows * 5;
                }
               
                var j = 0;
                for (var i = 0, j = 0; i < numCells; i++) {

                    var isData;
                    var columnPos = i % 5;

                    if (i == 4) {
                        isData = 0;
                        self.grids.push(new gridCell(ko.observable("Work Position ID"), ko.observable("P"), ko.observable(0), ko.observable(0), ko.observable(0), ko.observable(isData)));
                    }
                    else if (i == 8) {
                        isData = 0;
                        self.grids.push(new gridCell(ko.observable("NAME REMOVED Created REMOVED"), ko.observable("P"), ko.observable(0), ko.observable(0), ko.observable(0), ko.observable(isData)));
                    }
                    else if (i > 5 && (columnPos == 3 || columnPos == 4)) {
                        isData = 0;
                        self.grids.push(new gridCell(ko.observable("Unassigned System Area"), ko.observable("P"), ko.observable(0), ko.observable(0), ko.observable(0), ko.observable(isData)));
                    }
                    else if (j >= len) {
                        isData = 0;
                        self.grids.push(new gridCell(ko.observable("No Data"), ko.observable("E"), ko.observable(0), ko.observable(0), ko.observable(0), ko.observable(isData)));
                    }
                    else {
                        isData = 1;
                        var category = data[j];
                        self.grids.push(new gridCell(ko.observable(category.Name), ko.observable(category.DetailLevel), ko.observable(category.greenNum), ko.observable(category.yellowNum), ko.observable(category.redNum), ko.observable(isData)));
                        j++;
                    }

                }

            };

        };

    }
);

where REMOVED and NAMEREMOVED are where info has been omitted for posting in this public forum.

kthorngren

It makes a bit more sense but code snippets are hard to follow and debug.

Do you have code that calls draw() or ajax.reload()? If yes then possibly there is something that triggers the code to execute multiple times.

Is Datatables initialized in $( document ).ready(..) or does it reside in a function that is called?

Can you post your Datatables initialization code?

Kevin

WAdev_235

There is one .draw call in the code that only occurs if a user clicks a "Go" button to go to a particular numbered page that a user would have entered into a textbox. The button and textbox were added by one of the previous developers of the application and are not new for this development work. It is not firing in this error case - I previously checked by commenting out that whole section of code that defines the button .on('click', ...) just to make sure, and also previously tried commenting out the code that places the textbox and button into the page just to make sure that insertion code was not causing a redraw. (The function code that sets up the custom go-to paging is called gotoPage and it shows up in the drawCallback below.)

There are no ajax.reload calls.

The Datatables is initialized in the ko View Model. I tried pasting the initialization code here, but then the site gave an error - Body is 15881 characters too long. I had to put it here - https://jsfiddle.net/ansi235/7szk9014/ .

I appreciate the help and wonder if there is a way that I could do more to try to figure out what is happening with the ko and the datatable. I need to look into the ko documentation and whether there is a way to enable verbose console logging of everything occurring. I could not find a way to do that with datatable - is there a line in the datatable js code that I could set a variable to true, or set some value to some other enum value, to get the datatable code to output verbose info about what it is doing?

kthorngren

There isn't anything I know of to turn on detailed debugging in the datatables.js code. Since initComplete is running each time the Datatable is being reinitialized. Sorry for not thinking about this when asking about draw() or ajax.reload() as thy won't cause intiComplete to run.

I don't see anything obvious in the init code.

Is the init code inside a function or does it run from document.ready(). If in a function then I would place breakpoints at the locations in the code that call the function. Maybe you can backtrack from there.

Are you using a knockout method to init Datatables? If so place a breakpoint there and see if its hit when the problem occurs.

Or, if you want to prove its not a Datatables issue you could put a breakpoint on a statement that executes before Datatables is initialized or use a console.log statement. If the console.log or the breakpoint is hit when the problem is occurring then you know Datatables isn't reinitializing itself but something is calling that code for Datatables to initialize.

Kevin

WAdev_235

Thank you for the suggestions.

The ko based program uses an Init js file to load things:

define("WorkListInit",
function (require) {
    var $ = require("lib/jquery/jquery");
    var ko = require("lib/knockout/knockout");
    var viewModel = require("App/ViewModel/WorkListVM");
    var header = require("App/ViewModel/PageHeaderVM");
    var footer = require("App/ViewModel/PageFooterVM");
    require("lib/knockout/knockout-postbox");
    require("lib/jquery/blockUI");

    var self = this;

    // Executed when the View Model is created.
    function InitializeViewModel() {
        ko.applyBindings(new header(), document.getElementById("NAMEREMOVED-page-header"));
        ko.applyBindings(new viewModel(), document.getElementById("page-container"));
        ko.applyBindings(new footer(), document.getElementById("NAMEREMOVED-page-footer"));

    }

    $(function () {
        //TabFixForPortal()
        $.blockUI({ message: '<h3>Just a moment...</h3>' });
        // Initialize the appilcation.
        InitializeViewModel();
    });
});

I tried breakpointing each of the lines in InitializeViewModel and found out that on a normal page load, each one fires once, as expected. When the error occurs, each still only fires once. The new viewModel() line is where the WorkListVM code is only run once whether or not the error occurs. What is run three extra times is the table code inside WorkListVM. Each time, it initializes the datatable using the code in the previous post.

It normally looks like

define("WorkListInit",
define("App/ViewModel/WorkListVM",
return function WorkListVM() {
ko.applyBindings(new header(), document.getElementById("NAMEREMOVED-page-header"));
ko.applyBindings(new viewModel(), document.getElementById("page-container"));
bindListData = function (array, methodName, addSelectOneListItem, textBindPropertyName, valueBindPropertyName) {
self.Initiate = function () {
table = $("#" + self.currentTableId()).DataTable({
ko.applyBindings(new footer(), document.getElementById("NAMEREMOVED-page-footer"));

vs.

define("WorkListInit",
define("App/ViewModel/WorkListVM",
return function WorkListVM() {
ko.applyBindings(new header(), document.getElementById("NAMEREMOVED-page-header"));
ko.applyBindings(new viewModel(), document.getElementById("page-container"));
bindListData = function (array, methodName, addSelectOneListItem, textBindPropertyName, valueBindPropertyName) {
self.Initiate = function () {
table = $("#" + self.currentTableId()).DataTable({
ko.applyBindings(new footer(), document.getElementById("NAMEREMOVED-page-footer"));
table = $("#" + self.currentTableId()).DataTable({
table = $("#" + self.currentTableId()).DataTable({
table = $("#" + self.currentTableId()).DataTable({

for the error case (where each line is just the first line of some container/function, return function WorkListVM() is within the WorkListVM view model file, and the bindListData, self.Initiate, and table code lines are within the WorkListVM() function.)

I need to think about where to set more breakpoints.

WAdev_235

My original problem observation, and reason for submitting this forum request, is still outstanding - the state is not restored on the three extra draws, so that the sort by column and page size is reset.

Why would datatables not save its state? The programmer-defined state programmed in the stateLoadCallback and stateSaveCallback is being retained.

kthorngren

That is a good question. As I said before its likely a race condition that happens when the Datatable is reinitialized the 3 extra times. Since the 3 extra reinitializations seem to happen in an error condition then fixing the error condition should keep race condition from happening and the 3 extra re-inits.

I tried a quick simulation here:
https://live.datatables.net/xivohetu/1/edit

But the state is not affected. I'm sure I'm not getting the correct timing of the events. If you can post a link to a test case or maybe update mine to show the issue it can be investigated.

Kevin

WAdev_235

I found what is causing the redraws. There is a dropdown box in the new dashboard and its change event, when the dashboard loads, is programmed with a ko js event handler. The dropdown box data is loaded into the dropdown ko observable array prior to DataTable and dashboard creation and most of the time does not trigger the change event. Sometimes, for some reason, the change event is firing multiple times after the page and DataTable loads, and it is always 3 times. The event handler calls the code that re-initializes the DataTable because the dropdown value affects the search results, and the existing search capability is programmed to re-initialize the DataTable.

The dashboard design did not include for the possibility of the change event firing due to ko system activity. To stop it from happening, I used the accepted solution found at https://stackoverflow.com/questions/11078016/change-event-on-select-with-knockout-binding-how-can-i-know-if-it-is-a-real-cha . I started programming with ko years after the article was written to be able to maintain and enhance this ko application that is still around, and did not consider the possibility the solution programs against - I thought ko worked differently.

I will still see if I can try to duplicate this scenario in a jsfiddle or your simulation. I am not sure how long it will take or whether it will be successful. I will reply again if it is or if I decide to stop trying, either way. I would like to be able to help duplicate it in an environment you all have access to, because it seems like DataTables should have retained its state.

Kevin, thank you for all of your assistance/suggestions.

kthorngren

Great, glad you found the issue.

Kevin

WAdev_235

I created an example on live.datatables.net that has a similar but not the same structure and set of function calls that the actual application has. https://live.datatables.net/cefogetu/1/edit

With the example, I am able to reproduce the sometimes DataTable screen flicker (it is more than just the rows re-ordering), but not, so far, the loss of state that also occurred (where it would lose the sort-by column and the number of entries to show). I have Auto-run JS unchecked. Then when I click Run with JS, it sometimes flickers. With the developer screen open to be able to see the console, it shows multiple redraws firing, similar to the actual application. In the actual application, everything worked after the commented lines in the similar function to PosDropdownChanged function (below) are uncommented. Those lines were added after finding the _stackoverflow.com _ article.

            PosDropdownChanged = function (data, event) {
                console.log("PosDropdownChanged fired");
                //if (event.originalEvent) { //user changed
                      
                DashboardSearchRequest(data, event);
                //} else { // program changed

                //}
            };

Without them, the flicker and loss of state occurs. I am not sure what else to try to get the DataTable to lose its state. Just to prove it is occurring, I could maybe record a video and block out the data, but that would not get you all code to look at to investigate. This is as close as I can get for now.

WAdev_235

I am not sure if the way that the https://live.datatables.net environment is set up is keeping it from losing state somehow, or if the similar example is just not similar enough. I only minored in Computer Science.

kthorngren

Have you had an issue with the state not restoring properly since fixing the change event?

I tried your test case a few times and did not see the loss of state.

I am not sure if the way that the https://live.datatables.net environment

You can click the upward facing right arrow next to the Auto-run JS checkbox to run the page outside of the JS Bin environment. I tried that as well.but still no state loss.

I just think its a race condition that happens when the change event fires multiple times quickly. Race conditions are typically hard to reproduce in a different environment. Race conditions cause unpredictable results.

You can see in your output above stateLoadCallback is called 3 times consecutively in the error condition. Then the 3 Datatables complete their initialization. This suggests that there are 3 instances of the same Datatable trying to initialize at the same time. The stateSaveCallback is called only once during this time. The stateSave callbacks are out of sync during this condition.

Thanks for trying to recreate the issue.

Kevin

WAdev_235

Since using the code in the stackoverflow.com article in the change event, the dashboard data load and associated events are only happening once. The built-in DataTable state retention is working correctly and there is only one load of the data.

No further support is required. I posted again with the example because I wanted to try to recreate the original situation (clearly not ideal, because of the multiple data requests for the dashboard data and DataTable data) just so that the DataTable flicker and state loss could be observed. It seemed like it should not do either of these things even when everything else was not ideal.

Thanks for the troubleshooting help. Hopefully the DataTables example I provided is somehow useful, or maybe just the reference to the stackoverflow.com article will be useful to someone in the future.

kthorngren

I think the flicker is due to having the Datatable at various states of initialization (not completed) when setTimeout fires to start a new initialization. The previous table is displayed, usually unsorted, until a full initialization takes place. You can see the same in my test case.

Kevin