Scroller Plug-In - Issue with scroll bar when server side processing and large number of rows.

curts

When using DT and Scroller plug-in with server side processing and a large number of rows (e.g. 5million+) the scroll bar and scrolling are not proportional to the number of records specified by the iTotalDisplayRecords/iTotalRecords (both values are the same - there is no searching/filtering involved.)

A JS Fiddle is located at: http://jsfiddle.net/aJ6xL/

Therein you can play with the value at the top of the JavaScript:

[code] var nrecs = 5000000; [/code]

Note that when you change the value to something relatively small (e.g. 25,000) the scroll bar can be used to scroll all of the way to the bottom of the rows/table; however, if you change the nrecs value to something high (e.g. 5million+...it doesn't even have to be that large, but it's a good test) the scroll bar handle will be out of proportion (it will be large/long instead of small/short) and you are no longer able to use the scroll bar to scroll to the very bottom of the rows/table.

The behavior does seem to be different in various browsers, with IE being the worst in terms of not supporting a very large number of rows.

Anyone else seeing this issue? If so, any fixes?

Thanks!
Curt

p.s. Thanks to this poster (http://www.datatables.net/forums/discussion/comment/16761#Comment_16761) for supplying the server side processing simulation code used in my JS Fiddle.

curts

Bump...anyone else have this issue?
Thanks
Curt

allan

Hi,

Thanks for picking up the DataTables support option! The problem stems from the fact that browsers have a maximum height for individual elements (what that height is varies between the browsers - there is no defined value in the CSS specification). Scroller uses a 1:1 positioning ratio when working with scrolling data, so that if you move down 200 pixels, you get the row which would be shown 200 pixels later.

As such, the minimum hight of an element defines also the maximum number of rows that can be shown in scroller at the moment.

What needs to happen is that a non-1:1 scrolling mapping is used, but rather a calculated value based upon the number of records. This is something I've been aware of but never had the opportunity to dig into and resolve, but now with your support for the project, I'll get started on this forthwith and get back to you with how I'm getting on. Hopefully I'll have something to show you tomorrow for how to resolve this.

Regards,
Allan

allan

Hi Curt,

I've been working on this today, and I think I've just made the conceptual break thought I needed :-). The actually change isn't actually all that difficult in code terms, but wrapping my head around the changing it taking me a little while to figure out what needs to be done.

The main display aspect of the required change is now done, but there are a number of other changes which are also required to go with this, such as updating the paging information display, since it is slightly out of sync now that there isn't 1:1 mapping.

I'll keep plugging away with this and let you know how I get on in the next day or two.

Regards,
Allan

curts

That's great, thanks for the update Allan! I will be sure to order more support as things progress to assist in your development efforts!

Thanks again!
Curt

allan

Hi Curt,

Another little update. I'm afraid there are still a few more changes I need to make to Scroller before it is ready to be used with the changes to support this mode of scrolling... Sorry its taking a while! I'll try to post some working (or at least mostly working!) code over the weekend.

Allan

curts

No problem, thanks for the update! Please let me know if you'd like me to test any code or help with anything.

Curt

allan

Another update - been working on this for the afternoon and I've made some good progress I think, its working for clicking and dragging the scrollbar, but its jumping around when doing a mouse wheel in the scrolling area. Sorry this is taking so long - its proving to be a bit complicated! One this part is working, it be will be very close be being ready though.

Allan

allan

Here we go: http://live.datatables.net/ewihuv/edit#javascript,html - 1 billion rows :-).

So a couple of things to note:

1. There is still an issue with manually scrolling to the end of the table (or rather, jumping to the end, and manually scrolling up and then down to the end again - you can scroll past the end). This is something I need to sort out. You can jump tot he end using the scrollbar correctly.

2. When in the "virtual" mode (table > 1 million pixels) the scrollbar is non-linear. This is to make the start and end of the table work smoothly. At this point the scrollbar is going to be a "rough" finder anyway since one pixel can pre resent hundreds of rows, so I don't think this is a problem, although I would flatten the non-linear curve I've used a bit - had to dust off my maths memory for this much, going to have to dust off the books for much more :-).

Commit here:
https://github.com/DataTables/Scroller/commit/6c8b715

And full file:
https://github.com/DataTables/Scroller/blob/master/media/js/dataTables.scroller.js

Regards,
Allan

curts

This looks great!!!!!

Regarding your notes:

I am using, along with the scroll bar, the "fnScrollToRow" function hooked up to an input box that allows the user to jump to a specific row; it is not producing correct results (it is moving to incorrect rows).

Ideally we could set whether the "virtual" mode used a linear or non-linear scrollbar, which might make it easier to use the "fnScrollToRow" function to jump to the exact row.

Thanks for all your work on this thus far!!
Curt

allan

I've just committed a fix for fnScrollToRow to have it work with the new setup. The difficult part with it is the animation. Without the animation I'd just paint the table in its new position, but with animation, it needs to be smooth for both the currently drawn table, if the target row is in the drawn area, and for outside the drawn area when it goes into the virtual scroll. I think I've got it sorted out now, but it is tempting to remove the animation to simplify somewhat (although, I suspect it might also end up opening another can of worms!).

Latest update, as always, is here:
https://github.com/DataTables/Scroller/blob/master/media/js/dataTables.scroller.js

Allan

curts

Hi Allan,

Things are looking pretty good as far as the fnScrollToRow, including the animation bits. I did find a couple of issues this morning:

1) It is still off by a few pixels, which I guess it to be expected since we're potentially dealing with fractional pixels representing > 1 row. I think I solved it by introducing a callback for ajax data redraws, then determining the top offset of the scroller table and the top offset of the row I'm scrolling to, then taking the delta and applying that to the scroller table's scrollTop - it appears to give me pixel perfect positioning of the selected record at the top of the grid.

2) There seems to be an issue with the extremes - namely the upper extremes. For example, if I have a set of records in the table, ~2.6 million; if I use the fnScrollToRow to try to go either to or near the end of the table it doesn't do anything. It appears that in the _domain function, which is being called by the fnRowToPixels(iRow, false, true) line in the fnScrollToRow function, the calculation in _domain is returning NaN for the desired pixel value, thus nothing happens. I did adjust it a little and it seems to work, but can you please validate (I added the two -- if(val < 0){...} checks)?

[code]
if ( dir === 'virtualToPhysical' ) {
if ( val < yMax ) {
return Math.pow(val / coeff, 0.5);
}
else {
val = (yMax * 2) - val;
if (val < 0) {
return heights.scroll;
}
return (xMax*2) - Math.pow(val / coeff, 0.5);
}
}
else if ( dir === 'physicalToVirtual' ) {
if ( val < xMax ) {
return val * val * coeff;
}
else {
val = (xMax * 2) - val;
if (val < 0) {
return heights.virtual;
}
return (yMax*2) - (val * val * coeff);
}
}
[/code]

I'll keep testing and let you know if I find anything else.

Thanks Allan!!
Curt

allan

Hi Curt,

Nice one - thanks for this! I'll look into it in more detail and commit the changes shortly.

Allan

allan

Just a quick update to say that these changes have all been released in Scroller 1.2.0 now :-).

Thanks again, Curt, for supporting this new feature!

Allan