Chrome and Edge crashing when exporting large dataset

dowzer

Using DT 1.10.25 with Export option to export to CSV - is using AJAX to load the data

We have a specific scenario whereby when trying to export "large" datasets crashes Edge and Chrome with an out of memory error - Firefox does not have the same issue.

The table in this instance has 4,187 rows and 110 columns - total of 460,570 items.

The page by default loads 25 rows and there is no issue - however, to export the user needs to load all the rows onto the screen to make them visible and then export to CSV. At this point (ie rendering 460k items) Chrome and Edge crash due to being out of memory. If they choose to load 2,000 and then export it does not crash so somewhere between 2000x110 and 4187x110 the browser runs out of memory. We have other reports with 25,000 rows and, say, 10 columns which work with no issue etc

Broadly speaking the parameters being used are:

var parameters =
    {
        "lengthMenu":
        [
            [10, 25, 50, 100, 250, 500, 1000, 2500, 5000, -1],
            ['10', '25', '50', '100', '250', '500', '1000', '2500', '5000', 'Show All']
        ],
        "ordering": false,
        "searching": false,
        "autoWidth": true,
        "dom": '<"datatable-header"fBl><"datatable-scroll-wrap"t><"datatable-footer"ip>',
        "processing": true,
        "serverSide": true,
        "language":
        {
            lengthMenu: '<span></span> _MENU_',
            paginate: {
                next: '&gt;', // or '>'
                previous: '&lt;' // or '<'
            }
        },
        "ajax":
        {
            "url": "/reports/reportAjax.php",
            "data":
            {
                "reports_id": "12345"
            },
            "type": "POST"
        },
        "buttons":
        {
            "className": "btn-group",
            "dom":
            {
                "button":
                {
                    "tag": "a",
                    "className": 'btn btn-default'
                }
            },
            "buttons": [
                {
                    extend: 'copyHtml5',
                    text: 'Copy',
                    exportOptions:
                    {
                        columns: ':visible'
                    },
                    "charset": 'UTF-8',
                    "bom": true
                },
                {
                    extend: 'csvHtml5',
                    text: 'CSV',
                    fieldSeparator: fieldSeparator,
                    exportOptions:
                    {
                        columns: ':visible'
                    },
                    filename: 'Grade D and above',
                    "charset": 'UTF-8',
                    "bom": true
                },
                {
                    extend: 'excelHtml5',
                    text: 'Excel',
                    exportOptions:
                    {
                        columns: ':visible'
                    },
                    filename: 'Grade D and above',
                    "charset": 'UTF-8',
                    "bom": true
                },
                {
                    extend: 'pdfHtml5',
                    text: 'PDF',
                    exportOptions:
                    {
                        columns: ':visible'
                    },
                    filename: 'Grade D and above',
                    orientation: 'portrait',
                    title: 'Grade D and above',
                    "charset": 'UTF-8',
                    "bom": true
                },
            ]
        },
        "fnInitComplete": function(oSettings) {
            $( window ).resize();
        },
        "fnDrawCallback": function(oSettings) {
            $( window ).trigger('resize');
        }
    };

The specific data is sensitive data so it is hard to give an accurate recreation of the issue but this can be arranged using some test data if needed.

Is there something obvious I am missing? Is the export function different to the page load function in that it uses significantly more memory as the crash "only" happens when exporting?

Jason

allan

Hi Jason,

The CSV export should be the least memory intensive of the exporters - however, it is indeed quite possible that you are hitting a memory limit somewhere.

What I would suggest is in a local copy of buttons.html5.js location this line in the CSV exporter and comment it out. Then run it again and see if it crashes or not.

That line is the part that does the local file creation for download. If we isolate that first, it will either confirm that it is the issue, or if it is before in the data gathering from the DataTable.

If the page runs without error without that line (it won't download the file of course, but it would prepare the CSV text), then we need to look into that part. It would be interesting to know what output.length is when it is crashing.

I just went to have a look at the FileSaver.js to see if there is an upgrade available, and at the top of the readme it does note:

If you need to save really large files bigger than the blob's size limitation or don't have enough RAM, then have a look at the more advanced StreamSaver.js that can save data directly to the hard drive asynchronously with the power of the new streams API.

That said, the table below it shows that Chrome should support blob sizes to 2GB. That sounds like it should be easily large enough for your use case.

Allan

dowzer

Thanks for the reply Allan - with that line commented out it does still crash.

I have tried console logging output.length but it seems to crash before getting to the line so it never displays - any ideas where I could place it where output is built sufficiently to get the size but before it starts the download process which causes the crash?

Not sure if relevant but copy also does the same thing

Jason