DataTables with SOLR json

SOLR_JOHN

Disclaimer: I'm a total jquery/javascript newb, so my questions may appear ignorant to most

First time user of DataTables and I've decided to jump right in and use the beta. It also doesn't help that I'm trying to do something quite specific with it. Looking at previous forum posts, there's only really been one person asking about how to properly implement DataTables with SOLR and that was from a few of years ago.

I've managed to display some data on DataTables through JSONP from SOLR but it's still not quite there. In following sample code, I manage to use the new dataSrc to define where "aaData" is. But I don't have a way to define recordsTotal, recordsFiltered, and draw.

[code]
$(document).ready(function() {
$('example').dataTable( {
"processing":true,
"serverSide":true,
"ajax": {
"url": "http:///select",
"data" {'wt':'json', 'q':'*:*', 'rows':'10'},
"dataType": "jsonp",
"jsonp":"json.wrf",
"dataSrc": "response.docs"
}

"aoColumns": [
{"data" : "company" },
{"data" : "address" , "sDefaultContent": "" }
]
});
});
[/code]

Here's a sample SOLR json:

[code]
{
"responseHeader":{
"status":0,
"QTime":10,
"response":{
"numFound":100000,
"start":0,
"docs":[
{
"company":"ABC Company",
"address":"123 Knowyourrole blvd."
},
{
"company":"XYZ Company",
"address":"106 Park ave."
}]
}
[/code]

What's not working for me is that it is obviously not showing the proper amount of entries in the query parameters because I can't define recordsTotal, recordFiltered. I want to point "recordsTotal" to "numFOund" but it doesn't look like there is an easy way based on the documentation.

I also noticed that no matter how many actual rows I define in my SOLR query to return (to limit the dataset), DataTables just tends to grab everything back from my SOLR index. i.e. I query for 10 rows but in my DataTables result, I can keep clicking page after page of new data.

I just wanted to ask the community if anyone has had any success implementing DataTables as a viable UI view for SOLR.

allan

> But I don't have a way to define recordsTotal, recordsFiltered, and draw.

Is that information actually in the JSON data? Do you actually want to use server-side processing with the server doing all the processing?

If so, what you will need to do is define dataSrc as a function and map the values that SOLR replies with, to the property names DataTables expects: http://next.datatables.net/reference/option/ajax#function .

[code]
$('#example').dataTable( {
"ajax": function (data, callback, settings) {
var o = {
recordsTotal: data.response.numFound,
recordFiltered: data.response.numFound,
data: data.response.docs
};
callback( o );
}
} );
[/code]

or something like that - I doubt I've used the correct parameter names!

Would be interested in seeing your final code if you would be filling to share it, either here or as a comment on the Ajax doc page.

Allan

SOLR_JOHN

Thanks for the advice Allan. Yes I was hoping to use SOLR for all the data processing. I'm still having issues mapping the property names that DataTables expects. Using the example and the sample code you showed, I still can't seem to write the correct code to map the values. Again, my javascript knowledge is extremely limited so perhaps I'm doing something wrong or missing something.

[code]
$('#example').dataTable( {
"processing":true,
"serverSide":true,
"ajax": function (data, callback, settings) {
var o = {
recordsTotal: data.response.numFound,
recordFiltered: data.response.numFound,
data: data.response.docs
};
$.ajax {(
url": ""solr-url",
"data": {'wt':'json', 'q':'*:*', 'rows':'10'},
"dataType": "jsonp",
"jsonp":"json.wrf",
"dataSrc": "callback( o )"
)};
},
"aoColumns": [
{"data" : "company" },
{"data" : "address" , "sDefaultContent": "" }
]
} );

[/code]

I'm confused as to where to put the parameters for the ajax callback for the solr-url itself. The way I have it right now, I just get an error saying "Requested unknown parameter..." Which tells me it's not being mapped correctly.

allan

> "dataSrc": "callback( o )"

That's not going to work - it is looking for a parameter in the data source called `callback( o )` rather than executing your callback function.

You want something like:

[code]
"dataSrc": function ( json ) {
return {
recordsTotal: json.response.numFound,
recordFiltered: json.response.numFound,
data: json.response.docs
};
}
[/code]

Allan

SOLR_JOHN

I tried your suggestion and it feels like we're getting closer, but DT returned "no matching records found". That tells me it's not accessing the data correctly. Am I supposed to parse the JSON somehow to point at those values or does DT handle it somehow?

[code]
$('#example').dataTable( {
"processing":true,
"serverSide":true,
"ajax": {
url": ""solr-url",
"data": {'wt':'json', 'q':'*:*', 'rows':'10'},
"dataType": "jsonp",
"jsonp":"json.wrf",
"dataSrc": function (json) {
return {
recordsTotal: "response.numFound",
recordsFiltered: "response.numFound",
data: "response.docs"
};
}
},
"aoColumns": [
{"data" : "company" },
{"data" : "address" , "sDefaultContent": "" }
]
} );
[/code]

Thanks again for your help.

allan

> "response.docs"

Why are you giving it strings? The data is in the `json` object. Use the object structured like I suggested above and it might just work.

Edit - I see I had a typo where I used `data` rather than `json` - fixed now. But still don't use strings :-)

Allan

SOLR_JOHN

Thanks for the response. I did actually notice the typo prior to my last post. I did try with just the json object and the corresponding json path but no luck. I can't see what else could be missing. The data does return successfully if I just gave it "dataSrc: response.docs" but in the function json format, it doesn't even return any data back. Let alone the records total and filtered.

[code]
$('#example').dataTable( {
"processing":true,
"serverSide":true,
"ajax": {
url": ""solr-url",
"data": {'wt':'json', 'q':'*:*', 'rows':'10'},
"dataType": "jsonp",
"jsonp":"json.wrf",
"dataSrc": function (json) {
return {
recordsTotal: json.response.numFound,
recordsFiltered: json.response.numFound,
data: json.response.docs
};
}
},
"aoColumns": [
{"data" : "company" },
{"data" : "address" , "sDefaultContent": "" }
]
} );

[/code]

allan

I would need to be able to see the page to be able to understand what is going wrong. Can you give me a link?

Allan

phax

Btw. there is a quoting error in [code]url": ""[/code]
I assume it should be
[code]"url": "[/code]

allan

That'd do it... Eagle eyes :-)

Allan

SOLR_JOHN

Sorry for my super delayed response.

The typos are real. But that isn't the reason why the code is not working. I've been working offline, so I definitely have a few (a lot) of typos as I transfer some of my sample code.

I've taken the time to set it up using a solr indexer from berkley and the live datatables (thank you for providing it).

The first example shows the result when I have "dataSrc": "response.docs". And the results (as you can see there data being returned).

http://live.datatables.net/yinobuz/1/edit?html,css,js,output

The 2nd example shows the result when I use "dataSrc" : function(json). As you can see, nothing gets returned back.

http://live.datatables.net/gotucum/6/edit

If anyone has time to take a quick look at this for me and figure out what we can do to get this puppy up and running, I would owe a great debt of gratitude.

SOLR_JOHN

Any feedback on this one?

allan

Sorry about this - I lead you down the garden path! I thought I'd abstracted this out a bit, but it turns out not - specifically this is the issue here: https://github.com/DataTables/DataTablesSrc/blob/master/js/core/core.ajax.js#L295 .

The data source is evaluated _after_ the length parameters are parsed, and it is expecting the data array to be returned from it, not the full source object.

In which case, you need to make your own Ajax call: http://live.datatables.net/gotucum/9/edit

Allan

SOLR_JOHN

Hi Allan,

Thanks so much for your help and patience on this one. I was trying to write my own callback at some point but I just don't know jquery syntax well enough to get there. Your example code looks great and I'm excited to finally use datatables for my solution even if it means writing my own ajax call. I'll need to take some time to learn more client side programming as it's powerful stuff!

Keep up the great work, and I'm looking to donate some money for this awesome support and development. Perhaps this specific case will be covered in a future release of DT?

allan

Possibly yet. I'm not 100% sure how at the moment, as dataSrc needs to stay as it currently is, for backwards compatibility. So it would need a new mechanism. There has been a request for being able to change the server-side processing property names before, I just need to work out a clean and simple way of doing it!

Allan