How to debug solr exceptions/warnings for Alfresco (Part 2)

In this post I dig a bit deeper in how you can debug solr exceptions/warnings. In part 1 of this blog series we looked briefly at the problem where we got a timeout and how we can fix this in the general case.

This post will show how to get more information about what could be the problem causing these errors.

We will look at an example where we get the following error in the solr log, however the way of debugging would be the same or similar for other kinds of errors:

2015-04-28 15:02:50,413 WARN  [org.alfresco.solr.tracker.CoreTracker] Node index failed and skipped for 8727989 in Tx 9223372036854775807
org.json.JSONException: Unterminated string at character 1866
        at org.json.JSONTokener.syntaxError(JSONTokener.java:413)
        at org.json.JSONTokener.nextString(JSONTokener.java:244)
        at org.json.JSONTokener.nextValue(JSONTokener.java:344)
        at org.json.JSONObject.(JSONObject.java:206)
        at org.json.JSONTokener.nextValue(JSONTokener.java:347)
        at org.json.JSONArray.(JSONArray.java:125)
        at org.json.JSONTokener.nextValue(JSONTokener.java:351)
        at org.json.JSONObject.(JSONObject.java:206)
        at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:774)
        at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:2376)
        at org.alfresco.solr.tracker.CoreTracker.reindexNodes(CoreTracker.java:1057)
        at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:566)
        at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

In this warning in the log we get no real information about what is causing the error, just that it concerns a node with id 6899573 in the transaction 9223372036854775807 and that it has something to do with parsing a json.

So the natural next step would be to fetch information about the node which we in this case will do by manually fetching the JSON the same way as Solr does.

If we look at SOLRAPIClient.java:getNodesMetaData it is obvious that Solr makes a request to api/solr/metadata to fetch metadata about a node. This will return a Json string with the metadata for an object.

We want to do the same for this particular failing node which means that we need to construct a Json request to http://localhost:8080/alfresco/service/api/solr/metadata.

By default the solr endpoints are protected by SSL. To make things easier to debug you can disable SSL on Solr by following the guide in the Alfresco Documentation: http://docs.alfresco.com/4.2/tasks/running-without-ssl.html.

I used the Chrome plugin Postman for performing this action, but curl or any other tool capable of making a REST Post request will do.

The request looks as follows:

POST http://localhost:8080/alfresco/service/api/solr/metadata
Content-Type: application/json
Body:
{
fromNodeId: 8727989,
toNodeId: 8727989,
maxResults: 1
}

And the response is:

{
    "nodes": [
        {
            "id": 8727989,
            "tenantDomain": "",
            "nodeRef": "workspace://SpacesStore/2777063a-1beb-41c6-bf96-9882ec740de6",
            "type": "custom:document",
            "aclId": 3570765,
            "txnId": 10801744,
            "properties": {
                "cm:modified": "2014-05-27T13:04:19.009Z",
                "cm:content": {
                    "contentId": "5285762",
                    "encoding": "UTF-8",
                    "locale": "sv_",
                    "mimetype": "application/pdf",
                    "size": "60978"
                },
                "sys:node-uuid": "2777063a-1beb-41c6-bf96-9882ec740de6",
                "cm:name": "Test \u0015 document.pdf",
                "sys:locale": "sv_",
                "sys:store-protocol": "workspace",
                "cm:autoVersion": "true",
                "sys:node-dbid": "8727989",
                "vgr:dc.source.origin": "Alfresco",
                "cm:initialVersion": "true",
                "cm:creator": "admin",
                "cm:created": "2014-05-27T13:04:19.009Z",
                "cm:title": [
                    {
                        "locale": "sv_",
                        "value": "Test \u0015 document.pdf"
                    }
                ],
                "cm:autoVersionOnUpdateProps": "true",
                "cm:modifier": "admin",
                "sys:store-identifier": "SpacesStore"
            },
            "aspects": [
                "cm:auditable",
                "sys:incomplete",
                "sys:referenceable",
                "cm:titled",
                "sys:localized",
                "cm:versionable"
            ],
            "paths": [
                {
                    "path": "/{http://www.alfresco.org/model/application/1.0}company_home/{http://www.alfresco.org/model/site/1.0}sites/{http://www.alfresco.org/model/content/1.0}test/{http://www.alfresco.org/model/content/1.0}documentLibrary/{http://www.alfresco.org/model/content/1.0}Test_x0020__x0015__x0020_document.pdf"
                }
            ],
            "ancestors": [
                "workspace://SpacesStore/627af57f-566f-4389-a19e-1f3aa71124dc",
                "workspace://SpacesStore/fae70774-af0e-4ca5-890e-cc2e19414d07",
                "workspace://SpacesStore/9f9e873e-4590-40ac-b8a5-54300938354f",
                "workspace://SpacesStore/dfcd625a-af5e-4eee-90c8-1c2333a8fac5",
                "workspace://SpacesStore/98f20817-27c1-4adb-acdb-bc825059d582",
                "workspace://SpacesStore/741b92bb-80a5-4fd5-b116-741523513776",
                "workspace://SpacesStore/403b9392-43ca-496a-94f9-581482f49fdb"
            ],
            "parentAssocs": [
                "workspace:\/\/SpacesStore\/627af57f-566f-4389-a19e-1f3aa71124dc|workspace:\/\/SpacesStore\/2777063a-1beb-41c6-bf96-9882ec740de6|{http:\/\/www.alfresco.org\/model\/content\/1.0}contains|{http:\/\/www.alfresco.org\/model\/content\/1.0}Test \x15 document.pdf|true|-1"
            ],
            "parentAssocsCrc": 1921797508,
            "owner": "admin"
        }
    ]
}

I used a Json validator (http://jsonformatter.curiousconcept.com/ and copy-pasted the json what the validation result as). It complained about an invalid character in the following part of the Json and what we can spot here is that in the document name an utf-8 character has not been correctly escaped.

"parentAssocs":[  
            "workspace:\/\/SpacesStore\/627af57f-566f-4389-a19e-1f3aa71124dc|workspace:\/\/SpacesStore\/2777063a-1beb-41c6-bf96-9882ec740de6|{http:\/\/www.alfresco.org\/model\/content\/1.0}contains|{http:\/\/www.alfresco.org\/model\/content\/1.0}Test \x15 document.pdf|true|-1"
         ],

Comparing the file name from the metadata we see that it should be “Test \u0015 document.pdf”, whereas in the parentAssocs part it says “Test \x15 document.pdf” which is not a valid escape string in Json. Further investingation showed that this is a bug in solr.lib.ftl which comes from that Freemarker does not translate unicode characters below 0×20. Quick fix, remove the special character from the filename (which probably should not be there anyway) and reindex the node. The issue has been reported to Alfresco Support and if it becomes a Jira issue I’ll link it in this article.

I made a patch to solr.lib.ftl which fixes this issue and when running the same REST call again and then we get a valid Json back from the metadata endpoint:

{
    "nodes": [
        {
            "id": 8727989,
            "tenantDomain": "",
            "nodeRef": "workspace://SpacesStore/2777063a-1beb-41c6-bf96-9882ec740de6",
            "type": "custom:document",
            "aclId": 3570765,
            "txnId": 10801744,
            "properties": {
                "cm:modified": "2014-05-27T13:04:19.009Z",
                "cm:content": {
                    "contentId": "5285762",
                    "encoding": "UTF-8",
                    "locale": "sv_",
                    "mimetype": "application/pdf",
                    "size": "60978"
                },
                "sys:node-uuid": "2777063a-1beb-41c6-bf96-9882ec740de6",
                "cm:name": "Test \u0015 document.pdf",
                "sys:locale": "sv_",
                "sys:store-protocol": "workspace",
                "cm:autoVersion": "true",
                "sys:node-dbid": "8727989",
                "vgr:dc.source.origin": "Alfresco",
                "cm:initialVersion": "true",
                "cm:creator": "admin",
                "cm:created": "2014-05-27T13:04:19.009Z",
                "cm:title": [
                    {
                        "locale": "sv_",
                        "value": "Test \u0015 document.pdf"
                    }
                ],
                "cm:autoVersionOnUpdateProps": "true",
                "cm:modifier": "admin",
                "sys:store-identifier": "SpacesStore"
            },
            "aspects": [
                "cm:auditable",
                "sys:incomplete",
                "sys:referenceable",
                "cm:titled",
                "sys:localized",
                "cm:versionable"
            ],
            "paths": [
                {
                    "path": "/{http://www.alfresco.org/model/application/1.0}company_home/{http://www.alfresco.org/model/site/1.0}sites/{http://www.alfresco.org/model/content/1.0}test{http://www.alfresco.org/model/content/1.0}documentLibrary/{http://www.alfresco.org/model/content/1.0}Test_x0020__x0015__x0020_document.pdf"
                }
            ],
            "ancestors": [
                "workspace://SpacesStore/627af57f-566f-4389-a19e-1f3aa71124dc",
                "workspace://SpacesStore/fae70774-af0e-4ca5-890e-cc2e19414d07",
                "workspace://SpacesStore/9f9e873e-4590-40ac-b8a5-54300938354f",
                "workspace://SpacesStore/dfcd625a-af5e-4eee-90c8-1c2333a8fac5",
                "workspace://SpacesStore/98f20817-27c1-4adb-acdb-bc825059d582",
                "workspace://SpacesStore/741b92bb-80a5-4fd5-b116-741523513776",
                "workspace://SpacesStore/403b9392-43ca-496a-94f9-581482f49fdb"
            ],
            "parentAssocs": [
                "workspace://SpacesStore/627af57f-566f-4389-a19e-1f3aa71124dc|workspace://SpacesStore/2777063a-1beb-41c6-bf96-9882ec740de6|{http://www.alfresco.org/model/content/1.0}contains|{http://www.alfresco.org/model/content/1.0}Test \u0015 document.pdf|true|-1"
            ],
            "parentAssocsCrc": 1921797508,
            "owner": "admin"
        }
    ]
}

The steps outlined here can be used on other kind of errors/warnings as well. Hopefully it will be of use to someone!

How to debug solr exceptions/warnings for Alfresco (Part 1)
How to debug solr exceptions/warnings for Alfresco (Part 3)

This entry was posted in Alfresco, Solr. Bookmark the permalink.

4 Responses to How to debug solr exceptions/warnings for Alfresco (Part 2)

  1. Alexandra says:

    C’est vrai que certains sont vraiment incroyable, mais j’avoue que je ne m&u;arosqttarde plus.Au final, quand on voit le nombre de visiteurs de mon blog, j’ai de la chance, car j’ai pas trop de troll pour le nombre de visiteurs!

  2. Carmelo1979 says:

    Hello blogger, do you monetize your website ? There is easy way to
    earn decent money every month, just search on youtube : How to earn $25/hour selling articles

  3. Geotab GPS says:

    Hello colleagues, its wonderful piece of writing
    regarding tutoringand fully defined, keep iit up
    alll the time.

  4. 76Francisco says:

    Hello blogger, i must say you have hi quality content here.

    Your blog should go viral. You need initial
    traffic only. How to get it? Search for; Mertiso’s tips go viral

Leave a Reply to Alexandra Cancel reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>