SharePoint 2013 Search Service Activation Error
I was configuring a new SharePoint 2013 Search Service when everything became stuck on service toplogy activation. It doesn’t matter whether I tried to configure Search using PowerShell or Central Administration the search service topology simply won’t activate.
Search service & all its associated databases were created successfully but when I tried to activate a new search topology, created by cloning an existing topology, commands will fail.
My SharePoint 2013 environment
I was configuring a new Search Service Application on Windows Server 2012 Standard, SQL Server 2012 and SharePoint Server 2013 SP1. It was a single server SharePoint environment with SQL installed elsewhere. Servers have been joined to corporate domain & have Windows security features like Firewall etc activated.
Error messages in ULS logs
ULS logs were filling up with following error messages:
- “Content Plugin can not be initialized - list of CSS addresses is not set.”
- (Note: This error starts to appear in ULS logs as soon as PowerShell command “New-SPEnterpriseSearchServiceApplication” executes. It goes away once your search topology configuration is complete i.e. you successfully activate a search topology)
- SharePoint 2013 Skipping Topology Synchronize.The active topology does not have any Components.
- Could not connect to system manager, probably not started yet. exception: no system manager locations set, search application might not be ready yet.
PowerShell command execution errors
Following error appears when you try to activate a new search topology using PowerShell. Command waits for few minutes & ends with this error.
“Topology activation failed. No system manager locations set, search application might not be ready yet.”
Central Administration Search Service Application
This appears in Search Service Application administration page in Central Administration.
“Unable to retrieve topology component health states. This may be because the admin component is not up & running”
So what all these error are telling you? One thing is obvious; all above errors mean your Search Service Application isn’t configured properly. So what went wrong?
“Noderunner.exe” is what went wrong
All five components of SharePoint 2013 search – Admin, Crawl, Query Processing, Analytics Processing, and Content Processing - are run within a container process called “noderunner.exe”. If this process isn’t starting or working properly, your Search Service won’t be running either.
How do I find out noderunner.exe is loading & running?
First things first: Fire-up “Run --> Services.msc” & make sure following are running
- SharePoint Search Host Controller
- SharePoint Server Search 15
If these are running normally, then move to next step
“Process Monitor” tool is your friend in this case. Download the process monitor & fire it up on your SharePoint server where Search Service is being configured. Information presented can be overwhelming if you haven’t used Process Monitor before, but be patient & press “Ctrl + T” inside Process Monitor window to launch the “Process Tree Viewer”.
In process tree viewer, track down the “hostcontrollerservice.exe”. This service must have five child processes “noderunner.exe” under it (One child noderunner.exe for each search component). Basically process tree must look like screenshot 1 below. Timeline in screenshot indicates that child processes “noderunner.exe” have been running since its parent “hostcontrollerservice.exe”.
No “noderunner.exe” is running
If you don’t see any “noderunner.exe” process under your “hostcontrollerservice.exe” then you know something is wrong. Inside Process Monitor tool, uncheck the box “Only show processes at the end of current trace”. “Process Tree” window would refresh & this time will show processes that have been terminated recently.
If you still don’t see any “noderunner.exe”, then restart your “SharePoint Search Host Controller’ service. Once restarted, refresh the “Process Tree” view by checking & un-checking the “Only show processes at the end of current trace” box.
What do you see?
You should see a time line like screenshot 2. What this timeline indicates is that “noderunner.exe” is started but for some reason ends immediately.
From above diagnosis, you have tracked down the Search Service configuration problem to “noderunner.exe” not being working correctly & ending as soon as it starts.
So what are the solutions?
Possible solutions suggested on the Internet are
Memory configuration of Node Runner:
Lot of blogs suggests changing memory configuration of “noderunner.exe” process by going to c\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config & changing the value for “<nodeRunnerSettings memoryLimitMegabytes="0" />” to something like 100Mb or 512Mb or 1024Mb.
Don’t do it!
The memory problem existed but has been fixed by Microsoft & shouldn’t be an issue at all in SP1.
Again some articles, suggests installing patches. Not that easy, SP1 contains all the fixes & these patches aren’t going to help you.
Install .Net 3.5 features
An article I found recommended installing .Net 3.5 features but that seem to do nothing.
So what should we do?
Dig into your ULS logs.
- Change diagnostic logging settings for “Search” & “SharePoint Foundation Search” to verbose in Central Administration
- Fire up the ULS Log Viewer & view the ULS Real time feed
- Restart the “SharePoint Search Host Controller” service
- Wait for a while & search for “noderunner”, you should see entries relating to this process being started as indicated in Screenshot 3 below. Once you find such an entry, start to pay attention. Basically you want to track down anything that’s causing this process to fail & exit. E.g. in my case it was the following error which indicated some sort of permission issue
Failed to configure component Microsoft.Ceres.CoreServices.Services.Configuration.ComponentConfigurationException: Access to the path '/root' is denied
Solution for permission problem
Make sure SharePoint Search Service Account have Full Control access to all Index Partitions
From error indicated above, it certainly looks like some sort of file system permission rather than SQL. Hence try the following & everything should be good.
- Change service account of these two services to a server administrator account
- SharePoint Search Host Controller
- SharePoint Server Search 15
- Restart the services
- Recreate the Search Service Application; Delete the existing one & create new
- Create a new search topology & activate it
- Your activation should now succeed
Change Search Service account
Once search topology has been activated successfully, change the service account for both the services, back to a non-admin account one you should use specifically for search.
Now, if you are a perfectionist, you could re-create the Search Service Application again & everything should work smoothly.