SharePoint 2013 Search Service Activation Error

I was configuring a new SharePoint 2013 Search Service when everything became stuck on service toplogy activation. It doesn’t matter whether I tried to configure Search using PowerShell or Central Administration the search service topology simply won’t activate.

Search service & all its associated databases were created successfully but when I tried to activate a new search topology, created by cloning an existing topology, commands will fail.

My SharePoint 2013 environment

I was configuring a new Search Service Application on Windows Server 2012 Standard, SQL Server 2012 and SharePoint Server 2013 SP1. It was a single server SharePoint environment with SQL installed elsewhere. Servers have been joined to corporate domain & have Windows security features like Firewall etc activated.

Error messages in ULS logs

ULS logs were filling up with following error messages:

  • “Content Plugin can not be initialized - list of CSS addresses is not set.”
    • (Note: This error starts to appear in ULS logs as soon as PowerShell command “New-SPEnterpriseSearchServiceApplication” executes. It goes away once your search topology configuration is complete i.e. you successfully activate a search topology)
  • SharePoint 2013 Skipping Topology Synchronize.The active topology does not have any Components.
  • Could not connect to system manager, probably not started yet. exception: no system manager locations set, search application might not be ready yet.

PowerShell command execution errors

Following error appears when you try to activate a new search topology using PowerShell. Command waits for few minutes & ends with this error.

“Topology activation failed. No system manager locations set, search application might not be ready yet.”

Central Administration Search Service Application

This appears in Search Service Application administration page in Central Administration.

“Unable to retrieve topology component health states. This may be because the admin component is not up & running”

Diagnosis

So what all these error are telling you? One thing is obvious; all above errors mean your Search Service Application isn’t configured properly. So what went wrong?

“Noderunner.exe” is what went wrong

All five components of SharePoint 2013 search – Admin, Crawl, Query Processing, Analytics Processing, and Content Processing - are run within a container process called “noderunner.exe”. If this process isn’t starting or working properly, your Search Service won’t be running either.

How do I find out noderunner.exe is loading & running?

First things first: Fire-up “Run --> Services.msc” & make sure following are running

  • SharePoint Search Host Controller
  • SharePoint Server Search 15

If these are running normally, then move to next step

“Process Monitor” tool is your friend in this case. Download the process monitor & fire it up on your SharePoint server where Search Service is being configured. Information presented can be overwhelming if you haven’t used Process Monitor before, but be patient & press “Ctrl + T” inside Process Monitor window to launch the “Process Tree Viewer”.

In process tree viewer, track down the “hostcontrollerservice.exe”. This service must have five child processes “noderunner.exe” under it (One child noderunner.exe for each search component). Basically process tree must look like screenshot 1 below. Timeline in screenshot indicates that child processes “noderunner.exe” have been running since its parent “hostcontrollerservice.exe”.

No “noderunner.exe” is running

If you don’t see any “noderunner.exe” process under your “hostcontrollerservice.exe” then you know something is wrong. Inside Process Monitor tool, uncheck the box “Only show processes at the end of current trace”. “Process Tree” window would refresh & this time will show processes that have been terminated recently.

If you still don’t see any “noderunner.exe”, then restart your “SharePoint Search Host Controller’ service. Once restarted, refresh the “Process Tree” view by checking & un-checking the “Only show processes at the end of current trace” box.

What do you see?

You should see a time line like screenshot 2. What this timeline indicates is that “noderunner.exe” is started but for some reason ends immediately.

Solution

From above diagnosis, you have tracked down the Search Service configuration problem to “noderunner.exe” not being working correctly & ending as soon as it starts.

So what are the solutions?

Possible solutions suggested on the Internet are

Memory configuration of Node Runner:

Lot of blogs suggests changing memory configuration of “noderunner.exe” process by going to c\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config & changing the value for “<nodeRunnerSettings memoryLimitMegabytes="0" />” to something like 100Mb or 512Mb or 1024Mb.

Don’t do it!

The memory problem existed but has been fixed by Microsoft & shouldn’t be an issue at all in SP1.

Installing patches

Again some articles, suggests installing patches. Not that easy, SP1 contains all the fixes & these patches aren’t going to help you.

Install .Net 3.5 features

An article I found recommended installing .Net 3.5 features but that seem to do nothing.

So what should we do?

Dig into your ULS logs.

  • Change diagnostic logging settings for “Search” & “SharePoint Foundation Search” to verbose in Central Administration
  • Fire up the ULS Log Viewer & view the ULS Real time feed
  • Restart the “SharePoint Search Host Controller” service
  • Wait for a while & search for “noderunner”, you should see entries relating to this process being started as indicated in Screenshot 3 below. Once you find such an entry, start to pay attention. Basically you want to track down anything that’s causing this process to fail & exit. E.g. in my case it was the following error which indicated some sort of permission issue

Failed to configure component Microsoft.Ceres.CoreServices.Services.Configuration.ComponentConfigurationException: Access to the path '/root' is denied

Solution for permission problem

Make sure SharePoint Search Service Account have Full Control access to all Index Partitions

From error indicated above, it certainly looks like some sort of file system permission rather than SQL. Hence try the following & everything should be good.

  • Change service account of these two services to a server administrator account
    • SharePoint Search Host Controller
    • SharePoint Server Search 15
  • Restart the services
  • Recreate the Search Service Application; Delete the existing one & create new
  • Create a new search topology & activate it
  • Your activation should now succeed

Change Search Service account

Once search topology has been activated successfully, change the service account for both the services, back to a non-admin account one you should use specifically for search.

Now, if you are a perfectionist, you could re-create the Search Service Application again & everything should work smoothly.

Comments

Lionel Limozin said…
You are my today life saver!

I'm in the same context and encoutered the same errors !

The trick is to not forget to restart windows service (search and controler) after adding some admin privileges to the search service account.... !

Many thanks !
Thanks for the interesting and complete post.

For my case, I've used a dev env optimization script that you can find on this link : https://spdevoptimize.codeplex.com/, and one of the optimizations was reducing the memory limit of noderunner processes. Everything worked fine until I remove my search service application and install it again. When it comes to activating the new topology, the scripts waits for 5 minutes and then stopps. So, restoring the value to "0" solved the issue (and made my day).
Reindert Dk said…
I have exactly the same issue, noderunner.exe starting but immediately stopping. I tried your suggested soltion, but no luck. Any other ideas?

great post anyway, at least it tracked my issue down to being a noderunner issue
Will Chung said…
I found similar ULS log entries to the ones shown in this blog as well as an entry indicating database access issues.


I checked the accounts that the search services were running as and noticed that they were running under two different accounts. I made sure both of the accounts had the usermapping for the SPSearchDBAdmin role on the search databases, as well as temporarily adding the user accounts to the Administrators group on the application servers.

This seems to have done the trick. I was able to then reconfigure the search topology using powershell.


I've run search before without having these issues, but for some reason things aren't getting auto configured like they used to be. Had a lot of these issues after installing the SharePoint CU for Nov 2017.

Popular posts from this blog

VMWare Workstation - Assign a static IP address

Office365: Assigning licenses using Azure AD PowerShell