What Actually Happens When You Search a SolrCloud Collection

Every SolrCloud tutorial explains that a collection is split into shards and each shard has replicas. Almost none of them explain what the cluster is actually doing during a search — and that gap is where most “why is this slow” and “why are these results wrong” questions live.

Here is the version I wish I’d been given: the fan-out, the two round trips nobody mentions, the response flags that quietly lie to you, and the scoring bug that hides in every tenant-sharded cluster.

Whoever answers the phone does the work

There is no dedicated coordinator node in SolrCloud. When a search request arrives, the node that happens to receive it becomes the aggregator for that request. It asks one replica of every shard, merges the answers, and returns a single response. Your client never sees any of it.

Two consequences fall out of this immediately. First, a Solr node can answer as long as it reaches at least one replica of every shard — lose the last replica of any single shard and the whole search fails by default. Second, every node is simultaneously a front desk and a floor, which is exactly how a cluster can deadlock. More on that at the end.

A search is two round trips, not one

This is the part the reference guide never quite spells out, and it explains half the parameters on the page.

You asked for the top 10 results. Shard 1 cannot know whether its best document beats shard 3’s best document — it has never seen shard 3’s data. So every shard must return candidates before anyone can decide the real top 10. But shipping 10 full documents from every shard, when 20 of the 30 will be discarded, is wasteful. So Solr splits the work in two.

Which tells you exactly when distrib.singlePass=true is a good idea: when you return a handful of small fields. With large stored fields and ten shards you would ship 100 documents across the network to display 10. Note also that faceting still makes its own refinement round trips, so it never guarantees “one request per shard”.

status: 0 does not mean “complete”

Two independent things can go wrong, and people conflate them constantly. The node might not reach ZooKeeper — so it is working from a possibly stale cluster map. Or a shard might have no reachable replica — so data is genuinely missing from your answer.

Solr reports both in the response header, and here is the trap: all three of these responses carry "status": 0 and an HTTP 200.

By default shards.tolerant is false and a missing shard fails the request. Setting it to true buys you partial answers; setting it to requireZkConnected goes the other way and refuses to answer at all when ZooKeeper is unreachable. When something is wrong, add shards.info=true; the guide says distributed responses then “include information about the shard”, and in practice that is what lets you see which shard failed or lagged. The exact fields returned are not enumerated in the reference guide, so check them against your own Solr version rather than trusting any blog (including this one).

Comma widens, pipe substitutes

The shards parameter syntax looks fiddly until you learn the two rules behind it. A comma means “and this shard too”. A pipe means “or this copy instead”. Commas add breadth; pipes offer alternative servers for the same slice of data.

shards=shard1                          → one shard, random replica of it
shards=shard1,shard2                   → two shards  (comma widened the search)
shards=host1/solr/coll,host2/solr/coll → two shards, you named the machines
shards=host1/solr/coll|host2/solr/coll → ONE shard, two candidate copies
shards=shard1,host2/solr/coll|host3/…  → two shards; second one has a shortlist

Count the commas first and you know how many shards you are about to search. The related parameter shards.preference is a multi-column sort over the replicas of each shard — replica.type, replica.location, replica.leader, node.sysprop, written left to right in order of importance. It never excludes anything, so it can never cause a failure; it only changes who gets asked first.

One preference is worth singling out. The default tie-breaker is random, which spreads load beautifully and wrecks your caches: with three replicas, the same query gets computed and cached three separate times. replica.base:stable:hash:sessionId hashes a parameter instead, so the same user keeps landing on the same copy and their cache stays warm.

The scoring bug hiding in every tenant-sharded cluster

This is the subtlest thing in distributed Solr and the one most likely to be quietly ruining your relevance.

When Solr scores a document it weighs each term by how rare it is — inverse document frequency. That is what stops the from outranking photosynthesis. Solr’s default similarity is BM25Similarity (the schema docs state it is used implicitly for any field type without an explicit similarity), and Lucene’s BM25Similarity computes the term weight as idf = ln(1 + (N − n + 0.5) / (n + 0.5)) — N being the number of documents in the index and n how many contain the term. The formula is Lucene’s, not something the Solr reference guide restates.

And there is the problem: in SolrCloud, “the index” means one shard. Both N and n are local. The figures below are a worked illustration, not a benchmark — I invented the corpus, then ran the real formula over it, so the arithmetic is reproducible even though the scenario is made up:

Two reassurances before the fix. This changes scores only, never which documents match — numFound, filters and facet counts stay correct, which is precisely why it survives testing for so long. And if you sort by a field rather than by relevance, none of it applies.

Whether you have the problem comes down to routing. Hash routing on a random ID scatters documents evenly, every shard sees the same term distribution, and the default is fine — that is the case it was designed for. But if you shard by tenant, customer, language, source or date, your shards have genuinely different vocabularies and your relevance is skewed. That is worth sitting with, because composite routing by tenant is one of the most commonly recommended SolrCloud patterns.

The fix is to configure a global statsCache in solrconfig.xml, which gathers document and term statistics from every shard before scoring so all shards work from the same numbers:

<statsCache class="org.apache.solr.search.stats.ExactSharedStatsCache"/>

Four implementations ship with Solr. LocalStatsCache is the default and the one you are using if you never configured this. ExactStatsCache is the correctness benchmark, but it costs an extra network round trip on every query — the two-phase dance becomes three. ExactSharedStatsCache gets the same accuracy while reusing stats for repeated terms, which makes it the better production choice for most people. LRUStatsCache bounds the memory for very large vocabularies.

Then trim the waste: add distrib.statsCache=false to request handlers that never score — ID lookups, filter-only queries, field-sorted listings. They would otherwise pay for statistics they never read. And re-check your boosts afterwards: switching to global statistics changes every score in the collection, so anything you tuned previously was tuned against the distortion.

And the failure that looks like a hang

Back to that dual role. Every node serves top-level requests and serves sub-requests from other aggregators, both out of the same HTTP thread pool. Give two nodes one thread each, send each of them a top-level query at the same time, and both threads are occupied — each waiting on a sub-request the other has no thread left to serve. Circular wait. Nothing crashes, nothing logs an error; requests just stop finishing and the cluster looks “slow” rather than broken.

The guide’s own rule: “ensure that the max number of threads serving HTTP requests is greater than the possible number of requests from both top-level clients and other shards.” So size the pool for both kinds of traffic, not just the queries you can see arriving from outside.

The short version

1. Whoever gets the request does the coordinating. There is no special node — and that dual role is why deadlock is possible.
2. A search is normally two round trips, not one. Almost every performance question traces back to this.
3. You need one live replica of every shard. Lose the last one and the whole search fails by default.
4. status: 0 does not mean complete. Only zkConnected and partialResults tell you that — so log them.
5. Comma widens, pipe substitutes. That decodes the entire shards syntax.
6. shards.preference is a sort, not a filter. It can never cause a failure.
7. Shards score with their own statistics by default. If you shard by tenant, language or source, your relevance is skewed and nothing will tell you.

Where each claim comes from

Everything above is drawn from the Apache Solr Reference Guide. Because a few of the more useful points are behaviour the guide implies rather than states, here is the split, so you can check rather than trust:

Solr behaviour varies by release — this was checked against the latest reference guide, so confirm anything load-bearing against the version you actually run.