What may happen when I crawl MILLIONS of files in MOSS/MSS? Part II – Why I need X64 instead of X86?


In last post of this series we talked about the crawl time and and CPU usage. This time we will talk about process memory usage and x86/x64 issue.

Many customers asked me questions about x86/x64 comparison. Most of them consider x64 would be a benefit, but they don't know what kind of benefit it really brings.

So now I tell you why it's needed.

As we know, SharePoint/search server has three layers. Each layer can only be single architecture.

Layer 1: Web Front Server (WFE)

WFE is needed to host the web site, process different user events. This layer, of course can be and should be x64, unless you have some pretty old webpart which still need x86. At this layer, the most memory consummation comes from IIS(w3wp.exe). For IIS in X86, it can only use at about 1.1-1.2G memory. If you hit this barrier, the process may just hang there.  This situation happens when very big number of request lasted for a long time(several hours or days, it depends on how many users are accessing the site at the same time). You can have multiple WFE in one server farm.

It's quite important to have IIS recycle automatically, and sometimes you even need to manually recycle it.

Layer 2: Query Server.

Query Server hosts query engine. It continually receive index propaganda from index server. When users make a query, it will be sent to query engine. Query engine will check SQL Server for document properties, and check index for content chunk. So the disk performance is important for query server when query load is high.  Other workload, like query time security trimming by custom security trimmer, is also done by Query Servers.

You can have multiple Query Server per farm. And Query Servers, should be x64 if you have the hardware.

Layer 3: Index Server.

Index Server is the spider. Because you can only have one Index Server currently in SharePoint 2007/Search Server 2008, you need to take great care of it. DO NOT put ANY other applications on it, DO NOT share the box with SQL. If you do, you will quickly be hit by the bad performance.

Yes, Index Server should be x64. The problem is, sometimes we cannot make it x64. For example, we have a 32bit ifilter which does not have any x64 implementation but is very important to customer business, or we only have 32bit protocol handler like Lotus Notes PH... In these situations, you can only have a 32bit Indexer.

You may suffer from the same limitation when using 32bit Indexer, especially in Lotus Notes. When you continues crawl too many Notes docs into SharePoint, index engine may hang there for several hours because of memory limit. I suggest, if you happen to come across such problem, make a simple application to monitor and automatically restart search service after one DB is finished. This can manually recycle the memory used by the engine.

In the future, nearly everything can be made x64. So stay tuned.

Database: SQL Server

No matter querying or indexing, the backend database is always a main contributor to the whole performance. So make it independent, make it faster, and even make it cluster. This will help with overall performance. x64 is the best choice for DB.

Skip to main content