Have you worried about Reporting service constanly keeps consuming the physical memory even when the reporting service is not under stressful condition? This specifically happens when you have NUMA architecture enabled on the Server.
So, if you've NUMA architecture enabled and you see yourself running in to this issue, please don't be surprised. The reason is,
It’s a behavior where we just commit memory upto 60% of the configured WorkingSetMaximum, Which by default is equivalent to the available physical memory. The important thing to understand here is we are just committing the memory for ReportingServices, but not using it.
If the system is under memory crunch and there are other processes that require more memory, we immediately release the unused, committed memory, ie.., if there is any memory pressure, we shrink back to the memory amount which Reporting service is currently utilizing.
This behavior should not affect performance/scale of either ReportingServices or any other external process.
But if you feel uncomfortable with this behavior, you need to instruct Reporting service in such a way that it stops assuming the entire available memory belongs to it. By default it does assume.
This can be achieved by seeting the memory limits in rsreportserver.config. You can refer http://msdn.microsoft.com/en-us/library/ms159206(v=sql.105).aspx for this.
You can limit the memory utilization by limiting the value of WorkingSetMaximum and WorkingSetMinimum.
The above article provides some great insight as to how reporting services handles memory and how it reacts to memory pressure. I highly recommend you to carefully go through this article before making any changes.
[All the posts are AS-IS without any warranty]