I recently was helping a client troubleshoot why they couldn't receive mail. The server was running fine, they were able to send out messages and even reboots on the server didn't seem to resolve the issue. After looking through the event logs, etc, we were unable to discover any errors. Then, we found the support article http://support.microsoft.com/default.aspx/kb/153119. It s a great method to simulate an SMTP conversation with Exchange. While the syntax of the responses in the article from Exchange are different thatn Exchange 2007, you can mock up a test EHLO conversation and determine where it is going wrong when receiving on an external receive connector.
As soon as I opened a telnet and tried to do an EHLO command, the server responded 4.3.1 Insufficient System Resources.
This posting goes over how to resolve it.
Basically, you have to adjust the configuraiton file attached to the transport hub to ensure that it either doesn't monitor system resources, or that it has a higher tolerance for when the physical server resources drops below a specified limit. Changing the config file found at C:\Program Files\Microsoft\Exchange Server\Bin\EdgeTransport.exe.config will help with this and allow the server to resume normal functionality.
However it is important to note, that there is a warning in the event log that signals this, not an error. Also, the symptoms are that people will stop receiving SMTP-based mail. Of singificant importance is that the reason that the server stopped receving and processing mail is that the physical server resources were deemed insufficinet for continued Exchange Operations.