How MSMQT behind NLB works

MSMQT supports only one mode for MSMQ protocol -- transactional exactly once in order. Now, how MSMQ does that. The sender sends a TCP request for the destination queue to the port 1801 and establishes the connection. Then all the messages to this queue go through this already established connection. If the TCP connection breaks, the sender establishes the connection again.

If the receive side is actually a set of machines behind NLB, then when the sender sends the request for TCP connection, it is randomly assigned to one of the the machines behind NLB, and then it is a connection between the sender and this particular machine, at least until the connection breaks for some reason. Hence, while a single sender sends to a single queue, only one machine in the group services it, other machines cannot participate. But if the machine goes down, another machine can pick up the work. Because for BizTalk MSMQT the storage is common, if a connection breaks, another machine can get it when the senders reestablishes the connection.

Now, if different sender or different destiation queues are involved, new TCP connection is established for each, and NLB will assign new TCP connections to different machines, so in this case there will be load balancing. Sender A may be connected to the receiving machine 5, while sender B to machine 3, and sender C to the machine 4. So, that would provide scaling if enough senders and destination queues are involved. Still if one of these connections breaks, another machine may be used to reestablish it by NLB when the sender retries. 

Again, so far there is pretty much nothing BizTalk-specific. BizTalk just implements that with fidelity, nothing more.

See https://blogs.msdn.com/eldarm/archive/2004/09/17/230951.aspx on what BizTalk does for that to work.