[Update: .NET v4 solves this problem in an elegant way by leveraging the Dynamic Language Runtime. See BrowserInteropHelper.HostScript. The browser script integration works in Firefox too.]
When an XBAP is hosted in an HTML [i]frame, it has limited means to communicate with script in the host page. These are: passing parameters to the startup URL (which can be read from Application.StartupUri or BrowserInteropHelper.Source); exchanging HTTP cookies (Application.Get/SetCookie()); and, as a last resort, use a web service at the site of origin as an intermediary. The biggest obvious limitation of these methods is that they are passive: you don't have the equivalent of a method call. Thus, a commonly requested feature by XBAP developers is the ability to directly communicate with HTML script in the page hosting the XBAP. Until such feature is provided, there is a somewhat unobvious way to do this, but only from a full-trust XBAP and only in Internet Explorer.
The basic idea is this: Crawl the browser hosting container hierarchy starting from BrowserInteropHelper.ClientSite; get to the IWebBrowser2 of the frame hosting the XBAP; then get to the HTML DOM window object, and finally get its Script object, which represents the scripting environment of the HTML page. Having that object, "the magic" of COM Interop makes script function calls and parameter passing possible. On the managed side, you just do Reflection (late-bound) calls. Unfortunately, this is pretty ugly in C#, but with some helper methods it can be covered up. Since VB.NET has built-in support for late-bound method calls ("scripting"), the calling syntax can be fairly clean.
For HTML script to "call into" the XBAP can be arranged a little indirectly: The XBAP passes a 'scriptable object' to its host page, and then the late-bound calls from script are translated to managed method calls by COM Interop again, just in the opposite direction (via CCWs). The scriptable object needs to be marked with[ComVisible(true)] and expose some public API.
The attached project is a rudimentary demonstration of this technique. Incidentally, it also has an example of other interesting things you can do once you have IE's main interfaces: closing the tab/window in which the XBAP is loaded. (This is something for which there is no managed API. The XBAP can be running top-level in the browser, not in a frame.)
A note on the debugging setup: In Project Properties, the debugging option needs to be set to Start Browser with URL, pointing to HTMLPage1.htm in the bin\Debug output folder. (This setting is per user, so it's not saved in the attached project.) The page hosts the XBAP in an iframe. This is enough for Visual Studio 2008 to be able to start debugging the XBAP, even though it's launched indirectly. (The VS project system has some intrinsic knowledge of XBAPs and coordinates the launch with PresentationHost.exe, the XBAP host process, via the -debug and -event parameters.)
[Update, 10/14/09] On initial loading of the HTML page in IE, script execution will likely be blocked, with an explanatory message in the information bar (aka ‘goldbar’). You can click on the prompt and let IE enable running script, but this will start another instance of the XBAP. You could attach to it then with the VS debugger (attach to the running PresentationHost.exe). This gets tedious after a while. Just for debugging, you could let script run automatically by selecting the option to ‘Allow active content to run in files on my computer’ in Internet Options > Advanced tab > Security section. It is risky to leave the browser running with this setting, so remember to reset it. The script blocking won’t happen if you open the host HTMP page from a UNC path or HTTP URL, but unfortunately F5 debugging of the XBAP becomes impractical then.
Another issue you might run into is that the host HTML page will load without the XBAP in it. There may be different reasons for such mishap, but a likely one with the sample is that your anti-virus program sneakily opens the .zip file and puts the Zone.Identifier alternate data stream on the files in the archive. Use of Scott’s methods to remove the stream.
Finally, to run an XBAP with full trust, the easiest way is to just run it from the local file system, which is the case of debugging in Visual Studio. If you want to run it non-locally, you have to use the ClickOnce Trusted Application Deployment model, which entails installation of the application's signing certificate on the client computers. (This has been discussed broadly on the web.)