Here's a quick rundown of what you need to get started writing a custom channel for WCF. It doesn't matter whether you're writing a layered channel or transport, everything here is good to know about.
Tools You Need
Everyone needs the WinFX runtime for running applications. If you have beta of Windows Vista, this may already be installed on your machine. I'd recommend using the Microsoft Pre-Release Software WinFX Runtime Components - February Community Technology Preview (CTP) version until a newer CTP comes along.
You will need the Windows Platform SDK to get documentation and samples. The matching SDK is the Microsoft® Windows® Software Development Kit (SDK) for the February 2006 Community Technology Preview (CTP) for Windows Vista and WinFX Runtime Components. Yes, that's really the name of it.
I would strongly recommend having Visual Studio 2005. The reference documentation for Windows Vista in the February CTP is incomplete, making it very important to have Intellisense support. You can get the Microsoft Visual Studio Code Name “Orcas” Community Technology Preview - Development Tools for WinFX® tools addon, although I personally don't use this for development.
Blogs to Read
I'm assuming that you've found my blog. You should look through the Samples and Channels categories for information about writing a custom channel. Especially useful is the custom transport walkthrough overview, which is still helpful even if you're writing a layered channel instead of a transport.
Forums to Post On
There's really only one that I've seen get a lot of participation. Ask your questions on the Windows Communication Foundation ("Indigo") forum on forums.microsoft.com.
Samples to Look At
There's three more samples you want to look at when writing a custom channel.
Get the WSE 3.0 TCP Interop sample from windowscommunication.net. This is a custom transport that is basically TCP with a few framing tweaks.
Get the UDP sample from the Windows SDK. After installation, look in the SDK at Microsoft SDKs\Windows\v1.0\samples for a zip file called AllWinFXsamples.zip. The UDP sample is inside the zip file at WindowsCommunicationFoundation\TechnologySamples\Extensibility\Transport\Udp\CS. This is a very complete custom transport implementation.
Get the Chunking and Streaming Custom Channel sample from windowscommunication.net. This is a layered channel that supports fragmenting and reassembling messages.
Next time: Basics of Encryption and Hashing