The more I think about it… NNTP Doesn’t Cut It

So I’ve been posting ideas about how we could best merge NNTP usefulness and feature rich web solutions @ Microsoft.  The 3rd plan I outlined involves cutting ties with NNTP and creating a better offline client that also supports the added features of the ASP.NET 2.0 forums.  I haven’t really given much weight to this plan before since it involves the development of both a new client app that people would have to download and a set of web services to be exposed from the ASP.NET 2.0 forums for people to take advantage of the client.  And besides, windows comes with Outlook Express right?

But the more I (My opinion has also been influenced by feedback from Daniel and others internally)think about it adding NNTP to the ASP.NET 2.0 forums is sort of a hack that ignores most of the benefits for moving beyond NNTP in the first place.  People who wanted an offline, cached, experience could would download the client and the rest would probably be happy with the web support.  If the client and the web services are all open source then users would probably also expand this solution over time as well.  And the web services would be public so anyone could writing their own client as well. 

I do worry, however, about ignoring a standard that’s been around for a long time and replacing it with a set of web services that would, at first, only work with one client.  Though maybe we can convince OE to support them in Longhorn.  🙂

Comments (22)

  1. >Though maybe we can convince OE to support them in Longhorn.

    Isn’t OE end of lifed? I seem to remember reading that on Slashdot.

  2. Susan says:

    Not everyone wants a web experience and certainly not a web services that would work with one client. First off I think your baseline is incorrect if you are using Outlook Express as the shining NNTP client experience. It’s far from it. Go try Agent from Forte or Thunderbird or go grab Newsgator that plugs into outlook with the nntp posting add in.

    Outlook Express should not be your benchmark.

    Throw OE out the window and build a newsreader that is worthy of NNTP.

  3. <blockquote>Though maybe we can convince OE to support them in Longhorn. :-)</blockquote>

    Smile when you say that, stranger… 🙂

  4. The thing I like about NNTP is the ability to use Google to find information. Most of the stuff you find using the Web search via Google is junk from a bunch of SEO companies manipulating the engine, but the NNTP group searching is excellent. Most of the time when I have a problem I search NNTP postings, I think if you ignore newsgroups you lose all those people searching for answers there and all that exposure, traffic, etc.

  5. ray says:

    Why not work with the wider community (including Open Source) to extend the NNTP protocol to include all the features you are looking for (moderation, single sign on, etc.)?

    Release this as an RFC and everyone’s favourite NNTP reader can implement the standard.

  6. I love the newsgroups. I don’t really care about NNTP. What I want is a fast experience, so I can answer as many questions in a timeframe as possible. If the backend is a special MS-only protocol, I don’t care.

    The other thing is Googling. Google Groups is a huge asset. Whatever MSFT does, they need to ensure that the data is exposed to a good search engine.

    Fast client, good search. That’s all *I* care about.

  7. Daniel O'Connell says:

    Personally, I think NNTP is not something I would suggest striving to extend. It, like FTP and HTTP, are old protocols that really trade off *huge* amounts of functionality for effiency that is unneeded, and they are badly designed because of it. NNTP is a hacky protocol that has been over extended and maltreated to get it into a semi-usable state. However, right now, its the only standard that exists. Sometimes the dead-minded adherance to standards strictly because they are standards is not a good idea. NNTP really doesn’t need to exist anymore, IMHO. It will last a long time entirely based on the strength of its existence…not on any particular merit to the protocol.

  8. Daniel O'Connell says:

    [Bah, managed to hit submit before finishing that post. This post can be joined to it if the admin wishes to bother ;)]

    Now, the problem is designing a cleaner solution. Some of the really popular solutions we see these days, web services, xml, etc. suffer from the same problems we have in NNTP. HTTP is inherently stateless so NNTP has a leg up there. web services use xml and http, generally, xml has to be escaped and can’t contain binary data, so it is pretty equal to the trickiness of mime posts as they exist in NNTP(only upside of xml is schemas and namespaces, which are better than the X-Whatever style mime headers, IMHO).

    Other bits you’ve been talking about, multiple forum reputation, etc would require a new protocol of some sort for communication between servers, be it out of band from NNTP or an inherent feature in the new protocol.

    However, if a new protocol is created or if NNTP is just extended, I think Microsoft could do a few things to increase its usage and feasibility:

    1) definatly release an rfc or othersuch standard.

    2) provide a .NET assembly that implements the protocol in full. That would allow both 3rd party sites that use ASP.NET to easily access the backbone as well as allow smart clients to be built easily. This could be open source or not, I doubt it would matter. Ditto for being a part of the framework. Ideally it would work on mono however.

    3) Provide an open source CC++ implementation of atleast the reader portion that is as platform independent as possible. Not only does it show good faith, which is popular these days, it should ease the move for unmanaged readers, including outlook express, agent, what have you. I don’t know that a CC++ implementation of the server-server protocol(if its different) is as useful as say, a perl, php, or java one would be, however.

    As for google groups, the simplist solution is to expose a read only nntp server that google groups and other search engines could reference(maybe microsoft.readonly.<name>) until such an extended protocol gains enough ground for google to support it. Sites that allow posting would have to adapt to the new protocolextensions.

  9. Alex Lowe says:

    I don’t understand why we need a new format. This has been said before but, people like accessing and storing the data in different ways. The existing formats and venues provide ample choices for people to do what they want.

    I would argue that if you want to build a better and common user experience across the community sites then you should do the following:

    Create the two primary venues for exchanging information for *all* of the sites: NNTP and Forums. I mean, all community sites should offer both so the user has a choice.


    Use NNTP as the underlying storage mechanism and build forums that interact with the newsgroups. V2 of the forums support this concept and these guys seem to have a nice web based version of the .NET newsgroups. I fial to see what you lose when building online forums from the newsgroups?

    Wait, I hear someone in the back saying “But, I can’t easily search all of the information in the disparate sites.“. Ok, so build a federated search engine to search the material. I mean, isn’t there something similar being done by the VS.NET Community/Help folks already?

  10. Alex Lowe says:

    These guys was supposed to be these guys –

  11. josh ledgard says:

    Susan: I agree that there are better clients for NNTP. But the clients aren’t really my problem. My problems are some of the inherent problems that come with NNTP and lack of features that people request with modern web forums. I think it’s unfair to say "one client" if the services were a public part of the ASP.NET then anyone could build a good client that takes advantage of the features exposed from the forums.

  12. josh ledgard says:

    Patrick: Just for arguments sake lets pretend we went with a solution that exposed RSS. In ten years you can imagine that google (now starting with personalized searches) will have a RSS search for the developer space that would be even better than the newsgroups search. Just hypothetical. There are searches that could be better. For example: the google groups search returns me posts whether or not they have a credible answer attached with them.

  13. josh ledgard says:

    Ray: Another problem that won’t be fixed by extending NNTP is that of firewalls. A lot of users require a solution that doesn’t use non-web ports. This is why there are web front ends to newsgroups. One suggestion I heard though was to create an NNTP->Web bridge for client software to use.

  14. josh ledgard says:

    Micheal: "Fast client, good search. That’s all *I* care about. " – good to know. Most users aren’t attached to a technology as long as they get the answers they need quickly.

    Daniel: Thanks for your additional feedback. I agree that exposing read-only NNTP might be a good solution to google groups. And I also agree with regard to most of your client suggestions and about your good faith comments.

    Alex: I think the reason for building something new is that there is a demand for features that don’t exist in legacy applications. But I do agree that a federated search is important.

  15. ray says:

    Alex: I agree that NNTP (like all the other old protocols written for low-bandwidth usage) is hacky. Your solution is more what I had in mind, just written in a much clearer way 🙂

    What I meant with extending NNTP, was more along the lines of allowing current NNTP clients and servers to be extended in an easy fashion, rather than ripping out the current plumbing and replacing it with something brand new.

    The technical solution really does not matter that much, as long as it is not a proprietary protocol and developers for other platforms are brought on board at an early stage. This will allow servers, web clients and smart/offline clients to interoperate across the board.

    Josh: I agree with the firewall issue and this will become much bigger in future. However, the use of port 80/443 just shifts the problem for now, rather than solves it. The reason companies block outgoing ports at the firewall is to control Internet usage. In future these same companies will use stateful inspection firewalls with application-level filtering in the same way to block access to future protocols that pushes everything through the web ports.

  16. Daniel O'Connell says:

    ray: I’m not really convinced that the changes that are being discussed can be added to NNTP without fundamentally changing NNTP. While we may end up with the same commands and responses…I fear that the difference in semantics would be considerable enough that current clients would have to re-write the protocol and parsing code anyway. In that case breaking away from NNTP in name is probably the best thing to do, even if the protocol *looks* like NNTP.

    Personally I’d like a hybrid xml & textsoap socket based protocol. I am not a huge fan of the traditional view of web services on web servers over HTTP for high bandwidth scenarios, I’d also rather see persistent connections. One of the features I rather like about NNTP is the ability to get headers and the message body on demand; most news clients maintain a persistent or semi-persistent connection to the server, the protcol supports it. If I have to connect once for each newsgroupforum to get headers, and once for every message I choose to download…more than likely the cost of connection buildup and teardown is going to start to rival the cost of the actual message transfers. Also the performance of browsing the forums is going to plummet to close to the speed of using an actual web form. That is unpleasent, IMHO.

    This type of solution will, admittedly, cause firewall issues. But as you pointed out it is only a temporary fix to use 80 or 443 to get around the firewall. The web version of the forum would be the best chioce for circumventing firewalls, IMHO.

    Alex: The one thing I want is a more guarenteed identity. On the public newsgroups there have been several identify fakings, one of which posted some…rather nasty pictures under the name of someone who is now an MVP. Its disrupting and unpleasent. A unified identify system with single signon(using passport or whatever. If passport is pushed though, MS would have to either issue special licensing terms for community sites or reduce the prices considerably. Also, the protocol couldn’t rely on *one* login system, it’d have to extend out.) would benifit long-term, high posting users greatly. Another major value of reputation is to help reduce trolling. With a reputation system I could, theoretically, ignore anyone with a reputation under -1, newbies would start at 0, popular postersmvps could reach 5 or whatever. Also being able to mark messages as reliablecorrect is pretty useful, say you took 5 or 10 MVP’s or other high-reputation members of a given forum and allow them to easily mark a response as validcorrectworth reading, and then the user performing a search could specify that he only wants to see posts that have good responses or from people with reputation x or greater. Some of this is in practice over at slashdot and other forums, however I don’t know that its possible in NNTP. You could provide custom headers with the information, but due to the non-centralized nature of NNTP, there is no guarentee that those header updates will propagate all the way out to the fringe of the server mesh, nor is there a guarentee that the messages won’t be faked by someone with a little time and direct access to a NNTP server. Again, I’m not sure all of this is possible without violating NNTP in spirit, if not in the letter of the spec.

  17. Susan says:

    But Josh, Web interfaces right now are bloated and IMHO doesn’t lend themselves to searching [aka Google groups]. The small screen view, the lack of offline, the fact that people don’t even know that they can see answers under the "+" and we have to repost… I understand that NNTP is "old" but you are going to have to really get anything web based up to the same "durability" of NNTP for most of the folks that answer the questions of the people that use the Web interfaces.

    Web based newsreaders slow me down big time.

  18. Daniel O'Connell says:

    Susan: I for one agree, I won’t use web-based support forums due to the clunky feel and difficulty of usage, however I don’t think anyone has been arguing for just a web interface. The system would certainly *HAVE* to support a non-webui based(and preferabbly non-http based) protocol that can be used for both offline clients and intercommunication between peer sites(reputation, etc would need to be transferred). IMHO switching over to entirely HTTP(be it web based or just HTTP endpoints where the protocol flows through) is, in effect, going from one old protocol to another that has more problems. I think this proposed system has to be more than just a collection of stateless web services.

    Using a technology like xml(which, IMHO, is significantly better than mime headers and the mime dividing system, which are a touch inconsistent and *not* fun to write a parser for) to encapsulate the message and commands should make it fairly easy for clients to be written. However, there is the problem of bloat(xml command would probably be atleast a few dozen bytes larger) and general anti-xmlism. There is also the problem of "everybody has to write an extension". One of the major things I dislike about xml is that its flexibilty results in featureladden messages, sometimes with duplicated information(this is one of the problems I have with RSS).

    IMHO, its possible to divise something that is both better featurewise, of equal or better quality durability, flexibilty and usability wise, and easier to write an implementation too. However, as I’ve said a few times, I’m not convinced it can be done with HTTP.