Cool, MSN Search has their own web crawler now. Building a spider was one of the first projects I worked on at Microsoft back in 1996. Mine was a "web monkey", which basically meant that its link traversal was stochastic. I remember setting it free on some site and invariably within a couple of hours, it would be stuck in the domain. This was back in the day when everyone was including "best when viewed with Navigator" image links at the bottom of their pages. All roads led to Netscape in those days.

Comments (0)

Skip to main content