We prepared, as we had for the earlier dataset, the top-100K words list for the body stream for Apr10. You can download it here.
We decided to take a closer look at the dataset to how the top 100K lists changed between Jun09 and Apr10. Our findings are interesting:
- The union of the word set is just shy of 110K. This means that 10% of the words either fell in or out of the top 100K. This is a turnover rate higher than I expected.
- Some words that are newly in the top list are what you’d expect (unigram log10 probability difference shown parenthetically):
- espnlosangeles (21.88993), an ESPN satellite established during 2009
- debate2010 (21.53613)
- Some words took a predictable jump:
- ipad (2.560667), a product introduced mid-year
- Quite a few words newly in the mix are not conversational words:
- childreplyhtml (22.09848)
- focaladvid (21.76564)