Monday, June 1, 2009

Count the number of words using Python

This article represents a way to count the number of words, the number of unique words, and the number of each word occurrences in a text file.
  1. Read the text file.
    text = open("filename.txt").read()
  2. Replace non-alphanumeric characters as a whitespace.
    import re text = re.sub('[^w&^d]', ' ', text)
  3. Change all characters to lowercase.
    text = text.lower()
  4. Split words into a list.
    text = text.split()
  5. Display the number for words in the text file.
  6. Display the number of unique words in the text file.
  7. Display the number of occurrences for each word.
    from collections import defaultdict
    wordsCount = defaultdict(int)
    for word in text:
        wordsCount[word] += 1
        for word, num in wordsCount.items():
            print(word, num)