DIY corpus using Search Engines (like Google)

• Open source development of large corpora
• Creating general-purpose corpora using automated search engine queries
• An Crúbadán - web crawling software for the automatic development of large text corpora for minority languages

Natural Language Toolkit (NLTK)

• NLTK Over 40 corpora and corpus samples are included with the NLTK Corpus Distribution (750Mb).
  • Parsed Corpora
  • Tagged Corpora
  • Text Corpora
  • Lexicons
  • Categorized Corpora
  • Miscellaneous

ARPA

• CMU Hub4 - language model in ARPA format
• CMU ARPA-format bigram language model - 57138 unigrams and about 10 million bigrams (link)
• xvoice-sphinx language model

Possible sources of written data (written corpora) for the creation of Language Models

• U.S. Government Printing Office
• Gutenburg project
• The Universal Digital Library
• Wikipedia Spoken Articles
• Hansard Canada
  • House of Commons
  • Senate
• Google Research word n-gram models and training corpora
• Complete Works of William Shakespeare
• Moby Project
• Google Books
• ibiblio
• Corpora and Corpus-based Computational Linguistics - Manuel Barbera's Web Resources Reference Guide
• Alex: A Catalogue of Electronic Texts on the Internet
• Wiretap Electronic Text Archive- Gopher based (accessible via a browser)
• The Electronic Text Center of the University of Virginia
• Aligned Hansards of the 36th Parliament of Canada
• Enron emails
  • Enron Email Dataset William W. Cohen, CMU
  • Federal Energy Regulatory Commission website
• 20+ Places for Public Domain E-Books

Other Sources but with Licensing Restrictions

• TAPoR Text Analysis Portal for Research at the University of Alberta
  • WestburyLAB USENET corpus - Creative Commons Attribution-Non Commercial-No Derivs
• Google
  • Web 1T 5-gram Version 1 - linguistic education and research only

Multilingual Copora

• Leipzig Corpora Collection - licensing restrictions