The nltk.corpus package defines a collection of corpus reader classes, which can be used to access the contents of a diverse set of corpora. Each corpus reader class is specialized to handle a specific corpus format. In addition, the nltk.corpus package automatically creates a set of corpus reader instances that can be used to access the corpora in the NLTK data package. Section 1 ("Corpus Reader Objects") describes the corpus reader instances that can be used to read the corpora in the NLTK data package. Section 2 ("Corpus Reader Classes") describes the corpus reader classes themselves, and discusses the issues involved in creating new corpus reader objects and new corpus reader classes. Section 3 ("Regression Tests") contains regression tests for the corpus readers and associated functions and classes.
Table of Contents
The NLTK data package includes a diverse set of corpora, which can be read using the nltk.corpus package. Each corpus is accessed by means of a "corpus reader" object from nltk.corpus:
|
Most corpora consist of a set of files, each containing a document (or other pieces of text). A list of identifiers for these files is accessed via the files() method of the corpus reader:
|
Each corpus reader provides a variety of methods to read data from the corpus, depending on the format of the corpus. For example, plaintext corpora support methods to read the corpus as raw text, a list of words, a list of sentences, or a list of paragraphs.
|
Each of these reader methods may be given a single document's item name or a list of document item names. When given a list of document item names, the reader methods will concatenate together the contents of the individual documents.
|
If the reader methods are called without any arguments, they will typically load all documents in the corpus.
|
Here are the first few words from each of NLTK's plaintext corpora:
|
In addition to the plaintext corpora, NLTK's data package also contains a wide variety of annotated corpora. For example, the Brown Corpus is annotated with part-of-speech tags, and defines additional methods tagged_*() which words as (word,tag) tuples, rather than just bare word strings.
|
Similarly, the Indian Langauge POS-Tagged Corpus includes samples of Indian text annotated with part-of-speech tags:
|
Tagged corpora support access to simplified tags, e.g. where all nouns tags are collapsed to a single category N:
|
Use nltk.draw.pos-concordance() to access a GUI for searching tagged corpora.
The CoNLL corpora also provide chunk structures, which are encoded as flat trees. The CoNLL 2000 Corpus includes phrasal chunks; and the CoNLL 2002 Corpus includes named entity chunks.
|
Note
Since the CONLL corpora do not contain paragraph break information, these readers do not support the para() method.)
Warning
if you call the conll corpora reader methods without any arguments, they will return the contents of the entire corpus, including the 'test' portions of the corpus.)
The IEER corpus is another chunked corpus. This corpus is unusual in that each corpus item contains multiple documents. (This reflects the fact that each corpus file contains multiple documents.) The IEER corpus defines the parsed_docs method, which returns the documents in a given item as IEERDocument objects:
|
The Treebank corpora provide a syntactic parse for each sentence. The NLTK data package includes a 10% sample of the Penn Treebank (in treebank), as well as the Sinica Treebank (in sinica_treebank).
Reading the Penn Treebank:
|
Reading the Sinica Treebank:
|
NLTK also provides a corpus reader for the York-Toronto-Helsinki Parsed Corpus of Old English Prose (YCOE); but the corpus itself is not included in the NLTK data package. If you install it yourself, you can use NLTK to access it:
|
If the YCOE corpus is not available, you will get an error message when you try to access it:
|
The NLTK data package also includes a number of lexicons and word lists. These are accessed just like text corpora. The following examples illustrate the use of the wordlist corpora:
|
|
|
The CMU Pronunciation Dictionary corpus contains pronounciation transcriptions for over 100,000 words. It can be accessed as a list of entries (where each entry consists of a word, an identifier, and a transcription) or as a dictionary from words to lists of transcriptions. Transcriptions are encoded as tuples of phoneme strings.
|
Several corpora included with NLTK contain documents that have been categorized for topic, genre, polarity, etc. In addition to the standard corpus interface, these corpora provide access to the list of categories and the mapping between the documents and their categories (in both directions). Access the categories using the categories() method, e.g.:
|
This method has an optional argument that specifies a document or a list of documents, allowing us to map from (one or more) documents to (one or more) categories:
|
We can go back the other way using the optional argument of the files() method:
|
Both the categories() and files() methods return a sorted list containing no duplicates.
In addition to mapping between categories and documents, these corpora permit direct access to their contents via the categories. Instead of accessing a subset of a corpus by specifying one or more files, we can identify one or more categories, e.g.:
|
Note that it is an error to specify both documents and categories.
In the context of a text categorization system, we can easily test if the category assigned to a document is correct as follows:
|
The Propbank corpus provides predicate-argument annotation for the entire treebank. Each verb in the treebank is annotated by a single instance in the propbank corpus, containing information about the location of the verb, and the location and identity of its arguments:
|
Each propbank instance defines the following member variables:
- Location information: filename, sentnum, wordnum
- Annotator information: tagger
- Inflection information: inflection
- Roleset identifier: roleset
- Verb (aka predicate) location: predicate
- Argument locations and types: arguments
The following examples show the types of these arguments:
|
The location of the predicate and of the arguments are encoded using PropbankTreePointer objects, as well as PropbankChainTreePointer objects and PropbankSplitTreePointer objects. A PropbankTreePointer consists of a wordnum and a height:
|
This identifies the tree constituent that is headed by the word that is the wordnum'th token in the sentence, and whose span is found by going height nodes up in the tree. This type of pointer is only useful if we also have the corresponding tree structure, since it includes empty elements such as traces in the word number count. The trees for 10% of the standard propbank corpus are contained in the treebank corpus:
|
|
|
Propbank tree pointers can be converted to standard tree locations, which are usually easier to work with, using the treepos() method:
|
In some cases, argument locations will be encoded using PropbankChainTreePointers (for trace chains) or PropbankSplitTreePointers (for discontinuous constituents). Both of these objects contain a single member variable, pieces, containing a list of the constituent pieces. They also define the method select(), which will return a tree containing all the elements of the argument. (A new head node is created, labeled "CHAIN" or "SPLIT", since the argument is not a single constituent in the original tree). Sentence #6 contains an example of an argument that is both discontinuous and contains a chain:
|
The propbank corpus also provides access to the frameset files, which define the argument labels used by the annotations, on a per-verb basis. Each frameset file contains one or more predicates, such as 'turn' or 'turn_on', each of which is divided into coarse-grained word senses called rolesets. For each roleset, the frameset file provides descriptions of the argument roles, along with examples.
|
|
Note that the standard corpus distribution only contains 10% of the treebank, so the parse trees are not availalbe for instances starting at 9353:
|
|
However, if you supply your own version of the treebank corpus (by putting it before the nltk-provided version on nltk.data.path), then you can access the trees for all instances.
A list of the verb lemmas contained in propbank is returned by the propbank.verbs() method:
|
The Senseval 2 corpus is a word sense disambiguation corpus. Each item in the corpus corresponds to a single ambiguous word. For each of these words, the corpus contains a list of instances, corresponding to occurences of that word. Each instance provides the word; a list of word senses that apply to the word occurrence; and the word's context.
|
The following code looks at instances of the word 'interest', and displays their local context (2 words on each side) and word sense(s):
|
The Prepositional Phrase Attachment corpus is a corpus of prepositional phrase attachment decisions. Each instance in the corpus is encoded as a PPAttachment object:
|
The Shakespeare corpus contains a set of Shakespeare plays, formatted as XML files. These corpora are returned as ElementTree objects:
|
The Toolbox corpus distributed with NLTK contains a sample lexicon and several sample texts from the Rotokas language. The Toolbox corpus reader returns Toolbox files as XML ElementTree objects. The following example loads the Rotokas dictionary, and figures out the distribution of part-of-speech tags for reduplicated words.
|
This example displays some records from a Rotokas text:
|
The NLTK data package includes a fragment of the TIMIT Acoustic-Phonetic Continuous Speech Corpus. This corpus is broken down into small speech samples, each of which is available as a wave file, a phonetic transcription, and a tokenized word list.
|
|
The corpus reader can combine the word segmentation information with the phonemes to produce a single tree structure:
|
The start time and stop time of each phoneme, word, and sentence are also available:
|
We can use these times to play selected pieces of a speech sample:
|
The corpus reader can also be queried for information about the speaker and sentence identifier for a given speech sample:
|
|
The RTE (Recognizing Textual Entailment) corpus was derived from the RTE1, RTE2 and RTE3 datasets (dev and test data), and consists of a list of XML-formatted 'text'/'hypothesis' pairs.
|
In the gold standard test sets, each pair is labeled according to whether or not the text 'entails' the hypothesis; the entailment value is mapped to an integer 1 (True) or 0 (False).
|
The RTE corpus also supports an xml() method which produces ElementTrees.
|
The VerbNet corpus is a lexicon that divides verbs into classes, based on their syntax-semantics linking behavior. The basic elements in the lexicon are verb lemmas, such as 'abandon' and 'accept', and verb classes, which have identifiers such as 'remove-10.1' and 'admire-31.2-1'. These class identifiers consist of a representitive verb selected from the class, followed by a numerical identifier. The list of verb lemmas, and the list of class identifiers, can be retrieved with the following methods:
|
The classids() method may also be used to retrieve the classes that a given lemma belongs to:
|
The primary object in the lexicon is a class record, which is stored as an ElementTree xml object. The class record for a given class identifier is returned by the vnclass() method:
|
The vnclass() method also accepts "short" identifiers, such as '10.1':
|
See the Verbnet documentation, or the Verbnet files, for information about the structure of this xml. As an example, we can retrieve a list of thematic roles for a given Verbnet class:
|
The Verbnet corpus also provides a variety of pretty printing functions that can be used to display the xml contents in a more consise form. The simplest such method is pprint():
|
The NPS Chat Corpus, Release 1.0 consists of over 10,000 posts in age-specific chat rooms, which have been anonymized, POS-tagged and dialogue-act tagged.
|
We can access the XML elements corresponding to individual posts. These elements have class and user attributes that we can access using p.attrib['class'] and p.attrib['user']. They also have text content, accessed using p.text.
|
In addition to the above methods for accessing tagged text, we can navigate the XML structure directly, as follows:
|
NLTK's corpus reader classes are used to access the contents of a diverse set of corpora. Each corpus reader class is specialized to handle a specific corpus format. Examples include the PlaintextCorpusReader, which handles corpora that consist of a set of unannotated text files, and the BracketParseCorpusReader, which handles corpora that consist of files containing parenthesis-delineated parse trees.
When then nltk.corpus module is imported, it automatically creates a set of corpus reader instances that can be used to access the corpora in the NLTK data distribution. Here is a small sample of those corpus reader instances:
|
This sample illustrates that different corpus reader classes are used to read different corpora; but that the same corpus reader class may be used for more than one corpus (e.g., genesis and inaugural).
Although the nltk.corpus module automatically creates corpus reader instances for the corpora in the NLTK data distribution, you may sometimes need to create your own corpus reader. In particular, you would need to create your own corpus reader if you want...
To create a new corpus reader, you will first need to look up the signature for that corpus reader's constructor. Different corpus readers have different constructor signatures, but most of the constructor signatures have the basic form:
SomeCorpusReader(root, files, ...options...)
Where root is an absolute path to the directory containing the corpus data files; files is either a list of file names (relative to root) or a regexp specifying which files should be included; and options are additional reader-specific options. For example, we can create a customized corpus reader for the genesis corpus that uses a different sentence tokenizer as follows:
|
For a complete list of corpus reader subclasses, see the API documentation for nltk.corpus.CorpusReader.
Corpora vary widely in the types of content they include. This is reflected in the fact that the base class CorpusReader only defines a few general-purpose methods for listing and accessing the files that make up a corpus. It is up to the subclasses to define data access methods that provide access to the information in the corpus. However, corpus reader subclasses should be consistent in their definitions of these data access methods wherever possible.
At a high level, corpora can be divided into three basic types:
However, many individual corpora blur the distinctions between these types. For example, corpora that are primarily lexicons may include token data in the form of example sentences; and corpora that are primarily token corpora may be accompanied by one or more word lists or other lexical data sets.
Because corpora vary so widely in their information content, we have decided that it would not be wise to use separate corpus reader base classes for different corpus types. Instead, we simply try to make the corpus readers consistent wherever possible, but let them differ where the underlying data itself differs.
As mentioned above, there are only a handful of methods that all corpus readers are guaranteed to implement. These methods provide access to the files that contain the corpus data. Every corpus is assumed to consist of one or more files, all located in a common root directory (or in subdirectories of that root directory). The absolute path to the root directory is stored in the root property:
|
Each file within the corpus is identified by a platform-independent identifier, which is basically a path string that uses / as the path seperator. I.e., this identifier can be converted to a relative path as follows:
|
To get a list of all data files that make up a corpus, use the files() method. In some corpora, these files will not all contain the same type of data; for example, for the nltk.corpus.timit corpus, files() will return a list including text files, word segmentation files, phonetic transcription files, sound files, and metadata files. For corpora with diverse file types, the files() method will often take one or more optional arguments, which can be used to get a list of the files with a specific file type:
|
In some corpora, the files are divided into distinct categories. For these corpora, the files() method takes an optional argument, which can be used to get a list of the files within a specific category:
|
The abspath() method can be used to find the absolute path to a corpus file, given its file identifier:
|
The abspaths() method can be used to find the absolute paths for one corpus file, a list of corpus files, or all corpus files, depending on its argument type:
|
This method is mainly useful as a helper method when defining corpus data access methods, since data access methods can usually be called with a string argument (to get a view for a specific file), with a list argument (to get a view for a specific list of files), or with no argument (to get a view for the whole corpus).
Individual corpus reader subclasses typically extend this basic set of file-access methods with one or more data access methods, which provide easy access to the data contained in the corpus. The signatures for data access methods often have the basic form:
corpus_reader.some_data access(files=None, ...options...)
Where files can be a single file identifier string (to get a view for a specific file); a list of file identifier strings (to get a view for a specific list of files); or None (to get a view for the entire corpus). Some of the common data access methods, and their return types, are:
- I{corpus}.words(): list of str
- I{corpus}.sents(): list of (list of str)
- I{corpus}.paras(): list of (list of (list of str))
- I{corpus}.tagged_words(): list of (str,str) tuple
- I{corpus}.tagged_sents(): list of (list of (str,str))
- I{corpus}.tagged_paras(): list of (list of (list of (str,str)))
- I{corpus}.chunked_sents(): list of (Tree w/ (str,str) leaves)
- I{corpus}.parsed_sents(): list of (Tree with str leaves)
- I{corpus}.parsed_paras(): list of (list of (Tree with str leaves))
- I{corpus}.xml(): A single xml ElementTree
- I{corpus}.raw(): str (unprocessed corpus contents)
For example, the words() method is supported by many different corpora, and returns a flat list of word strings:
|
On the other hand, the tagged_words() method is only supported by corpora that include part-of-speech annotations:
|
|