tokenize_sents {idiolect} | R Documentation |
Tokenize to sentences
Description
This function turns a corpus of texts into a quanteda
tokens object of sentences.
Usage
tokenize_sents(corpus, model = "en_core_web_sm")
Arguments
corpus |
A |
model |
The spacy model to use. The default is "en_core_web_sm". |
Details
The function first split each text into paragraphs by splitting at new line markers and then uses spacy to tokenize each paragraph into sentences. The function accepts a plain text corpus input or the output of contentmask()
. This function is necessary to prepare the data for lambdaG()
.
Value
A quanteda
tokens object where each token is a sentence.
Examples
## Not run:
toy.pos <- corpus("the N was on the N . he did n't move \n N ; \n N N")
tokenize_sents(toy.pos)
## End(Not run)
[Package idiolect version 1.0.1 Index]