MarkdownDocumentChunks {ragnar} | R Documentation |
Markdown documents chunks
Description
MarkdownDocumentChunks
stores information about candidate chunks in a
Markdown document. It is a tibble with three required columns:
-
start
,end
— integers. These are character positions (1-based, inclusive) in the sourceMarkdownDocument
, so thatsubstring(md, start, end)
yields the chunk text. Ranges can overlap. -
context
— character. A general-purpose field for adding context to a chunk. This column is combined withtext
to augment chunk content when generating embeddings withragnar_store_insert()
, and is also returned byragnar_retrieve()
. Keep in mind that when chunks are deoverlapped (inragnar_retrieve()
orchunks_deoverlap()
), only the context value from the first chunk is kept.markdown_chunk()
by default populates this column with all the markdown headings that are in-scope at the chunk start position.
Additional columns can be included.
The original document is available via the @document
property.
For normal use, chunk a Markdown document with markdown_chunk()
; the
class constructor itself is exported only so advanced users can generate or
tweak chunks by other means.
Arguments
chunks |
A data frame containing |
document |
A |
Value
An S7 object that inherits from MarkdownDocumentChunks
, which is
also a tibble
.
See Also
Examples
doc_text <- "# A\n\nB\n\n## C\n\nD"
doc <- MarkdownDocument(doc_text, origin = "some/where")
chunk_positions <- tibble::tibble(
start = c(1L, 9L),
end = c(8L, 15L),
context = c("", "# A"),
text = substring(doc, start, end)
)
chunks <- MarkdownDocumentChunks(chunk_positions, doc)
identical(chunks@document, doc)