.process_vocab {wordpiece}R Documentation

Process a Vocabulary for Tokenization

Description

Process a Vocabulary for Tokenization

Usage

.process_vocab(v)

## Default S3 method:
.process_vocab(v)

## S3 method for class 'wordpiece_vocabulary'
.process_vocab(v)

## S3 method for class 'character'
.process_vocab(v)

Arguments

v

An object of class wordpiece_vocabulary or a character vector.

Value

A character vector of tokens for tokenization.


[Package wordpiece version 2.1.3 Index]