View source on GitHub
|
A container for the Regex tokenizer 1 metadata information.
tflite_support.metadata_writers.metadata_info.RegexTokenizerMd(
delim_regex_pattern: str, vocab_file_path: str
)
Args | |
|---|---|
delim_regex_pattern
|
the regular expression to segment strings and create tokens. |
vocab_file_path
|
path to the vocabulary file. |
Methods
create_metadata
create_metadata() -> tflite_support.metadata_schema_py_generated.ProcessUnitT
Creates the Bert tokenizer metadata based on the information.
| Returns | |
|---|---|
| A Flatbuffers Python object of the Bert tokenizer metadata. |