Tokenizer
Overview
Tokenizer is a class in Aspose.Html FOSS for Python.
WHATWG HTML tokeniser (§13.2.5).
This class provides 4 methods for working with Tokenizer objects in Python programs.
Available methods include: __init__, set_state, tokenize, tokenize_fragment.
All public members are accessible to any Python application after installing the Aspose.Html FOSS for Python package.
Properties: errors, state.
Properties
| Name | Type | Access | Description |
|---|---|---|---|
state | TokenizerState | Read | The current tokeniser state. |
errors | list[ParseError] | Read | Parse errors collected during tokenisation. |
Methods
| Signature | Description |
|---|---|
__init__(text: str, initial_state: TokenizerState) | Calls init(text, initial_state) on this Tokenizer instance. |
set_state(state: TokenizerState) | Inject a new state into the tokeniser. |
tokenize() → Generator[AnyToken, None, None] | Tokenise the input text and yield tokens one at a time. |
tokenize_fragment(context_element_name: str, text: str) → Generator[AnyToken, None, None] | Tokenise text as a fragment for innerHTML parsing. |