tokenizers
==========


.. currentmodule:: hearth.text.tokenizers








Tokenizer
---------

.. autoclass:: Tokenizer
    :members:
    :undoc-members:
    :no-inherited-members:

----




WordPieceTokenizer
------------------

.. autoclass:: WordPieceTokenizer
    :members:
    :undoc-members:
    :no-inherited-members:

----