Abort is not a great error handling strategy

[stellaraccident]

Calling these two functions with illegal data causes a stack-trace/abort in Rust code (when it tries to unwrap a failing error object):

::tokenizers::Tokenizer::FromBlobJSON(json_blob)
::tokenizers::Tokenizer::FromBlobByteLevelBPE(vocab_blob, merges_blob, added_tokens)

The sentence piece initializer doesn't fail but it is probably just silently not parsing the protobuf:

::tokenizers::Tokenizer::FromBlobSentencePiece(model_blob)

Such functions in a library aren't a great design: it would be better to at least have a simple way to marshal the failure to create to the user. For code that is trying to be light-weight and not use exceptions, I would probably use a signature like:

static std::unique_ptr<Tokenizer> FromBlobJSON(const std::string& json_blob, std::string &error_message);

And define it such that returning a nullptr indicates error and the library makes some attempt to populate error_message in this case. Since you likely already have things calling these and those aren't expecting a nullptr return, maybe rename the to a WithError suffix or something.