Skip to content

Huggingface complains about a forked process when using BobCat and the QuantumModel #218

@dimkart

Description

@dimkart

Using BobCat to parse sentences, any method call of the QuantumModel will lead to the following warning message:

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

This spams the command-line output and needs to be fixed.

A hacky fix would be to add these two lines to the code:

import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"

Metadata

Metadata

Assignees

No one assigned

    Labels

    fixA non-critical bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions