Export tokenizer.model for parakeet export

[mergennachin]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16494

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 43 Pending

As of commit 12949ee with merge base 09b5bdb ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[Copilot]

Pull request overview

This PR adds functionality to extract and export the tokenizer.model file from the Parakeet model's .nemo archive during the export process.

• Adds a new extract_tokenizer() function that downloads and extracts tokenizer.model from the cached .nemo file
• Integrates tokenizer extraction into the main export workflow before model loading
• Adds necessary imports (shutil, tarfile, tempfile) to support tar archive extraction

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The function opens a tar file without specifying error handling for potential exceptions. If the .nemo file is corrupted or not a valid tar archive, this will raise an exception that is not handled. Consider adding a try-except block to handle tarfile.ReadError and other potential exceptions, and return None with an appropriate error message.

Suggested change

	with tarfile.open(nemo_path, "r") as tar:
	# Find tokenizer.model in the archive (may be in root or subdirectory)
	tokenizer_member = None
	for member in tar.getmembers():
	if member.name.endswith(tokenizer_filename):
	tokenizer_member = member
	break
	if tokenizer_member is None:
	print(f"Warning: {tokenizer_filename} not found in .nemo archive")
	return None
	# Extract to temp directory
	tar.extract(tokenizer_member, tmpdir)
	extracted_path = os.path.join(tmpdir, tokenizer_member.name)
	# Copy to output directory
	shutil.copy2(extracted_path, output_path)
	try:
	with tarfile.open(nemo_path, "r") as tar:
	# Find tokenizer.model in the archive (may be in root or subdirectory)
	tokenizer_member = None
	for member in tar.getmembers():
	if member.name.endswith(tokenizer_filename):
	tokenizer_member = member
	break
	if tokenizer_member is None:
	print(f"Warning: {tokenizer_filename} not found in .nemo archive")
	return None
	# Extract to temp directory
	tar.extract(tokenizer_member, tmpdir)
	extracted_path = os.path.join(tmpdir, tokenizer_member.name)
	# Copy to output directory
	shutil.copy2(extracted_path, output_path)
	except (tarfile.ReadError, tarfile.TarError, OSError) as e:
	print(f"Error: failed to open or extract from .nemo archive '{nemo_path}': {e}")
	return None

[Copilot]

The return value from extract_tokenizer is not checked. If the tokenizer extraction fails (returns None), the script will continue execution without the tokenizer file. Consider checking the return value and handling failure appropriately, such as by logging a warning or raising an error if the tokenizer is required for the export.

Suggested change

	extract_tokenizer(args.output_dir)
	tokenizer_path = extract_tokenizer(args.output_dir)
	if tokenizer_path is None:
	raise RuntimeError(
	f"Failed to extract tokenizer into output directory: {args.output_dir}"
	)

[Copilot]

Potential path traversal vulnerability when extracting from tar archive. A malicious .nemo file could contain members with absolute paths or paths with ".." that could write files outside the intended output directory. Consider validating that the extracted member path is safe before extraction, or use a safer extraction method that prevents path traversal attacks.

Suggested change

	with tempfile.TemporaryDirectory() as tmpdir:
	with tarfile.open(nemo_path, "r") as tar:
	# Find tokenizer.model in the archive (may be in root or subdirectory)
	tokenizer_member = None
	for member in tar.getmembers():
	if member.name.endswith(tokenizer_filename):
	tokenizer_member = member
	break
	if tokenizer_member is None:
	print(f"Warning: {tokenizer_filename} not found in .nemo archive")
	return None
	# Extract to temp directory
	tar.extract(tokenizer_member, tmpdir)
	extracted_path = os.path.join(tmpdir, tokenizer_member.name)
	# Copy to output directory
	shutil.copy2(extracted_path, output_path)
	with tarfile.open(nemo_path, "r") as tar:
	# Find tokenizer.model in the archive (may be in root or subdirectory)
	tokenizer_member = None
	for member in tar.getmembers():
	if member.name.endswith(tokenizer_filename):
	tokenizer_member = member
	break
	if tokenizer_member is None:
	print(f"Warning: {tokenizer_filename} not found in .nemo archive")
	return None
	if not tokenizer_member.isfile():
	print(f"Warning: {tokenizer_filename} in .nemo archive is not a regular file")
	return None
	# Safely extract the tokenizer file contents directly to the output path
	with tar.extractfile(tokenizer_member) as src, open(output_path, "wb") as dst:
	shutil.copyfileobj(src, dst)

[Copilot]

The huggingface_hub import is placed inside the function rather than at the module level. While this can avoid import errors if the package is not installed, it's inconsistent with the module-level imports used for other dependencies like torch and torchaudio. Consider either importing at the module level for consistency, or documenting why this particular import needs to be local.