YOLO Workshop

SAM2 Labeling on Video Example (left) and YOLOv8 Fine-Tuning Example (right)

SAM2 Video autolabeler, fine-tuning, and inference pipeline for YOLOv8-seg using ultralytics. This project consists of:

• Frontend: React/Next.js app
• Backend: Python + YOLOv8, hosted on FastAPI

Prerequisites

Make sure you have the following installed:

• Node.js (v18+ recommended)
• pnpm npm install -g pnpm
• NVIDIA GPU + CUDA (optional but highly recommended)

Install

1. From the repo root, create the Conda environment:

conda env create -f backend/environment.yml
conda activate sam2env

2. Start the backend:

cd backend
uvicorn app.main:app --reload --workers 1

Backend should now be running at http://localhost:8000

3. (Optional, for fine-tune and inference) Install ultralytics models:

python3 install_ultralytics_models.py

4. Install frontend dependencies via pnpm:

cd frontend
pnpm install

5. Configure environment variables, create a .env.local file in frontend if needed

cat <<EOF > .env.local
NEXT_PUBLIC_API_BASE_URL=http://localhost:8000
EOF

6. Start frontend dev server:

pnpm dev

Frontend should now be running at: http://localhost:3000

Video-Player and Segmentation Help

Saving segmentations leads to empty datasets

This issue relates to how the uvicorn server uses multiple workers. The easiest fix is to restart the backend server with 1 worker:

uvicorn app.main:app --reload --workers 1

Model Fine-tuning Help

Known Limitations

• Currently only 1 segmentation per dataset is allowed

TODO

• Support for pose models (figure out labeling service)

License

MIT License

Notes

This project provides training and deployment tooling. Users are responsible for ensuring model weights, datasets, and downstream deployments comply with applicable licenses and regulations.

Sources

@ekberndt's YOLOv8 Instance Segmentation Fine-Tuning helped me with the fine-tuning setion of this project.