Script and Optimisation for 4Go VRAM, and launch script.

README.md

@@ -1,3 +1,70 @@
+# AnimateHYBRID
+
+Mini-Guide: AnimateDiff <4 GB VRAM with N3R
+1️⃣ Pick the Right Model
+
+N3RModelOptimized → ~3.6 GB VRAM, full features, stable.
+
+Mini GPU Mode + generate_latents_mini_gpu_320 → ~2.1 GB VRAM, ultra-light for quick tests.
+
+2️⃣ VRAM-Friendly Settings
+
+final_latent_scale → reduces the final latent resolution to save memory.
+
+num_fraps_per_image → limit the number of frames per input image.
+
+block_size & overlap → tweak for streaming decoding efficiency.
+
+3️⃣ Adaptive N3R Fusion
+
+Channel-wise normalization to prevent artifacts.
+
+Strict clamping of latents: torch.clamp(latents, -1.0, 1.0).
+
+Controlled latent injection:
+
+fused_latents = latent_injection * latents_frame + (1 - latent_injection) * n3r_latents
+4️⃣ Motion / LoRA / VAE
+
+Motion modules and LoRA can be enabled, but watch VRAM usage → use attention slicing if needed.
+
+Light VAE + blockwise decoding ensures GPU stability.
+
+5️⃣ Pro Tips
+
+Free VRAM after each frame:
+
+del latents
+torch.cuda.empty_cache()
+
+Adaptive embeddings for UNet → avoids dimension mismatch errors.
+
+decode_latents_ultrasafe_blockwise → stable decoding with high-quality output.
+
+💡 Bottom Line: With these settings, AnimateDiff can run even on GPUs with 3–4 GB VRAM without sacrificing output quality.
+
+And yes… N3R did it for you! 🚀
+n3rProtoBoost:
+```
+python -m scripts.n3rProtoBoost \
+                      --pretrained-model-path "/huggingface/miniSD" \
+                      --config "configs/prompts/2_animate/1080.yaml" \
+                      --device "cuda" \
+                      --vae-offload \
+                      --fp16*
+```
+n3rProBoostNet:
+```                      
+python -m scripts.n3rProBoostNet \
+                      --pretrained-model-path "/mnt/62G/huggingface/miniSD" \
+                      --config "configs/prompts/0_n3r/960.yaml" \
+                      --device "cuda" \
+                      --vae-offload \
+                      --fp16
+```                      
+
+
+
 # AnimateDiff
 
 This repository is the official implementation of [AnimateDiff](https://arxiv.org/abs/2307.04725) [ICLR2024 Spotlight].

configs/prompts/0_animate/128.yaml

@@ -0,0 +1,61 @@
+#n3r_tiny.yaml
+# -------------------------
+# Tiny-SD config
+# -------------------------
+W: 128
+H: 128
+L: 4
+steps: 15                # un peu plus que le test pour meilleure qualité
+pretrained_model_path: "/mnt/62G/huggingface/miniSD"
+text_encoder: null
+tokenizer: null
+dtype: float16
+
+fps: 12
+guidance_scale: 4.5
+init_image_scale: 0.85
+use_real_esrgan: true     # activé, mais batch plus petit
+
+scheduler:
+  type: DDIMScheduler
+  steps: 15
+  beta_start: 0.00085
+  beta_end: 0.012
+
+dreambooth_path: null
+lora_model_path: null
+
+key_frames: 4
+inter_frames: 4
+
+block_size: 128
+overlap: 16
+num_frames_per_image: 4 # générer par petits batches (tu peux répéter pour 60 frames)
+
+motion_module: scripts/modules/motion_module_tiny.py
+controlnet_adapter: null
+
+inference_config: configs/inference/inference-v4.yaml
+device: cuda
+offload_folder: /tmp/offload
+accelerate: true
+low_cpu_mem_usage: true
+
+seed: 1234
+
+enable_xformers_memory_efficient_attention: true
+vae_slicing: true
+vae_tiling: true
+vae_path: null
+
+upscale_factor: 4           # ESRGAN actif
+
+prompt:
+  - "best quality, 1 girl walking, natural dynamic pose, arms swinging, legs mid-step, balanced torso and head, flowing hair, consistent outfit and hairstyle, outdoor spring, cherry blossoms, petals, smooth motion across frames, coherent animation, vibrant colors, cinematic lighting"
+  - "natural walk cycle, same character, forward facing, slight tilt for realism, smooth transitions, crisp lines, photorealistic shading, coherent body movement, stable proportions, simple background"
+
+n_prompt:
+  - "low quality, blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background"
+
+input_images:
+  - "input/image_128x0.png"

configs/prompts/0_animate/256.yaml

@@ -0,0 +1,77 @@
+# -------------------------
+# n3r_tiny_256_vram_opt.yaml
+# -------------------------
+# Tiny-SD 256x256 VRAM-safe optimisé pour 5D AnimateDiff
+# - Frames fluides et cohérentes
+# - FP16 UNET & TextEncoder, VAE FP32 pour stabilité couleurs
+# - Motion module Tiny intégré
+# -------------------------
+
+# Image size
+W: 256
+H: 256
+L: 4
+steps: 15
+
+# Model paths
+pretrained_model_path: "/mnt/62G/huggingface/miniSD"
+vae_path: "/mnt/62G/huggingface/vae/vae-ft-mse-840000-ema-pruned.safetensors"
+
+# FP settings
+dtype: float32           # VAE en FP32 pour stabilité couleurs
+fp16_unet: true           # UNET et text_encoder en FP16
+
+# Video / motion
+fps: 15
+num_frames_per_image: 12   # permet séquences fluides
+init_image_scale: 0.85
+use_real_esrgan: true
+
+# Scheduler
+scheduler:
+  type: DDIMScheduler
+  steps: 15
+  beta_start: 0.00085
+  beta_end: 0.012
+
+# Motion module
+motion_module: scripts/modules/motion_module_tiny.py
+key_frames: 1
+inter_frames: 11
+
+# Tiling VAE
+vae_slicing: true
+vae_tiling: true
+block_size: 256            # décode l'image entière → plus de mosaïque
+overlap: 32
+
+# Inference / memory
+offload_folder: /tmp/offload
+accelerate: true
+low_cpu_mem_usage: true
+batch_size: 8
+enable_xformers_memory_efficient_attention: true
+
+# Random seed
+seed: 1234
+
+# Diffusion guidance
+guidance_scale: 4.5
+creative_noise: 0.0
+
+# Upscale factor
+upscale_factor: 4
+
+# Prompts
+prompt:
+  - "best quality, 1 girl walking, natural dynamic pose, arms swinging, legs mid-step, balanced torso and head, flowing hair, consistent outfit and hairstyle, outdoor spring, cherry blossoms, petals, smooth motion across frames, coherent animation, vibrant colors, cinematic lighting"
+  - "natural walk cycle, same character, forward facing, slight tilt for realism, smooth transitions, crisp lines, photorealistic shading, coherent body movement, stable proportions, simple background"
+
+n_prompt:
+  - "low quality, blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background"
+
+# Input images
+input_images:
+  - "input/256/image_256x0.png"
+  - "input/256/image_256x1.png"
+  - "input/256/image_256x1.png"

configs/prompts/0_animate/Readme.txt

@@ -0,0 +1 @@
+Sample prompt for low Vram MAX 4Go

configs/prompts/0_n3r/512-a.yaml

@@ -0,0 +1,122 @@
+# -------------------------
+# Tiny-SD 256x256 VRAM-safe FP16 pour 5D AnimateDiff
+# Avec Motion Module Tiny et VAE offload
+latent_injection: 0.70 # 85% proche de l'image init - 0.6 on s'éloigne de l'init - 0.5 creatif - Valeur Stable (0.70)
+creative_noise: 0.08 # moins de liberté, plus de cohérence 0.08 ou 0.15 rendu creatif; 0.00 rendu cinéma (0.10) -- Valeur test à conservé (0.00)
+# Exploration maximale du latent 0. (0.25)  - Augmenter init_image_scale → donne plus de signal à partir de ton image d’initiation. (1.0 image original)
+init_image_scale: 0.5  # - on s'éloigne de l'init 0.5 - presque tout le signal de l'image d'origine -- Valeur test 0.85
+init_image_scale_end: 0.75   # Plus proche de init_image_scale pour limiter le flou
+# guidance_scale = 12 → les prompts positifs et négatifs auront un impact plus marqué.
+guidance_scale: 8.5 # un peu plus strict pour que le chat ressorte 8.0 - 1.5 rendu proche de l'input - 4.0  creatif - Valeur test (2.7)
+guidance_scale_end: 6.5       # légère décroissance pour variation naturelle
+steps: 50 # diffusion steps faible pour VRAM - Augmenter steps ou fps → le motion module a plus de temps pour transformer les latents. (20)
+# -------------------------
+use_n3r_model: true
+use_mini_gpu: true
+n3r_L: 6
+n3r_N_samples: 24
+n3r_L_low: 4
+n3r_L_high: 8
+verbose: False
+
+final_latent_scale: 0.25
+tile_size: 64
+#--------------------latent_scale_boost 5.71 ------------
+latent_scale_boost: 1.0 # Permet d'augmenter le rendu à utiliser doucement :)
+#
+W: 536
+H: 960
+L: 4
+fps: 12 # moins de frames simultanées
+
+# -------------------------
+# Modèles n3oray
+# -------------------------
+# Liste des modèles à utiliser (ordre = interpolation possible) LoRA est FP16 ou compatible avec ton UNet SDXL.
+
+n3oray_models:
+  #cyber_skin: "/mnt/62G/huggingface/cyber-fp16/Cyber skin_fp16.safetensors" #OK
+
+motion_module: "v3_sd15_mm.ckpt"
+
+upscale_factor: 2
+
+pretrained_model_path: "/mnt/62G/huggingface/miniSD"
+unet_dtype: float32
+dtype: float16 # VRAM safe FP16 pour 4Go-
+use_real_esrgan: true
+
+scheduler:
+  type: DDIMScheduler
+  #type: PNDMScheduler
+  steps: 40
+  beta_start: 0.00085
+  beta_end: 0.012
+
+num_fraps_per_image: 12        # plus de frames par image pour plus de cohérence
+transition_frames: 1          # transitions plus douces
+key_frames: 1
+
+
+#80*0.6+80 = 128 → ok.
+block_size: 160 #80 ou 100 ou 110 ou 120 ou (128) ou 156 ou (160 MAX soit 160x0.6+160=256)  160
+overlap: 96   #40 ou 60 ou 66 ou 72 ou 76 ou (80) ou 100 ou (96)  96
+vae_slicing: true
+vae_tiling: true
+vae_device: cpu   # force VAE sur CPU pour VRAM safe
+motion_module_device: cpu
+#motion_module_device: cuda
+
+#   Vérifier le motion module → certains modules “lite” sont trop conservateurs avec des images petites ou des frames peu nombreuses.
+#motion_module: scripts/modules/motion_module_tiny.py
+#motion_module: scripts/modules/motion_module_cam.py
+#motion_module: scripts/modules/motion_module_cam2.py
+motion_module: scripts/modules/motion_module_show_safe.py
+#motion_module: scripts/modules/Motion_module_Masked.py
+#motion_module: scripts/modules/Motion_module_Enhanced.py
+#motion_module: scripts/modules/Motion_module_Wind.py
+#motion_module: scripts/modules/motion_module_lite.py
+#motion_module: scripts/modules/motion_module_ultralite_debug.py
+#motion_module: scripts/modules/motion_ulta_lite_fix.py
+#motion_module: scripts/modules/motion_module_show.py (test ok)
+
+device: cuda
+offload_folder: /tmp/offload
+accelerate: true
+low_cpu_mem_usage: true
+batch_size: 1  # chaque image traitée individuellement
+
+seed: 1234
+
+enable_xformers_memory_efficient_attention: true
+
+# VAE complet pour couleurs correctes
+#vae_path: "/mnt/62G/huggingface/vae/vae-ft-mse-840000-ema-pruned.safetensors"
+#vae_path: "/mnt/62G/huggingface/vae/vaeKlF8Anime2_klF8Anime2VAE.safetensors"
+#vae_path: "/mnt/62G/huggingface/miniSD-fp16/vae/diffusion_pytorch_model_fp16.safetensors"
+vae_path: "/mnt/62G/huggingface/miniSD-fp16/vae/diffusion_pytorch_model.safetensors"
+#vae_path: "/mnt/62G/huggingface/miniSD-fp16/vae"
+
+
+prompt:
+  - "female superhero, consistent character, same face, long flowing black hair, blue eyes, upper body, cyber armor, mask, comic book style Marvel, cel shading, bold outlines, dramatic action pose, neon lighting, vibrant colors, glowing effects, dynamic perspective, cinematic composition, energy lines, stylized shadows, inked lines, comic panel framing, halftone shading, high contrast lighting, happy expression"
+  - "female  superhero, consistent character, same face, long flowing black hair, blue eyes, upper body, cyber armor, mask, Marvel comic style, dynamic pose, neon city background, glowing circuits, vibrant colors, cinematic lighting, dramatic angles, stylized shadows, action scene, inked lines, comic panel framing, halftone shading, high contrast lighting"
+  - "female superhero, consistent character, same face, long flowing black hair, blue eyes, upper body, cyber armor, mask, katana on back, Marvel comic style, synthwave neon city, pink and purple lighting, reflective wet streets, dynamic perspective, cinematic composition, glowing atmosphere, motion energy, inked lines, comic panel framing, halftone shading, high contrast lighting"
+  - "female superhero, consistent character, same face, long flowing black hair, blue eyes, upper body, cyber armor, mask, katana on back, Marvel comic style, angelic transformation, glowing neon wings, radiant energy, floating pose, dramatic lighting, vibrant colors, stylized aura, inked lines, comic panel framing, halftone shading, high contrast lighting"
+  - "female superhero, consistent character, same face, long flowing black hair, blue eyes, upper body, cyber armor, mask, katana on back, Marvel comic style, cyberpunk city night, dense skyline, neon signs, holograms, rainy streets, cinematic composition, dramatic shadows, dynamic perspective, inked lines, comic panel framing, halftone shading, high contrast lighting"
+
+
+n_prompt:
+  - "blurry, deformed, extra limbs, bad anatomy, low quality, messy background, dark colors, scary, realistic, human features"
+  - "blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background, realistic, clear, detailed, structured, photo-realistic, bad anatomy, extra fingers"
+  - "blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background, realistic, clear, detailed, structured, photo-realistic, bad anatomy, extra fingers"
+  - "blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background, realistic, clear, detailed, structured, photo-realistic, bad anatomy, extra fingers"
+  - "blurry, deformed, stiff pose, unnatural movement, distorted anatomy, missing parts, extra limbs, broken motion, low resolution, inconsistent colors, messy background, realistic, clear, detailed, structured, photo-realistic, bad anatomy, extra fingers"
+
+
+input_images:
+  - "input/540-960/1.png"
+  - "input/540-960/2.png"
+  - "input/540-960/3.png"
+  - "input/540-960/4.png"
+  - "input/540-960/5.png"