Any Future Plans To Add New Models?

[VectorKnight]

Hi there!

Just curious if there might be any plans to add the newly released MatAnyone2 as a foreground selection option or Omnimatte Zero as a more advanced intelligent fill to DiffuEraser?

[calledit]

Looks interesting!

MatAnyone2 is a matting framework, its far better masking would not help with erasing objects as in-painting models generally want masks that are dilated a bit so they cover more than just the object. It could ofc be useful if you only use VideoVanish for mask creation which is a valid use case.

Omnimatte Zero looks amazing, i can't test it out as I only have a 24Gb GPU. It says you need a 32GB+ VRAM GPU in the README.

[VectorKnight]

Ahhh damn! (re: 32GB VRAM woes). I'm more than happy to test builds (have access to an RTX Pro 6000 through work) but understand that would be annoying/potentially not worth the hassle.

I guess I was wondering if MatAnyone2 could be implemented almost like a one shot transparency layer, and then a mask expansion could be made to create the clean plate area that DiffuEraser/Omnimatte would fill in? So the end result would be all the layers you'd need to then move on to replacing a character or an object, or modifying the individual layers separately? MatAnyone2 for a really crisp/detailed transparency layer of the character or object, and then DiffuEraser/Omimatte for the perfect clean plate where the person/object had been?

I definitely could see a ton of potential use cases for that kind of video prep.

[calledit]

I guess foreground extraction would fit pretty well in to the use case of VideoVanish. Which is what you are essentially suggesting if i understand you correctly.?

but understand that would be annoying/potentially not worth the hassle.

The thing is that I want VideoVanish to be usable on consumer hardware.

[VectorKnight]

Yep! Foreground extraction would be a great feature.

Here's another potential video in-paint that's even newer than OmnimatteZero but I'm not sure what the VRAM requirements are: https://henghuiding.com/EffectErase/

On the topic of consumer hardware, I do agree that making software that can only run on 32gb+ of VRAM isn't something that the vast majority of people will be able to use locally. However, most users are likely trying to process videos at higher resolutions than 720p, and that's where the VRAM bottlenecks do tend to appear with existing models. There may be a way to implement a tile based or deferred RAM rending approach to allow smaller cards to handle outputs up to a production-ready 4K resolution (Omnimatte, for example, is based on the LTX 0.9.7 diffusers which can theoretically get up to a 4K output with the right hardware while it seems like EffectErase uses WAN 2.1) but having the overhead room in the build for the user to push their hardware specs also opens up the possibility of running it on Runpod or MassCompute for the projects where they want a full high res output.

There's also nothing wrong with having a warning for non-DiffuEraser options that warns the user about the VRAM requirements of other models while still offering that functionality.