About Models from SD to Flux

[LIU-Yuxin]

Thanks for the nice work!
I wonder if you have tried to train a model to convert SD latent to Flux latent, and if you got some observations (or problems)? It would be very helpful if you could share some experiences as I am thinking of training such converters myself and would like to know if you have run into any feasibility issues.

[city96]

I did indeed try it originally, but the model itself seems too limited to reconstruct the flux latent properly. The sd1 and to some extent sdxl latents are fairly low complexity compared to the flux ones, so the current simple convnet version can match them fairly well. The flux latent on the other hand stores a lot more data (even if we don't look at the complexity of the latent, the actual latent itself is 16 channels instead of 4 at the same downscale ratio, so 4x as much data in total. So flux -> sd1 has a lot of extra data to work with, whereas sd1 -> flux has to make up and interpolate for the missing parts, which show up as artifacts in the output).
At least with the current model & codebase, I was never able to get it to output acceptable looking images when converting from sd1 to flux.
I guess a secondary reason is also the fact that running flux takes ages compared to models like SDXL and SD 1.5, at which point the VAE decode & encode passes barely take up any time, at least on my card :,)

[LIU-Yuxin]

Thank you for the detailed response! I am trying the training code on some initial experiment, and the experiment also reveals certain issues like you mentioned. Now I am trying further on enlarging the capacity of the network and adopting different training hyperparameters.
On the other hand, I noticed that you have a v1 to v3 converter, did it encounter similar issues like the flux one, since it is also 4 channels to 16 channels?