The extension is quietly corrupted

[Sakura-Luna]

With the update of the extension, some results can no longer be reproduced, the problem is now worse, it no longer works on certain models, the results will gradually become pure black within a few iterations. I started rolling back the WebUI and extensions and even thought there was a hardware failure XD until I rolled back the extensions to 741a4d9 and now it works.

[Sakura-Luna]

This problem can be reproduced stably without a specific sample, the following is a reference, you'll get half of the image that's almost completely black.

1girl
BREAK <Lora::1>

RP Active: True, RP Divide mode: Vertical, RP Calc Mode: Latent, RP Ratios: "1,1", RP Base Ratios: 0.2, RP Use Base: False, RP Use Common: False, RP Use Ncommon: False, RP LoRA Neg Te off: False, RP LoRA Neg U off: False

You also get corrupted results using base, just not the same.
sample model: https://we.tl/t-Riuv9OOcF6
test environment: Latest releases or fork

[Sakura-Luna]

I've found the extension to perform poorly on a class of lora, it's easier to get poor nonsensical results with these models, and rolling back to 741a4d9 is just a reduced impact rather than a fix. If I want to reproduce old results, I also need to roll back to earlier.

[Sakura-Luna]

This is an example of using the prompt above, but the weight of lora is slightly lower. With

Steps: 20, Sampler: Euler a, CFG scale: 7
RP Calc Mode: Latent, RP Ratios: 1, RP Base Ratios: 0.2, RP Use Base: True

you can see it gradually changes in the preview Dark, if you turn up the weight it will end up being a completely black image.

[Symbiomatrix]

Which lora is it? That wetransfer platform is requesting registration for some reason.
Well, we do know loras tend to have reproducible corruption issues in latent mode, and couldn't find the cause for it yet. Darkness resembles the green noise I got from megumin, though mine was mostly one sided - the image you posted seems to be on the same track, with a visible cutoff point by the leg.
The only modification to the extension since then that I believe makes a difference in this case, is the addition of optional textual + unet negation in base; for reproduction of previous version results, tick those two (the default behaviour was changed to not negate by the looks of it, perhaps unintended). "reduced impact rather than a fix" sounds about right. Still, I think we should probably flip the flags since reduced impact on regions seems better than loras being a perfect match in base=1.

If I want to reproduce old results, I also need to roll back to earlier.

Have you found any version in which two latent loras work correctly in separate regions?

It'd be interesting to see if moving the lora to the base, with a low base weight, both settings off, produces a decent output.

[Sakura-Luna]

Which lora is it? That wetransfer platform is requesting registration for some reason.

I have never seen a download that requires registration. You may have clicked on the wrong location. Just click "download". The original website is not working every day now, and I can't send it out.

Well, we do know loras tend to have reproducible corruption issues in latent mode, and couldn't find the cause for it yet.

In the preview, it can be clearly seen that it gradually becomes black, as if some result is lost. Using a single lora to cover the base will turn the whole image black, while single lora and single prompt partitions often have one of them losing almost all content.

[Sakura-Luna]

The only modification to the extension since then that I believe makes a difference in this case, is the addition of optional textual + unet negation in base. Still, I think we should probably flip the flags since reduced impact on regions seems better than loras being a perfect match in base=1.

On some samples the on option did improve, maybe it should be reversed. @hako-mikan
But even then can't reproduce the result, there are still some differences, it's strange what else interferes with it, its size is indeed a multiple of 8.

Have you found any version in which two latent loras work correctly in separate regions?

I'm not confident what version is normal now, because I haven't gotten the same result, and it may cause confusion if there are too many versions rolled back. At present, I can say that 741a4d9 is slightly better. Maybe I need to test last month's version.

It'd be interesting to see if moving the lora to the base, with a low base weight, both settings off, produces a decent output.

This is equivalent to reducing the lora weight in disguise, and the same is true for direct reduction.

[Sakura-Luna]

I tested the version that just added Latent mode, and it still gives different results than 741a4d9, just by a relatively small amount.

It seems that this is actually two issues, the difference introduced by the update and the compatibility with some lora.

[Sakura-Luna]

I tried to specify the weight of Lora in the negative textencoder and U-net, and found that it has a very large impact, which can make the results completely different, and it is even possible to get not only pure black but also pure white output, so I think it is necessary to specify it.

On some of my samples, specifying intermediate values resulted in a better blend.

[Symbiomatrix]

I tried to specify the weight of Lora in the negative textencoder and U-net, and found that it has a very large impact, which can make the results completely different, and it is even possible to get not only pure black but also pure white output, so I think it is necessary to specify it.

On some of my samples, specifying intermediate values resulted in a better blend.

Thanks for the experiments.
Yes, that's what I was trying to explain in the other post, about loras seeming to unbalance to model. causing the results to diverge - all black, all white, all green, I believe these all stem from the same phenomenon - absolute values growing too large over forward iterations, a positive feedback loop.
Tweaking the unet weight is a good start, but wouldn't be too effective when multiple loras are combined.
If this counteracting weight is dependent only (or mostly) on the lora itself, and not on the keywords in effect, then it might be a good idea to store these weights per lora in presets. Or better yet, find a way to normalise the lora automatically.
If not, sounds to me like a tweaking nightmare. But then, the user's already tweaking the lora weight anyway and the prompt's, so it might be acceptable.

I have an idea, will see if I can code it tomorrow.

[Sakura-Luna]

If this counteracting weight is dependent only (or mostly) on the lora itself, and not on the keywords in effect, then it might be a good idea to store these weights per lora in presets. Or better yet, find a way to normalise the lora automatically.
If not, sounds to me like a tweaking nightmare. But then, the user's already tweaking the lora weight anyway and the prompt's, so it might be acceptable.

It is more reasonable to specify the negative weight of each lora separately, but as you said, the adjustment is very frequent and the meaning is not clear. Saving to a preset doesn't seem to make much sense, do you have evidence that the preset values can still achieve good blending results when multiple loras are blended?

I also have two ideas here, maybe we can try to determine a relationship and slightly change the negative weight according to the number of lora used, but this requires a lot of testing; another idea I think is better, whether we can make a lora in the negative weight calculation Affect other lora areas, set a level to make the effect blend, so as to maintain the overall consistency.

[Sakura-Luna]

@Symbiomatrix Do you have any progress, I want to observe the effect of gradually changing the lora negative weight in sampling steps, but the extension does not seem to have this kind of intervention.

[hako-mikan]

It's a difficult LoRA to deal with. I've tried a few things and it seems to improve with smaller values of the CFG scale. assuming a value of 3 or so gave me good results.

[Sakura-Luna]

@hako-mikan This Lora has obvious overfitting and is not compatible with many models, but it is just an example, we don't need to achieve good results on it.
It's worth considering if there's any way to improve some Lora's results on Latent mode. Considering that this kind of degradation is accumulated gradually, some adjustments may be made during iterations, but I am not familiar with the SD process, and I am at a loss.

[Symbiomatrix]

Hello.
Well, I haven't gotten around to implementing my idea yet, which is simply making the unet/text weights a list, applied to loras with broadcasting rules so we could better test different combinations of loras.
I did however come across this curious bit:

• In denoised_callback, we're setting layers of input to each region with filters, and the remainder to the negative conditioning layer (created by gluing uncond to tensor in sd_samplers_kdiffusion.forward) which remains unchanged itself: x[a + b*areas, :, :, :] = x[a + b*areas, :, :, :] * self.filters[a] + x[x.size()[0]+(b-batch), :, :, :] * self.neg_filters[a]
• After some processing in sd_samplers_kdiffusion.forward, the layers are combined in combine_denoised to one, via an addition method whilst subtracting the negative weights from before: denoised[i] += (x_out[cond_index] - denoised_uncond[i]) * (weight * cond_scale)
• This "weight" seems to be 1 for all layers, cond_scale is cfg. Meaning, every layer outside the region should be getting a value of 0 within the region, whilst the region layer gets full weight.

So what I'm wondering is, whether this method of calculation might be thrown off by the introduction of loras. Theoretically, if everything works as it should, then a lora should have only affected its own layer adversely, but what we're seeing is that non lora regions are also impacted (and especially when unet is nonzero). Therefore, it might be that uncond is altered, or the difference is nonzero (which would mean the more regions you have, the greater the cumulative effect).

Edit: It doesn't look like a nonzero difference, Like hako said, a low cfg over a large number of steps suffers less corruption, and I can see why it might. For lora combination though, still pretty bad, so that might be something else.

[Symbiomatrix]

Another thought: If we were to create multiple versions of the uncond layer, one for every region, and match them, would that help the separation? Is that feasible?