How far back should I move the frog for a thicker iron?

First check the frog position rather than filing anything. The frog should support the iron on its flat face. Set it so the frog face meets the sloped rear edge of the mouth, not so far back that the iron or bevel bears on the plane body. If the iron is being pushed forward by the mouth, that can make the opening seem too tight even though the frog is already at its rear limit.

Try this with the iron by itself, without the chipbreaker: loosen the frog screws, place the iron against the frog, and move the frog until the cutting end clears the mouth without touching the sole. If you cannot get that position with the frog seated properly, then you may have a fit or compatibility issue rather than an adjustment issue.

I would not move the frog back until that ridge sticks out. The iron needs to bear on the frog, not get levered by the sole at the back of the mouth. Retract the iron, check that it is bevel down, and make sure the chipbreaker sits closely against it. Then see whether the iron still needs the frog pushed that far back.

If it does, compare the iron and chipbreaker with the original parts. A thick replacement can expose a mismatch, but a little extra thickness by itself should not mean the frog has to be positioned beyond its proper support.

Filing can be a reasonable last resort, but I would not file the frog or its bearing face to make the parts fit. First identify exactly what is touching. If the back of the mouth is interfering while the frog is correctly positioned and the iron is fully supported, carefully removing a small amount from that rear mouth edge may help. Go slowly and recheck the fit often.

Since you want to use different irons, keep in mind that opening the mouth is a permanent change and can affect how the plane behaves with a thinner iron. I would confirm the replacement iron and chipbreaker are assembled correctly before altering the casting.

Before reaching for a file, try the original iron again. If it seats normally, the aftermarket iron or its chipbreaker may simply not suit this plane's geometry. A separate plane for the thicker or specialty iron is less elegant, but it avoids permanently changing a mouth you want to use with other setups.