Are these caterpillars going to ruin my dill?

They sound like black swallowtail caterpillars. They use dill as food and can eat a lot of the plant, but they grow into butterflies. If you can spare the dill, leaving them there lets them feed.

Good for the butterflies, hungry for your dill. If you need the plant for cooking, you could keep extra dill elsewhere as a caterpillar patch and protect the plants you want to harvest. Just expect the caterpillars to take a noticeable share.

I’d leave them if you have enough dill. They may strip much of it, but the plant can recover, and the caterpillars are the larval stage of swallowtail butterflies. If losing the dill would be a problem, decide how much you’re willing to share before they get bigger.