Yellow mushrooms popped up in my money tree pot. Should I do anything?

If the tree is otherwise doing well, the mushrooms usually aren't a threat to it. You can leave them or pluck the visible mushrooms out. They may come back, since removing them doesn't necessarily remove the fungus in the soil. Don't eat them.

The mushrooms aren't a reliable overwatering alarm on their own. They can grow in ordinary potting mix with organic material, and moisture helps them appear. Check how quickly the pot dries and make sure it drains, then water based on the soil and the tree's needs rather than the mushrooms alone.

I'd remove them if you don't like the look or don't want spores dropping around the pot. A little surface cleanup is enough. Repotting seems like overkill unless there's another problem, such as persistently soggy soil or trouble with the plant itself.