Does Cigarette Smoke Linger in Granite Countertops?

Hey everyone! We recently moved into a new home where the previous owners were smokers. While the house itself doesn't smell like smoke, I've noticed that my paper towels turn yellow when I wipe down the granite countertops. Could it be that the granite is holding onto smoke residues and slowly releasing it? I've also seen similar yellowing on the bathroom walls. Any insight on this would be appreciated!