Yes, mold spores can travel on clothing, and I learned that after wearing clothes from a damp, musty space. At first, I did not think much of it, but later I noticed the smell had come with me, and I realized the spores could move from my clothes to other parts of my home.
What helped me was changing right away, keeping those clothes away from clean laundry, and washing them as soon as possible. If the fabric can handle it, I use warm or hot water and dry everything fully. That simple routine made me feel more in control and helped stop the problem from spreading.