I was always taught that the towel’s final role is to abrade and collect the dirt/grime/skin that has been loosened by showering, but that wasn’t washed away (which iirc is mostly just the skin and oily grime, not dirt or other large particulates).

This is nasty to me. Do you not scrub yourself in the shower? Like, if you have dirty hands, and wash them in the sink, you can dry your hands on a white towel and it should not get dirty. Same for your body except you usually have more scrubbing tools to exfoliate your skin and remove the dirt so you should be even cleaner come dry time.