I would be more concerned about what is in the colored vinyl. Black vinyl uses very finely ground up carbon (graphite) but the dyes mixed in with the virgin vinyl probably contain a lot of metal oxides.
Carbon (graphite) is a lubricant, most metal oxides are abrasive. Aluminum oxide (not used in pigments as far as I know) is used to make grinding wheels. Iron oxide (a pigment) is used in crocus cloth for polishing metals, etc.
Long term, what if anything does it do to your diamond stylus?
Not trying to ring alarm bells, but if I had a pricey stylus, I'd want some answers as to what is in colored vinyl.
BillWojo