Stick with the heavier line. Put some rollers on the stays.
I've seen short lengths of PVC slipped over the stays. Not pretty
but it works.

Another option, you could splice in some more core in the section of
the line where it meets the winch. That would be more difficult with
older line but might be worth the effort.

"Scotty" wrote
I have Lewmar ST 40 s on my boat. The jib sheets are big and
fuzzy ( 1/2'' IIRC ) soft to the touch, but they 'hang up'
on the stays in light air. I bought some smaller line (
Stayset 3/8 IIRC ) . It sorta works with the self tailing,
but slips half the time, or comes out of the ST part
completely. Someone suggested the new line is too slippery
and maybe it will wear in. Is there anything I can do to the
winch to remedy this situation?