I have never liked fast things. Cannot stand going over 14 miles an hour, and I will never get over it--And I have ridden down the Spanish end of the Alps (forget what they are called.) I've decided other people need to get over me riding slow, not vise versa.
And I always use just my back brakes. I know this is a bad habit that I have tried to break and failed. But I don't ride them, and I make sure they are always perfectly adjusted, and I never skid because I never go fast.
I think you just need to get very confident of your braking ability. Stop riding them, practice alternating left and right so you always have the feel of both in case one fails. And learn to adjust and check them yourself.