What you are missing is windage aft to counteract your bow windage at low revs and reduced rudder effect.

Any extra windage abaft the centre of effort will help. One surprisingly efficient trick is to set a small sail (an old dinghy jib would do) on the backstay when you want to back in the conditions you describe. You can use the topping lift if you have a rigid boom strut or any other spare halyard - set doesn't matter.

Let it flap until you want to turn, then haul it to weather. It will spin you round into the wind quite easily and you have perfect control according to the amount of weight you put on the sheet (you only need one) When you want to stop turning, let fly and drop the sail at your leisure. A little practice in open water helps, as this is a powerful way to turn the boat. Give it a try.