I would "bite the bullet" and run the heavy wire. You will end up with
one more battery to mainatin, in a poor location to start with. You

It's 40' to the battery, so there's an 80-foot circuit.

If it's 40-feet to the battery, this is probably a 45 or 50-foot boat, at
least, and an appropriate windlass could easily draw 60-70 amps. That will
require a #2 wire to prevent a loss of no more than 10% of the voltage over
that distance. (It would take 3/0 to get voltage loss down to the 3% allowed
for "critical applications", but the windlass is not in the critcal category.

#2 marine cable sells for about $3 a foot, so even before considering terminal
connections, etc, there's $250 in wire costs alone.

As far as the gauge of the wire needed, some of the new winches draw a
lot less power than older units, it might need smaller wire than you
think. I'd check it out first, that extra battery sounds like nothing
but trouble to me than it's worth. Put the wire in and forget about
it.

There's no need to worry about using the compass or the autopilot at the same
time you're hauling with the windlass, but if there were that 80-foot circuit
could effect every compass aboard each time the windlass were activated. :-)

Avoiding an additional battery for the windlass can be sound advice, but is
less sound when the normal bank is 40-feet away.