All-the-time-all-wheel-drive seems a no-brainer.

Is there a downside?

The down side is when you "lose" a wheel, you lose all 4. You car becomes a
hockey puck.