There are lots of other channels which are connected by the ICW, which you
transit
if following the ICW, which are marked differently

That sort of defeats the "south to north" assumption then doesn't it? The
places I have seen only have one set of markers in any specific segment and
they may switch directions/numbering at every pass to the sea. You are right
about having a good chart if you lack the local knowlege because there are
plenty of places where the markers are confusing at best and contradictory at
the worst.
Personally I think they should require different markers for ICW, Local
government and privately maintained but they are all identical.
Here on the west coast of Fla we don't even have a decent standard about how
deep a marked channel has to be. The Estero River channel across Estero Bay
uses ICW style markers and you can see wading birds in the middle on a winter
low tide.