You said:

You will need to show that you actually have been
using it as such [second home]. In addition, if it's very close to where you live, they
may disallow it.

There is no IRS requirement for either of the above that I am aware of.