While the island is easily accessible, you'll have to find your way to the cache. There is a gate and it closes at "dusk", there isn't a set hour it opens in the morning. While it looks like you can get out after that, I wouldn't want to experiment.
You'll pass some interesting ruins nearby, you do not need to cross a fence! Take a moment and walk to the southern tip of the island. You can pay $20 to watch the July 4th fireworks from here. It's supposed to be spectacular.
The cache is a small tupperware container, will fit small TB's. Please hide as well or better than you found it. Keep your eye out for people nearby, they appear quickly and unexpectedly.
You shouldn't need the clue, I'm more interested in your reactions to the location than confusing lots of people. Check out Hudson River Park View for a micro that a lot of people miss.
PS: Thanks to Baldwin Moose for protecting the cache!