While sailing to the island that was later named for him, explorer Adrian Block discovered Fishers Island in 1614. Block may have named the island for one of his navigators, a man by the name of Vischers. The future governor of Connecticut, John Winthrop Jr., purchased the tiny island two miles off the Connecticut shore in 1644. When Winthrop became governor of Connecticut in 1657, he ensured that Fishers Island was included in the state's royal charter. But the ownership of the island was soon confused by another royal charter in 1664, granting the Duke of York Fishers Island in addition to Long Island. It was the beginning of a 200-year battle for ownership.

After invading Long Island, the troops blockaded the Sound and, throughout the occupation, raided Long Island's adjacent islands such as Fishers in search of food and forage. The king's soldiers stole nearly 100 sheep and some cattle from farmers before its residents removed their herds later in 1776 to the safety of the Connecticut shore. On a particularly merciless foraging expedition in 1779, the troops burned homes on Fishers Island before leaving with what booty they could find.

In 1879, a joint committee of officials from New York and Connecticut awarded Fishers Island to New York. Despite the declaration, residents of Fishers Island even today identify more readily with Connecticut, just two miles away and connected by regular ferry service, than New York, about 10 miles to the southwest. With fewer than 300 year-round residents, Fishers Island is primarily a remote summer getaway for the extremely rich with names such as Whitney, du Pont and Firestone.