Jim Doty started his athletic career in track and field earning a varsity letter three years at Harvard throwing the hammer and putting the shot with 1960 Olympic gold and silver medalists Hal Connolly and Al Hall.

In 1956, he finished 78th in the Boston Marathon and decided that longer was better.  He had been spending his summers on Black Cat Island, Lake Winniepasaukee, New Hampshire.  One time he had to swim one half mile for help after his boat broke down.  This swim coupled with his 26-mile marathon run and a liking for long distances, Jim started swimming long distances in 1968, training with Jack Starrett, 1964 English Channel Swimmer.  He trained a couple of summers with Hall of Fame coach Charlie Silvia at Pine Knoll Swim School in Massachusetts.  He went on to swim across the lengths of most New Hampshire lakes.

Then he started on the professional marathon swimming circuit doing the 24 hour La Tuque and 28 mile Chicoutimi swims in Quebec.  He joined the World Professional Marathon Swimming Association tour with swims in Rhode Island, Chicago, Hamilton (Ontario) and Lac Simon (Quebec).  He has done the around Cape Ann (Gloucester, MA) (12h 31m), Boston Light (14 times) (5h), double Graves Light (11h 45m) and double Boston Light (10h).

In 1978, Jim incorporated the New England Marathon Swimming Association (NEMSA) as a charity to promote swimming and study water conditions.  It re-started the Boston Light race on an annual basis, the previous race being held before World War II.  Hall of Famer Annette Kellerman gained fame doing this swim.  The Association administers various local swims in the Boston area.

In the 1980s, Jim joined the "L" Street Brownies Frozen Water Swim Club started in 1902.  It promotes "polar bear" swims in frozen lakes by chopping through the ice.

Through his efforts, Jim continues promoting and competing in open water swimming events in the New England area. He is the keeper of the records and archives for the area.