The song was written in 1962 by Chad Stuart, who comprised Chad & Jeremy with Jeremy Clyde; it is the first song Stuart ever wrote. Stuart shares the song's writing credit with Wendy Kidd, the manager of a band Stuart belonged to at the time of the song's writing; according to Stuart, Kidd was given the songwriting credit in return for allowing him to compose "Yesterday's Gone" on her piano. Kidd also facilitated Stuart's being hired as a staff writer at Rogers Music, which published "Yesterday's Gone", although the song remained unrecorded until Stuart and Clyde began performing as a duo, eventually recording "Yesterday's Gone" in July 1963 in a session at Abbey Road Studios produced and arranged by John Barry, who'd discovered Chad & Jeremy at a London club and signed them to Ember Records, a newly formed independent label in which Barry was a partner. According to Chad Stuart, John Barry was unhappy with the duo's first attempts to record their vocal for "Yesterday's Gone": "he told us...we sounded like a locker room full of football players...in the end in desperation he said: 'Whisper it', so we kind of backed off a bit and so that sort of slightly sotto voce sound came about".