Harrell was born to musician parents in New York City: his father was the baritone Mack Harrell and his mother, Marjorie McAlister Fulton (1909–1962), was a violinist. At the age of eight, he decided to learn to play the cello. When he was 12, his family moved to Dallas, Texas, where he studied with Lev Aronson (1912–1988). After attending Denton High School, Harrell studied at the Juilliard School in New York with Leonard Rose and then at the Curtis Institute of Music in Philadelphia with Orlando Cole. He made his debut in 1961 playing with the New York Philharmonic Orchestra at Carnegie Hall.