He is a member of the University of Miami's Board of Trustees.

In May 2005, the University of Chicago's Board of Trustees authorized funding for a $42 million dollar addition to the library, which was completed mid-2011.

Spencer also served as a member of the University of Cincinnati's Board of Trustees.

He served on and, for a time, became president of University of Pittsburgh's Board of Trustees.

In November 2012, he was selected by statewide election to serve on the University of Michigan's Board of Regents.

A few weeks later he was appointed to the University of North Carolina's Board of Visitors.

In addition, he served as a member of the University of Vermont's Board of Trustees from 1849 to 1853.

He later served as a member of the University of Illinois' Board of Trustees for 17 years.

From 1996 to 2001, he was chair of the University of Calgary's Board of Governors.

He was reprimanded for his remarks by the University of Illinois' Board of Trustees, but they did not try to unseat him.