The College was founded according to the terms of the will of a Somerset landowner, Nicholas Wadham, who died in 1609. The fulfilment of his rather vague design was due to his widow Dorothy Wadham, a remarkable woman in her late seventies who not only provided a large part of the original endowment, but carried through the negotiations for a royal charter, purchased a site and chose architects to such effect that the original buildings were completed and the College ready to receive students in 1613.

The original building, a striking example of Jacobean architecture, is still the centre of the College, incorporating the main entrance and porters' lodge, the Warden's lodgings, rooms for fellows and students, and the magnificent hall and chapel. The College expanded gradually over the following centuries, taking in and adapting what were for the most part domestic buildings on the periphery of the College.

Since 1950 there have been three major building programmes, adding very substantially to the number of rooms available, expanding social facilities, and providing a modern library building.