Towards the latter end of the first century the Romans built a lighthouse to guide cross channel shipping round the Foreland to Richborough, which was at that time the principal port. The remains of this lighthouse (or Pharos) still stand another lighthouse was built sometime later on the Western Heights. During the Roman occupancy of Britain a civil settlement grew up near the mouth of the River Dour, to defend the coast from the frequent pirate attacks inflicted on it, a fort with garrison and harbour facilities was built at Dubra (Latin for Dover). After the withdrawal of the Roman garrison in the early fifth century little is known of Dover until, sometime before 640, Eadbald, King of Kent, founded a monastery in castro within the old Roman walls. Largely because of the fact that Richborough was silting up, Dover gradually grew in importance as a port. During this period the next oldest building, St. Mary's Church, was built but it was not until 1064 when Harold's bond came, from which William the Conqueror professed to derive his right to England, and expressly stipulated that he build up on the Confessor's death, a Castle of Dover, with a well of water in it. A few days after the battle of Hastings the Conqueror took the Castle and appointed his half-brother Odo of Bayeux Constable and made elaborate provision for its defence. Upon Odo's downfall, the post fell to John de Fiennes, together with the wardenship of the Cinque Ports. Most of the walls and towers are Norman work, probably Fiennes's or built upon his foundations; the Keep dates from the time of Henry II. It was while resident at Dover Castle that John made his shameful submission to the Papal Legate, Pandulph.

In 1216 the Castle was besieged by Louis, the Dauphin of France, and the allied barons, but was so stoutly defended by the Constable Hubert de Burgh with a mere handful of men that it only needed the arrival of John de Pencester with a reinforcement of 400 horse to make the Frenchmen raise the siege. The incident of Pencester's arrival is stirringly depicted in one of the memorial windows of the Maison Dieu In the reign of Edward I Dover enjoyed the distinction of being the first of the Cinque Ports to be incorporated by charter. This was in 1278, but seventeen years later the town was sacked and burnt by the French during the absence of the English fleet and although the invaders were eventually put to flight with heavy loss, the resources of the port were crippled for several generations. Henry VIII despoiled St. Martin's Priory and the Maison Dieu, but spent considerable sums upon the improvement of the harbour. In the defeat of the Spanish Armada Dover took a leading part, and furnished the fire-ships which were the beginning of the Spaniards trouble. At the outbreak of the Civil War a citizen of Dover Drake or Dawkes by name, conceived the bold idea of scaling the cliff on the side next the sea, where it was not thought necessary to defend it. With a few daring supporters he succeeded in surprising the Royalist garrison, who surrendered under the impression that Drake was backed by a greatly superior force. The King made repeated attempts to retake the fortress, but his forces were repulsed without much difficulty, By one of the little ironies of history it was at Dover of all places that Charles II landed at the estoration. The townspeople's demonstrations of loyalty were so effusive that Charles is said to have remarked to his attendants Oddsfish gentlemen these good folk seem so happy to see us that surely it was our own fault we did not gratify them sooner." Charles was at Dover Castle again ten years later (1670) with his brother, the Duke of York, to meet their sister Henrietta Maria of Orleans. The merry junketings on that occasion, extending over a fortnight, furnish the setting for Anthony Hope's stirring novel, Simon Dale.

When Napoleon was making his great preparations on the opposite coast for the invasion of England, the defenses of Dover naturally came in for a good deal of attention at the hands of the authorities. Never since have the forts been suffered for long to remain neglected, and the defenses generally were strongly augmented during the two World Wars. Fortunately the Castle escaped serious damage in the last war, though subjected to heavy aerial bombardment and shelling attacks. The Castle grounds occupy about thirty-five acres, and consist of an Upper and Lower Ward, both with surrounding walls or curtains.