Francis and Elizabeth had twelve children but only six survived to adulthood, all of whom were daughters;
+ Frances (1572-1665) m. John Drake m2. Montague Wood
+ Bridget (1566-1629) m. Percival Willoughby
+ Dorothy (1574-1632) m. Henry Hastings
+ Margaret (1570-97) m. Robert Spencer
+ Abigail (1576-1654) m. William Pargiter
+ Winifred (b.1578) m. Edward Willoughby
The couple had a number of sons, however none of them survived childhood; the last died aged six in 1580. Due to his lack of a son and heir, in 1583 Francis had his daughter Bridget married to a cousin named Percival Willoughby, and made him his heir.
Francis and Elizabeth's marriage was a disastrous one and it appears that both sides were at fault. Arguments turned into abusive behaviour. Francis placed restrictions on his wife; Elizabeth was allowed access only to her rooms and Francis refused to provide for her attendants. She was allowed to only occupy herself with raising her daughters, needlework, reading, playing the virginals, card games and conversing with her attendants. Elizabeth was almost constantly ill and often visited her physician in London and took visits to Buxton, which was an expense that Sir Francis was not happy to maintain. By the 1570's the couple's relationship was crumbling and only worsened throughout that decade. In 1572 Elizabeth only had a handful of women attendants in her household; Elizabeth Mering and Marjory Gardner - two gentlewomen to attend her, two nurses for the children, a fool named Mary and two other women, whereas her husband had around 50 men in attendance. This situation worsened over the decade; eventually Elizabeth found herself under house arrest by her husband; he complained that she led a 'disorderly life', kept company with people he disapproved of and continually reviled him to his face. These reports may not be entirely inaccurate as when Elizabeth was asked by Sir Fulke Greville why she refused to be ruled by her husband she replied that she was 'the Queen's sworn servant and knew not but Sir Francis might command her something against Her Majesty's proceedings'. She was confined to her own rooms, was not allowed to 'discharge or receive any servant' or 'strike or evil entreat any servant' and she was not allowed to see her children who were then in the care of their nurse Joan. Elizabeth had to follow the orders of two household Captains, was banned from the household stores and from buying anything and had 'no authority to command anything in the house except necessary diet for herself'. It didn't help the situation that Margaret was making the couple's arguments public knowledge, and the household servants were also interfering and taking sides.
The couple separated in 1579, and Sir John asked Francis to provide an allowance for Elizabeth to live on. When the couple's only son died in 1580 Elizabeth offered to reconcile with Francis in the hope that they could have another son, but this did not happen. In 1582 Queen Elizabeth ordered Francis to pay his estranged wife £200 per year 'for Elizabeth's separate maintenance'. Elizabeth went to live with her father and had to rely upon him for 'comfort in my griefe, assistance in my troubles and succor in my necessities'. During the years of her separation from her husband, the malicious rumours against Elizabeth eventually convinced even her father of her bad character, and she was required to plead with him to 'take pitie of me as yor naturall childe, have compassion of me as a distressed woman'. Elizabeth became unhappy with this separated state of her marriage and often wrote to Francis asking him to take her back, despite all of their previous marital disharmony. Francis fathered an illegitimate son in 1585. Francis and Elizabeth reconciled in 1588, however Elizabeth's health deteriorated and her behaviour became the same as before. Her visits to Buxton recommenced but seem to have had little effect; Francis writing in April 1589 that 'my wife hath beene longe sicke, and for the recoverie of her health is at Buckstones, wheare havinge receaved noe healpe is growne to suche weakenesse, that nowe beinge desyrous to retorne home is not able to adventure the iorneye eyther on horsebacke or in a coache'*. Elizabeth died on the 4th of June 1594.
In August 1595 Francis married an 'astute widow', Dorothy Colby, daughter and heiress of Thomas Colby. Dorothy was the widow of John Tamworth (1562-94) whom she had married in 1583 and had two daughters, Dorothy and Katherine, with. Dorothy and Francis were married for only fifteen months before Francis' death on the 16th November 1596. There were suspicions that Dorothy had murdered her husband with poison, however she was never formally accused. At the time of his death Dorothy was pregnant. She gave birth to a daughter Frances on the 3rd May 1597, however the child died soon after. Dorothy married her third husband Philip Wharton (1555-1625) within two months of giving birth to Frances. After an unhappy third marriage, Dorothy Wharton died in 1621.
For a more complete view of the family, Francis and Elizabeth's descendant Cassandra, Duchess of Chandos, compiled and published the family papers which give an intimate look at the couple's marriage.
* = Letter from Francis Willoughby to Bess of Hardwick, 26 April 1589