One thing to remember: those people came as conquerors, not as immigrants. They imposed their own languages and cultures on the survivors of the conquered lands. And it wasn't only English speakers. Many parts of the Caribbean and West Africa speak French to this day. Most of Central and South America speak Spanish or Portugese. Even Canada retains a remnant of French colonialism in Quebec. And let's not forget, French and Spanish and Italian, of course, are derived at least in part from the Latin spoken by their Roman conquerors. Even German and Russian have Latin influences, notably the titles Kaiser and Czar, which come from the Latin Caesar.