Defence Secretary Michael Fallon last night said the jihadis had posed a “very direct threat to this country”.
And he vowed those detained would be “brought before the courts”.
IS is a banned terror organisation and being a member is a crime, which carries a maximum 10 years in jail.
Intelligence experts are monitoring the movements of 500 Brits who travelled to Syria and Iraq to join the terror network.
But it is the first time that it has been revealed so many of them have slipped back into the country.
Some of the Brit jihadists in Syria are known to have contacted the UK authorities asking if they can come back after being shocked by the violence displayed by IS. Yesterday House Of Commons leader William Hague, 53, warned of a “mushrooming” threat from IS as more trained fighters may now try and head back to Britain to dodge an aerial bombardment.
Mr Fallon, 62, told BBC Radio 4’s The World This Weekend: “We have toughened up now on the issue of passports and the watch on our ports and airports for returning fighters.