The men were arrested during raids at 5.45am at a rubbish depot in central London based on an intelligence tip off received overnight.
The suspects, aged 26, 27, 36, 40 and 50 were arrested by officers from Scotland Yard’s Counter Terrorism Command on suspicion of the commission, preparation or instigation of acts of terrorism.
They are said to be from a variety of nationalities including a number of Algerian origin.
The depot where they worked is less than a mile from Hyde Park where the Pope was due to hold a prayer vigil tomorrow evening.
As street cleaners they would have been able to move relatively freely and inconspicuously through the London crowds.
Westminster City Council said the men worked for Veolia Environmental Services, a contractor which employs 650 on-street staff to keep Westminster's streets clean and free from rubbish.
The men have been taken to a central London police station where they will be interviewed by detectives.
Searches are being carried out under the Terrorism Act 2000 at the rubbish depot and at residential premises in north and east London.
Sources said that initial searches had not uncovered any bomb-making or hazardous items.
A spokesman for the Metropolitan Police said the arrests were made after they received “information” and added: “Following initial inquiries by detectives a decision was made to arrest the five men.”
Pope Benedict XVI’s security arrangements were reviewed following the arrests and Scotland Yard said “we are satisfied that our current policing plan remains appropriate. The itinerary has not changed.”
There was no change to the terrorism threat level which remains at “severe” meaning that a terrorist attack is highly likely.
The Pope continued with his engagements, unaffected by the developments.
The Pope has been informed of the arrests, Vatican spokesman Fr Lombardi said. He told reporters there was "no need to change anything about the programme with the Pope and the security for this trip".
"The Pope is calm and happy and we go on with this trip with courage and joy," he added.
"We are totally confident in the work of the police and Scotland Yard."
A multimillion-pound operation is in place to protect the Pope during his tour of Scotland, London and the West Midlands, with thousands of officers involved.
The pontiff is also protected by members of the Swiss Guard.
He often travels in the Popemobile, which has a bulletproof glass enclosure. The Vatican abandoned the use of an open-topped vehicle after the assassination attempt on Pope John Paul II in 1981.
A spokesman for the Catholic Bishops' Conference of England and Wales said: "We have absolute confidence in the security operation to protect both the Pope and the public.