A highly-criticised undercover police unit was so secretive even Scotland Yard bosses were unaware that it existed, it has emerged.
The force's shadowy Special Demonstration Squad was kept under wraps by Special Branch commanders from more senior officers including deputy commissioners and commissioners during the four decades it was in operation.
It has come under fresh criticism today for keeping records of personal information linked to bereaved families who were fighting for justice after loved ones were murdered or died after contact with police.
A new report revealed that references to 17 campaigns dating between 1970 and 2005 had been discovered, in addition to records already unearthed about groups fighting for justice for murder victim Stephen Lawrence, and warned that more may emerge.
Relatives of Brazilian Jean Charles de Menezes, who was shot in 2005 after police mistook him for a terrorist, have said they may sue the force. The families of Cherry Groce, whose death sparked the Brixton riots, and Ricky Reel, who died in mysterious circumstances in 1997, were also mentioned in the records.
Derbyshire Chief Constable Mick Creedon, who is leading an investigation into the SDS, said: "What is surprising to me is the number of people, the most senior levels in the Met working in covert policing, working in public order command, who did not know about the unit at all."
He heavily criticised the SDS, Special Branch and Metropolitan Police senior management for the fact that the information was kept at all, calling the scale of the record keeping by the unit "staggering".
One reference was to an unnamed person planning to go to a funeral, even though "there was no intelligence to indicate that the funeral would have been anything other than a dignified event", the report said, and Mr Creedon confirmed that there were "more personal examples".
He added: "Quite simply put unless the information could have prevented crime or disorder it should not have been retained and certainly not for the period it has been.
"I can understand why this is likely to be distressing and astonishing for those families and friends who campaigned often for years for justice - to know that details of your deceased or innocent family member and your campaign was mentioned in reports stored - often stored for years - in Special Branch records. This must seem inexplicable."
Scotland Yard was found to have ignored national guidelines on the storing of information.
The report said: "It is quite clear that maintaining the secrecy of the unit and protecting the identity of the officers was of paramount importance to all involved - and in being so focused on this aspect the management of the SDS, of the Metropolitan Police Special Branch and ultimately the Metropolitan Police Executive Leadership of the day collectively failed."
So far investigators have spoken to three former commissioners as part of their inquiry, which is expected to last for another year.
Mr Creedon said: " We have spoken to officers who held the rank of Commissioner and Deputy Commissioner, along with other very senior officers involved in covert policing and public order and the vast majority of them knew nothing of the existence of the SDS.
"This is astonishing given the suggested pivotal role the SDS played in reporting on violence and extremism and preventing disorder and the question has to be asked about the lack of executive scrutiny, of intrusive senior management and of effective supervision."
Operation Herne is continuing to examine the culture of the unit and will later give details of "some behaviour that was not appropriate", the report said.
Former SDS officers are already facing possible criminal charges for allegedly tricking women into sexual relationships, and have been accused of using dead children's identities without permission while undercover.
Today's report said: "Over the 40 years that the unit existed, senior Metropolitan Police management of the day either knew nothing about the existence and activities of the unit or, when they did, they appeared to have allowed the SDS to exist in secret isolation in a manner that was complacent and possibly negligent."
Assistant Commissioner Martin Hewitt from Scotland Yard stressed that undercover officers had infiltrated violent groups that tried to align themselves to legitimate campaigns, rather than the peaceful groups or families themselves.
He said: " I can absolutely apologise for the distress that this has caused the families. We need to understand all the facts of everything that has gone on.
"I really do hope that today's report which is categoric in stating that undercover officers were not tasked to go and gather intelligence directly from the families or directly from those groups that were legitimately supporting their cause, will go some way to reassure people."
Chairman of the Home Affairs Select Committee Keith Vaz said he would write to former Scotland Yard bosses to ask what they knew about the unit.
"I am shocked by the potential extent of this spying," he said. "The police should never snoop on vulnerable people to gain information, least of all on grieving families. Their role in these circumstances is to help not to hinder.
"It is vital that every instance of unacceptable practice is investigated and those responsible are held to account.
"As we have discovered in our hearings, despite Operation Herne costing £3.5 million, involving at least 26 officers and taking almost two years, no-one has been prosecuted and no senior officer from the Met has appeared to have taken responsibility.
"We have asked Mr Creedon to come before the committee when the House returns in September to update Parliament on these issues. I will also be writing to all of the previous commissioners to understand what they knew about these activities."
Shadow home secretary Yvette Cooper said: "For the police to gather secret information on victims' grieving families is appalling.
"Whilst covert police operations do have a role to play in modern policing, this report clearly demonstrates the need for robust management, oversight and processes to prevent this kind of abuse.
"Last summer, we called for a tightening of the oversight of undercover operations, including independent authorisation for long-term operations and spot inspections by a new stronger police standards authority to ensure the conduct of officers is appropriate.
"This report highlights why such measures are urgently needed if we are to maintain public confidence in policing."