Justice Secretary Chris Grayling has denied the prison system is in crisis despite concerns about violence and the suicide rate behind bars.
The Cabinet minister acknowledged that the service is facing "pressures" and is having to cope with an increased number of prisoners, but insisted "we are meeting those challenges".
However, Prison Governors Association president Eoin McLennan Murray said staff shortages mean it is impossible for some jails to run a "safe, decent regime".
Last month, figures released by the Ministry of Justice painted a bleak picture of English and Welsh jails, with high levels of violence, including attacks on prison staff hitting the highest level for six years.
But Mr Grayling told the BBC: "We've actually got a prison estate where violence today is at a lower level than it was five years ago. We've got challenges from an increased population that was not expected in the last 12 months.
"We are meeting those challenges, we are recruiting more staff - but I'm absolutely clear there is not a crisis in our prisons.
"There are pressures which we are facing but there is not a crisis."
He said cost-cutting measures were based on proposals from the prison unions as an alternative to privatisation: "I accepted how we should do things, as recommended by our staff, and that is what we are doing .
"So the approach we are now putting in place in our prisons is what was recommended by the unions and by staff."
Mr McLennan Murray told the BBC Radio 4 Today programme there is a shortage of staff and "we haven't been able to recruit in the numbers that we need in order to run safe, decent regimes".
He said the lack of staff means "we have to lock prisoners up for more time than we would want to" and "many prisons are having to run restricted regimes".
Mr Grayling's comments came as i nspectors raised fears over levels of violence at a jail with a high population of gang members, where a shortage of staff is highlighted as an issue.
In 2013, there were 254 fights and assaults at Isis prison, which holds young men aged 18 to 30, with 120 in the previous six months, Her Majesty's Inspectorate of Prisons (HMIP) said.
The report on Isis, in Thamesmead, south east London, found that many of the incidents were serious, some were gang-related and a higher proportion than usual involved weapons.
Chief Inspector of Prisons Nick Hardwick said: " The location of the prison, the volatile population it holds and continuing staff shortages mean the risks and challenges it faces are significant.
"Opening any new prison is complicated and although Isis has now been open for three years it is still in a settling phase."
In late 2013, staff shortages led to a restricted emergency regime which, although intended to be temporary, was still in place at the time of the unannounced inspection in February.
Inspectors said the restricted regime had presented a temporary solution by keeping identified gangs apart.
However, the report added that this was "an unsophisticated plan and inadequate as a long-term solution".
However, inspectors did praise the high level of prisoners - three-quarters - who go into employment or training on release.
Michael Spurr, chief executive of the National Offender Management Service, said: " Isis manages a difficult population of young adult men - many of whom are serving sentences for violence and have links with London gangs.
"The governor and his staff have done some excellent work with the police to challenge gang affiliation and are committed to tackling and reducing violence.
"As the chief inspector makes clear, Isis is an improving prison. In particular, it is doing really good work to support resettlement. Three-quarters of prisoners go into employment or training on release - this is crucial for successful rehabilitation.
"Recruitment is taking place to address the staff shortfalls and the prison is currently receiving detached duty support from other prisons to ensure it can deliver a consistent and safe regime."
Shadow justice secretary Sadiq Khan said: "Chris Grayling is still burying his head in the sand about the crisis in our prisons.
"Under David Cameron and Chris Grayling there has been a complete leadership gap in the criminal justice system, which has led to deteriorating jails, increasing violence in prisons and less and less being done to rehabilitate offenders and treat those suffering with poor mental health.
"The Government needs to listen to these warnings from the chief inspector about the deteriorating situation in our prisons and act now to prevent putting public safety at risk.
"We simply can't afford to go on like this. Five more years of the Tories would mean five more years of failure in our prisons."