There has been no attempt to curtail prisoners' access to books, Chris Grayling has insisted in an open letter to poet laureate Carol Ann Duffy.
The Justice Secretary said he had only ever been involved in one discussion on the availability of reading material behind bars - and that was to allow EL James' erotic thriller Fifty Shades Of Grey to be circulated in women's prisons.
The row over posting books to prisoners has been triggered by a ban on sending parcels, which Mr Grayling stressed was a vital security measure.
The poet and other leading figures from the arts world staged a protest against the ban, which came into force in November last year, outside the gates of Pentonville Prison in London.
In his letter Mr Grayling said: " Firstly, neither I nor any other minister have made any policy changes specifically about the availability of books in prisons.
"Despite some reports, we have not sought to include them in a list of privilege items that have to be earned by offenders - to do so would be wholly wrong.
"The only discussion about prison books that I have been involved in as Secretary of State was to agree to make available the novel Fifty Shades Of Grey in the libraries in women's prisons because I judged that it might help encourage some women offenders to read more, something I regard as highly desirable."
Mr Grayling said the ban on sending parcels was designed to make sure there were consistent rules in all prisons.
He said criminals found "ingenious" ways of smuggling contraband into prisons, including hiding drugs inside a hollowed out Weetabix.
"There have always been pretty tight rules about the receipt of parcels in prisons, under both this Government and the last one," he said.
"There is good reason for this. Our prison staff fight a constant battle to prevent illicit items, such as drugs, extremist materials, mobile phones, SIM cards and pornography getting into our prisons.
"The routes used to try to do so are wide-ranging and ingenious. We see drugs and weapons sown into the lining of shoes, concealed in clothes, and hidden in essential household items.
"We have even seen drugs concealed inside a hollowed out Weetabix."
Mr Grayling also highlighted how difficult it would be to check any literature sent to inmates, saying a prison librarian had pointed out the risk of paedophiles "accessing illegal written pornographic material through print materials coming into prison which are not properly checked for their content".
He said: " I'm afraid that it is inconceivable that we could impose the additional operational burden on our staff of carrying out detailed assessments of an unlimited number of parcels coming into prisons.
"T his is something that has never happened before and could not happen now."
But prisoners had "full access to the same public library service in prisons as every other citizen" and could order books from online retailer Amazon through the prison shop using their earnings or money sent by relatives.
Prisoners were allowed up to 12 books at any one time and Mr Grayling said: " If prisoners are reading a fraction of this total I would be delighted."
He invited Duffy to visit a prison library and view the work of the librarians as " many of them have been dismayed by the suggestion that the good service and professionalism that they provide is inadequate".