The UK’s female prison population has doubled in 20 years. Justice minister Simon Hughes says half of them should not be there. But what is the alternative?

“When I was taken down, they said, ‘Look, when you get to prison, you have to put your head down and walk forward.’ So that’s what I did. I was having a shock reaction. A lot of it I can’t remember, even though it was only three-and-a-half months ago.” Ayishah is demure, intensely distressed, in her early 50s and beginning a three-and-a-half-year sentence for money laundering at New Hall women’s prison in Wakefield. Previously, she was a GP.

We are talking in one of the squat, single-room buildings that sit opposite one residential wing of New Hall. Ayishah is one of the people supported by an organisation called Muslim Women in Prison. Nicola, 33, is nearing the end of a three-and-a-half-year sentence for robbery and is part of the Together Women Project (TWP). She was in prison once before, but isn’t a classic recidivist – there was a decade between her two sentences. “The last time I was in prison,” she says, “none of these service were available. You were in jail, you were sent out; if you had nowhere to go, that was your problem.”

But they are, and many who serve one sentence will go on to serve another. As everyone agrees, the prevention of reoffending is the key aim of the penal system. From that starting point, everything else will flow: all successful rehabilitation, all prevention of crime and all avoidance of creating more victims will come from the prison estate dealing imaginatively with what happens after inmates are discharged. The big untold story in all of this is how much is down to individuals: prison officers, psychologists and people from the third sector working inside and outside the prison walls. These people are often working way beyond their job descriptions, not to targets or directives, not to improve a measured outcome, but because that’s how they are.

As a declaration of interest, I’m a trustee of the Butler Trust, which was established 30 years ago to recognise the achievements of prison service staff. The thinking – from a prison governor, the reverend Peter Timms – was that it was rare for people who worked in prisons to have their excellence noticed. There are public-sector spheres where people just assume you’re doing a good job – the army – and places where the public will regularly witness you doing a good job – a hospital, for instance. No news but bad news ever comes out of a prison, which, apart from making it hard to build morale, is somewhat unfair. In all that has changed about prisons over the past three decades, this has remained constant; you might exceed your duty, perhaps by miles, but it’s not likely that the wider community will get to hear about it.

Joanna Smith, 30 – who has received a Butler Trust commendation – has been based in New Hall since 2009, although she works for TWP (she’s commissioned rather than employed by the prison, but her team includes many uniformed prison staff). She testifies to the peculiar challenges of a women’s prison. As she puts it, “men can go into custody and they can come back out, and their wife is still at home and their children are still at home. You send a woman into custody and it’s chaos. The children get removed, the property is lost … so TWP was set up originally to find alternatives to custody. But, inevitably, you’re always going to have a female prison population. It would have been stupid of us to ignore those women.”

Instead, she’s brought the services in for people like Nicola, who says: “My family says I’m growing up in prison. My dad always says, when he comes to visit, ‘You’re a young lady now. You’re exactly where I want you to be.’” It’s a coproduction – Nicola wouldn’t change if she wasn’t trying to change; but when she describes the support she’s had – “from someone to talk to if you’re feeling a bit low”, to getting on to a drug rehab programme, to getting a number of NVQs (two in catering, one as a barista), to getting access to the people who’ll support her on the outside, to getting a bank account – it is plain there is an army of people willing her to succeed.

But the key innovation is really subtle: Smith noticed that, once women were released, they weren’t keeping their appointments with the people who had been lined up to help them. “I used to sit here and fill forms in and say, ‘Go and see this person, they’ll take you here, they’ll get you this.’ And women weren’t turning up. It’s intimidating. It’s a new building, it’s a new person, they’re not going to go. A lot of them have come from really quite abusive backgrounds, and they’re quite chaotic in the community. Trusting people, particularly when they’ve got children – trusting that the support they get won’t go against them – is really hard. But if they build up a rapport before they leave, they will go.”

It’s tiny, but it’s massive; they call these “through-the-gate” services, and there are people everywhere, in the civil service, in academia, in probation, trying to improve the through-the-gate experience for prisoners. But some things can only be done by the people who are close enough, and who care enough to notice.

Other problems may be easier to spot, but a lot less easy to rectify. Sofia Buncy from Muslim Women in Prison, talking at 1,000 words a minute, describes the situation: “I’ve seen with the male prisoners, they’ve got their mums going in, children going in, wives going in. The women I work with have no visitors, they’re not going back to the communities they’re from. Muslim women carry the reputation of the full family, especially of the head of the house, which is your father or your brother, so you’ve dishonoured your father’s name, your brother’s name. The rest of the community hold you to account – if you were a decent father, why would you talk to her again?”

Many of the women don’t speak English, she says. “They don’t even understand how the prison regime works. It’s quite strict – there are lockup times, lockdown times, times for education.” There are often immigration issues after a criminal conviction. “I’ve got a case at the moment, a lady who has been issued her deportation letter … [she’s] the one you visited, Ayishah. She got her papers about two days later: explain why you would like to stay in this country, because you’re a criminal now and you’re no longer a citizen.”

It’s such a shocking detail – she has spent her professional life in this country, and she has a British daughter who is 16 years old. Her English is obviously fine, but, without help, she has no money and would be in no position to take this on. “I’ve had to go back out to the community and seek help from an immigration contact: ‘How much is this going to cost, can you do it as a favour?’” Sofia explained.

It’s hard to think yourself into the minds of the people who hurl themselves at these challenges; everything, from a distance, looks so intractable, so insurmountable. And it would be so much easier to write off the inmates as people who had made their own beds and are now lying in them. “We have to be here in a non-judgmental capacity to work with them,” Joanna says. “It’s about saving future victims, that’s what you’ve got to remember. There are victims of the crimes they’ve already committed, but we’re doing our bit to prevent further crimes.”

The past 30 years have seen seismic changes in the prison service. Simon Shepherd, director of the Butler Trust, who trained and originally practised as a forensic psychologist, describes the way it was when he arrived in 1989. “Wewere at the tail end of what was called humane containment, which comes from the 70s idea that nothing works.” That’s what the criminological research was actually called – “nothing works”. It was coined by an academic, Robert Martinson, in response to the therapeutic ideas of the 60s and early 70s, which said that care and empathy, patiently expressed, will solve anything. After some research – not all of it as methodologically sound as it thought it was at the time – this was found not to be the case.

“That 70s approach actually works reasonably well for people with mild anxiety,” Shepherd notes, “but you’re not going to stop Harold Shipman killing people with an approach like that.” Into the “nothing works” policy environment gradually filtered new thinking, a lot of it from Canada, centred on cognitive behavioural therapy, anger management, thinking skills – programmes delivered by psychologists but gradually drawing in the disciplinary staff as well. “In the 80s,” Shepherd says, “the job was locks, bolts and bars. When the offending behaviour programmes started to be introduced, and uniformed staff started to get involved as facilitators, many of them found the work really interesting and found their jobs as a whole more rounded and rewarding.”

It is only now, 20 years later, that prison suicides have started to go up again. Kathy Biggar, who was part of the Samaritans then, has always been intricately involved with the prison service. Telling me about the sea change in Wandsworth prison some time ago, she said: “There will be prisoners who want to kill themselves, who are just waiting for their mothers to die and thenthey’ll do it. There are people who have done some terrible things. But then there are inmates who kill themselves on their first night, or in the middle of a crisis. And none of us could ever accept those. We were thinking about what to do constantly.”

In a way, the knock-on effect of that was that prison regimes started to listen to prisoners, to the extent that now you’ll find prisoners elected by their peers helping make decisions. “It’s not ‘The lunatics are taking over the asylum.’ It’s just a recognition that they’ve got useful things to contribute,” Shepherd observes.

Even seeing nationwide effects, one still notices individuals more than anything – individual members of staff and governors. No two are the same. Diane Pellew, 46, is the governor of New Hall, and mentions two things that are almost palpable in her prison. The first is: “It doesn’t matter who is employed by whom, if they come through that prison gate they are my responsibility, and I treat them as part of my team.”

Almost all the innovations in New Hall – there’s a behaviour-management wing as well, working with an NHS team – would have been impossible without that attitude. It’s a puzzling cross between the matriarch and the MBA (Pellew does have an MBA, in fact; she is the first person I’ve met who doesn’t just use it to slow down other people’s meetings). Secondly, she says, “I treat everybody exactly the same. Doesn’t matter who I’m speaking to – one of the women, a member of staff – it’s exactly the same.” An institution that is, by definition, the most authoritarian you will ever walk into can feel like the least hierarchical.

A lot of prisons are quite beautiful, in an angry, Victorian sort of way. Not this one, with its hodgepodge of buildings, randomly thrown up at the end of a long lane. True, it was snowing last week, which makes everything look different – quieter, more isolated. But it is definitively outside society, invisible from the road, invisible to the world. It is a remarkable thing, what the people within it have built: a community that is at times very difficult, but also rich and warm.

Some names have been changed