Could AI help elderly people and refugees reconstruct unrecorded pasts? | Science and Technology

In 2015, at the height of the refugee crisis in Europe, as a record 1.3 million people, mostly Syrians fleeing civil war, sought asylum, Pau Aleikum Garcia was in Athens, helping those arriving in the Greek capital after a perilous sea journey.

The then 25-year-old Spanish volunteer arranged housing for refugees in abandoned facilities like schools and libraries, and set up community kitchens, language classes and art activities.

“It was kind of a massive cascade of people,” Garcia recalls.

“My own memory of that time is oddly patchy,” he admits. Though there was one encounter that stood out.

In one of those schools in Athens’ Exarcheia neighbourhood, where refugees painted the external wall to illustrate their memories of their journeys, Garcia met a Syrian woman in her late 70s.

“I’m not afraid of being a refugee. I have lived all my life. I’m happy with what I have lived,” he recalls her telling him. “I’m afraid that my grandkids will be refugees for all their life.”

When he tried to reassure her that they would find a place to start anew, she protested: “No, no, I’m worried, because when my grandkids grow [up] and they ask themselves, ‘Where do I come from?’ they won’t be able to answer that question.”

The woman told him how, during the family’s journey to Greece, all but one of their photo albums were lost.

Now, she said, all the memories of their lives in Syria existed only in her and her husband’s minds, unrecorded and unrecoverable for the next generation.

Connecting generations

The woman’s story stayed with Garcia after he returned to Barcelona and his work as cofounder of the design studio, Domestic Data Streamers (DDS).

Over the years, the studio has grown into a 30-person team of experts in varied disciplines such as psychology, architecture, cognitive science, journalism and design. The studio has collaborated with diverse institutions such as museums, prisons and churches, as well as the likes of the United Nations, and uses technology to bring “emotions and humanity” to data visualisation.

Then, in around 2019, with the rise of generative artificial intelligence – a model of machine learning that uses algorithms to create new content from data scraped from the internet – the team began to explore image-generating technology, following the release of ChatGPT.

As they did, Garcia thought of the grandmother from Syria and how this technology might help someone like her by constructing images based on memories.

He believes that memories – captured through records like photographs – play an integral role in connecting generations.

“Memories are the architects of who we are. … It’s a big part of how social identities are built,” he says.

He also likes to cite Montserrat Roig, a Catalan author, who wrote that the biggest act of love is to remember something.

But in the past, people had fewer opportunities to document their lives than their mobile phone-wielding contemporaries, he says. Many experiences have been omitted or erased from collective memory due to lack of access, persecution, censorship or marginalisation.

So with this in mind, in 2022, Garcia and his team launched the Synthetic Memories project to use AI to generate photographic representations of memories that were lost, due to missing photos, for instance, or never recorded in the first place.

“I don’t think there was an eureka moment,” Garcia says of the evolution of the idea. “I’ve always been intrigued by how documentaries reconstruct the past … our goal and approach were more focused on the subjective and personal side, trying to capture the emotional layers of memory.”

For Garcia, the chance to recover such memories is an important act in reclaiming one’s past. “The fact that you have an image that tells this happened to me, this is my memory, and this is shown and other people can see it, is also a way to say to you, ‘Yes, this happened’. It’s a way of saying, of having more dignity about the part of your history that has not been depicted.”

Building memories

To create a synthetic memory, DDS uses open-source image-generating AI systems such as DALL-E 2 and Flux, while the team is developing its own tool.

The process starts with an interviewer asking a subject to recall their earliest memory. They explore various narratives as people recount their life stories before picking the one they think can be best encapsulated in an image.

The interviewer works with a prompter – someone trained in the syntax that the AI uses to create visuals – who inputs specific words to build the image from the details described by the interviewee.

Nearly everything, such as hairstyles, clothing, and furniture, is recreated as accurately as possible. However, figures themselves are usually depicted from behind or, if faces are shown, with a degree of blurriness.

This is intentional. “We want to be very clear that this is a synthetic memory and this is not real photography,” says Garcia. This is partly because they want to ensure their generated images don’t add to the proliferation of fake photos on the internet.

The resulting images – usually two or three from each session, which can last up to an hour – can appear dreamlike and undefined.

“As we know, memory is very, very, very fragile and full of imperfections,” Garcia explains. “That was the other reason why we wanted a model that could be full of imperfections and also a bit fragile, so it’s a good demonstration of how our memory works.”

Garcia’s team found that people who took part in the project said they felt a stronger connection to less detailed images, their suggestive nature allowing for their imagination to fill in the blanks. The higher the resolution, the more someone focuses on the details, losing that emotional connection to the image, Airi Dordas, the project’s lead, explains.

The team first trialled this technology with their grandparents. The experience was moving, Garcia says, and one that grew into medical trials to determine whether synthetic memories can be used as an augmentation tool in reminiscence therapy for dementia sufferers.

From there, the team went on to work with Bolivian and Korean communities in Brazil to tell their stories of migration, before partnering with Barcelona’s city council to document local memories. The sessions were open to the public and held last summer at the Design Museum in Barcelona, generating more than 300 memories.

Some wanted to work through traumatic experiences, like one woman who was abused by a relative who avoided jail and wanted to recreate a memory of him in court to share with her family. Others recalled moments from their childhood, like 105-year-old Pepita, who recreated the day she saw a train for the first time. Couples came to relive shared experiences.

There was always a moment, Ainoa Pubill Unzeta, who carried out interviews in Barcelona, says, “when people actually saw a picture that they would relate to, you could feel it … you can see it”. For some, it was just a smile; others cried. For her, this was confirmation that the image was done well.

One of the first memories Garcia recorded during their pilot sessions was that of Carmen, now in her 90s. She remembers going up to a stranger’s balcony as a child, her mother having paid the owners to let them in, because it looked into the courtyard of the jail where her father, a doctor for the Republican front during the Spanish Civil War, was being held. This was the only way the family could see him from his cell window.

By incredible coincidence, Carmen’s son was employed in the same prison as a social worker decades later, but neither son nor mother knew that. When the whole family came to see an installation at the Public Office of Synthetic Memories last year, her son recognised the prison immediately from his mother’s reconstruction. “It was a kind of closing the loop … it was beautiful,” Garcia says.

Clandestine assemblies

The team was particularly interested in telling stories of civic activists who have played a key role in different social movements in the city over the last 50 years, including those concerning LGBTQ and workers’ rights. While initially the focus was not on the dictatorship era, it “naturally brought us to engage with people who, by the historical circumstances, were activists against the regime,” Dordas explains.

One of them was 74-year-old Jose Carles Vallejo Calderon.

Born in Barcelona in 1950 to Republican parents who faced oppression under General Francisco Franco, Vallejo came of age during one of Europe’s longest dictatorships, which lasted from 1939 to 1975. During the civil war of 1936-39, and following the defeat of the Republican forces by Franco’s Nationalists, enforced disappearances, forced labour, torture and extrajudicial killings claimed the lives of more than 100,000 people.

Vallejo became involved in opposition to the fascist regime first at university, where he attempted to organise a democratic student union, and then as a young worker at Barcelona’s SEAT automobile factory.

He recalls an atmosphere of fear, with most people terrified of speaking out against the authoritarian government. “That fear sprang from the terrible defeat in the Spanish Civil War and from the many deaths that occurred during the war, but also from the harsh repression from the post-war period up to the end of the dictatorship,” he explains.

Informants were everywhere, and the circle of trusted individuals was small. “As you can imagine, this is no way to live – this was living in darkness, silence, fear, and repression,” Vallejo says.

“There were few of us – very few – who dared to move from silence to activism, which involved many risks.”

Vallejo was imprisoned in 1970 for attempting to set up a labour union among SEAT employees, spending half a year in jail, including 20 days being tortured by Barcelona’s secret police. After another arrest in late 1971 and the prosecution demanding 20 years for what were then considered crimes of association, organisation and propaganda, Vallejo crossed the border with France in January 1972. He ultimately gained political asylum in Italy, where he lived in exile before returning to Spain following the first limited amnesty of 1976, which granted pardons to political prisoners after Franco’s death in 1975.

Today, Vallejo dedicates his time to human rights activism. He presides over the Catalan Association of Former Political Prisoners of Francoism, created in the final years of the dictatorship.

He learned about synthetic memories through Iridia, a human rights organisation that collaborated with DDS to help visualise memories of police abuse victims during the regime in a central Barcelona police station.

Vallejo was drawn to the project, curious about how the technology might be applied to capturing resistance activities too dangerous to record during Franco’s rule.

In 1970, SEAT workers organised clandestine breakfasts in the woods of Vallvidrera. On Sunday mornings, disguised as hikers, they would make their way through the dense forests surrounding the Catalan capital to discuss the struggle against the dictatorship.

“I think I must have been to more than 10 or 15 of these forest gatherings,” Vallejo recalls. Other times, they met in churches. No records of these exist.

Vallejo’s synthetic memory of these meetings is in black and white. The image is vague, almost like someone has taken an eraser to it to blur the details. But it is still possible to make out the scene: a crowd of people gathered in a forest. Some sit, others stand beneath a canopy of trees.

Looking at the image, Vallejo says he felt transported to the clandestine assemblies in the Barcelona woods, where as many as 50 or 60 people would gather in a tense atmosphere.

“I found myself truly immersed in the image,” he says.

“It was like entering a kind of time tunnel,” he adds.

Vallejo suffered memory loss around the ordeal of his arrests, imprisonment and torture.

The process of creating the image provided “a feeling – not exactly of relief – but rather of reconciling memory with the past and perhaps also of filling that void created by selective amnesia, which results from complicated, traumatic, and above all, distant experiences”. He found the reconstruction a “valuable experience” that helped him process some of these events.

‘We are not reconstructing the past’

Emphasising that memory is subjective, Garcia says, “One of the things that we are kind of drawing a very big red line about is historical reconstruction.”

This is partly due to the drawbacks of AI, which reinforces cultural and other biases in the data it draws from.

David Leslie, director of ethics and responsible innovation research at the Alan Turing Institute, the United Kingdom centre for data science and AI, cautions that using data that was initially biased against marginalised groups could create revisionist histories or false memories for those communities. Nor can “simply generating something from AI” help to remedy or reclaim historical narratives, he insists.

For DDS, “It is never about the bigger story. We are not reconstructing the past,” Garcia explains.

“When we talk about history, we talk about one truth that somehow we are committed to,” he elaborates. But while synthetic memories can depict a part of the human experience that history books cannot, these memories come from the individual, not necessarily what transpired, he underlines.

The team believes synthetic memories could not only help communities whose memories are at risk but also create dialogue between cultures and generations.

They plan to set up “emergency” memory clinics in places where cultural heritage is in danger of being eroded by natural disasters, such as in southern Brazil, which was last year hit by floods. There are also hopes to make their finished tool freely available to nursing homes.

But Garcia wonders what place the project could have in a future where there is an “over-registration” of everything that happens. “I have 10 images of my father when he was a kid,” he says. “I have over 200 when I was a kid. But my friend, of her daughter, [has] 25,000, and she’s five years old!”

“I think the problem of memory image will be another one, which will be that we are … [overwhelmed] and we cannot find the right image to tell us the story,” he muses.

Yet in the present moment, Vallejo believes the project has a role to play in helping younger generations understand past injustices. Forgetting serves no purpose for activists like himself, he believes, while memory is like “a weapon for the future”.

Instead of trying to numb the past, “I think it is more therapeutic – both collectively and individually – to remember rather than to forget.”