May 5, 2022 — The analysis of the millions of artworks that are part of the cultural and artistic heritage is a job that seems impossible for human beings, but not so for supercomputers. The European project Saint George on a Bike, coordinated by the Barcelona Supercomputing Center (BSC) in collaboration with the Europeana Foundation, started in 2019 with the aim of using the MareNostrum 4 supercomputer to train Artificial Intelligence models, which help disseminate among citizens the wealth of European cultural heritage and recognition of its value as well as the understanding and awareness of its conservation and promotion. The project aims to generate automatic descriptions of hundreds of thousands of images from various cultural heritage repositories using natural language processing and deep learning algorithms.

In a second phase of the project, the researchers launched a crowdsourcing campaign in Zooniverse, a citizen science portal based on open peer-to-peer collaboration, to collect thousands of manual annotations that help better train these Artificial Intelligence models. The campaign is completely open and anyone can participate by accessing this link.

“Our project will allow quick access to enriched cultural information, which can serve equally well for cultural and social ends, education, tourism, and possibly for historians or anthropologists. Indirectly the citizens can benefit from better public services, when these are based on the insight that the richer metadata we produce offers – such as web accessibility for the visually impaired or narratives that can expose social injustice or integration and gender issues through cultural heritage corpora and help create a more tolerant European identity” says Maria-Cristina Marinescu, BSC researcher and Saint George on a Bike project coordinator.

To date, no AI system has been built and trained to help in the description of cultural heritage images with the maximum coverage of topics, objects and iconographic relations while factoring in the time-period and scene composition rules for sacred iconography from the 14th to the 18th centuries.

“This ambitious project interprets images according to their context for the first time, and thus seeks to give machines a certain common sense, which is one of the great barriers to Artificial Intelligence today” says Joaquim Moré, BSC researcher and project’s expert in computational linguistics. “For instance, For example, when it first identifies a motorbike in a 15th century painting of St George, it corrects itself and identifies the most plausible object for the period, which is a horse. This adaptation will be also made according to the cultural context. For example, in the Japanese cultural context, what in Europe we would call a knight would be a samurai” he concludes.

The project has also launched an inspiring video (below) that highlights the use of Artificial Intelligence to detect images and compositions never seen before, the extraction of relations between thousands of images or the opportunity to organize virtual exhibitions with related paintings from all over the world.

Further information about the project: https://saintgeorgeonabike.eu.

---

Source: BSC