The character Calliope awaits instruction …

The Muses of Poetry is one of the current projects at the Research and Development Department of the Institute of Animation at Filmakademie Baden-Württemberg in Germany, that intends to bring poetry – its emotionality, auditory structures and nuances when words meet elocution – to a larger audience. This is achieved through an interactive installation that makes use of character animation, voice synthesis and affective semantic analysis of poems, all in real-time.

The project, funded by the Ministry for Science, Research and Art in Baden-Württemberg, started in November 2012 with the goal of finding new ways to experience poetry. As a result, a first prototype has been created with a realistic female character named Calliope, a poetess in The Muses of Poetry. For this project, visual and tonal representations of gender are important and must be considered in programing. Cordite Poetry Review has provided some guidance in the reading of poetry and, more importantly, a first batch of poems for Calliope to engage with. Further collaboration is planned in 2013.

The interaction begins when the virtual character invites a user to select three words from a word cloud, allowing her to decide the poem to be recited. Underpinning The Muses of Poetry is an extensive sequence in intelligent semantic analysis, novel Human-Computer Interaction (HCI) techniques, real-time automatic lip-sync and facial animation – developed and implemented at the Research and Development Department of the Institute of Animation.

Before The Muses of Poetry, our R&D team had been involved in several projects regarding conversational virtual characters. One of the most notable is the Agent Framework, which is a development platform for character-centric application prototypes. It has been designed as a set of modules that allow the fast and easy creation of realistic virtual characters. Among the projects we find a short film featuring Klaus Kinski as a 70 year old man, for which a completely digitised, aged version of the deceased German actor’s head and face were created; Emote, a system for creating animated messages for mobile devices; and Nikita, an installation with a female character that replies to questions according to a script, becoming visibly sad when insulted and happy when complimented. Behind all these applications is the Facial Animation Toolset – a plugin for Autodesk Maya – that facilitates the creation of facial expressions in different characters. The satisfactory results of these works inspired Volker Helzle, head of R&D, to develop a ‘poem reader’ that interacts with an audience, reciting and expressing the affective content of poems through facial expressions and finely-tuned audio controls.

‘For the Road’ by Carol Jenkins

Why poetry?

During the last three decades, interaction with computer-generated content has become a more natural and common activity. Human-computer interaction (HCI) has undergone a number of major improvements – from virtual characters on display in kiosks at museums and airports, to state-of-the-art augmented reality installations where a user embodies an avatar and engages with environments through its eyes.

But full interaction within a literary environment is quite new and one scenario where such interaction can provide an enhanced experience occurs within the topography and lexicography of poetry. In a poetry reading, for instance, the poet stands in front of an audience, reading his or her poem without interruption. But what would happen if, before the reader began, the audience suggest words for the poet to then extemporaneously settle upon a poem that best fits the cues? And for immediate recitation at that? This is the precise scenario we intend to recreate in Muses of Poetry. Achieving this goal, however, poses a number of questions: How to assess a poem’s emotional content? How to restrict a selection of words on the user’s side? How to organically interact with the character? Where to obtain the poems? I’ll address all of these questions and explain how we intend to achieve our objectives.

A bit of Greek mythology, a lot of semantic analysis

The Muses of Poetry is namesake to the muses of Olympus who inspired poets of the ancient Greek era. Therefore, our first character has been named Calliope – the ‘beautiful-voiced’ – who was the muse of epic poetry and wisest of all Olympus muses.

With this as inspiration, we faced our first big challenge: exactly how to analyse poems in order to obtain its affective content. A tool complicit in this task is the Dictionary of Affect in Language (DAL), created by the psychologist Cynthia Whissell ¹, which is based on results of users’ evaluation of words in a variety of media. The version of DAL we are using includes 10,368 English words with affective connotations, where each one is described with regard to the dimensions of Activation (or Arousal) and Evaluation (or Pleasantness).

‘Genie’ by Kate Lilley

DAL operates as a licensed stand-alone application, and allowed us to introduce, then scrutinise the text of a poem to obtain relevant affective information of that poem. This information gives us a general assessment of the poem in terms of activation, evaluation and imaginary dimensions. It also provides a detailed classification of each major cue word in the poem according to the following states: pleasant, nice, fun, passive, sad, unpleasant, nasty, active, high imagery and low imagery. In the current alpha version of The Muses of Poetry, we only consider the pleasant, nice, fun, sad, unpleasant and nasty states. Beta versions and beyond will engage with an ever-broader assessment of emotional states.

Using DAL’s output, we developed an algorithm for the elicitation of the emotional expressions in the character. This prompted two analyses. First, we assessed if the whole poem was emotionally positive or negative. Then, equipped with this information, we evaluated each line of the poem by considering its individual words. For instance, if a line contained more unpleasant words and the poem was assessed as negative, then that line was tagged as ‘unpleasant’. This procedure was performed offline for all the available poems we had (most from Cordite). Thus, when loaded to The Muses of Poetry system, it would know which emotional facial expressions to trigger according to pre-tagged text.

As a starting point, we realise the process noted above is a literal reading of words and their definitions. We acknowledge that idiom, metaphor, simile and double entendre are tenets of poetry and aim to accommodate those tools in future versions.

Frapper: facial expressions and lip-sync

At the Institute of Animation, we have developed a custom framework that allows us to implement different applications that range from conversational characters to stereoscopics. Frapper, short for ‘Filmakademie Application Framework’, is a node-based tool that enables developers to focus on the core functionality of their application and research, while providing the basic foundation for operation.

One of the main advantages of Frapper is its modular architecture that allows any developer to plug in their app nodes. This first prototype of The Muses of Poetry has been developed using Frapper and features real-time animation rendering. But before their final display, these animations required creation in a program like Autodesk Maya or 3ds Max. Once complete, the animations were exported to a Frapper-compatible format. This was done through the Open Source 3D Graphics Engine.