At present, our Calliope character displays the affective state(s) of a poem via dynamic facial expression. Hence, various visual expressions of unpleasantness, sadness and joy were created and associated to the different states produced by the semantic analysis. Additionally, animations for interim states like ‘waiting’ or for moments when Calliope provides feedback on a selected word add a more life-like countenance.

Once animations are loaded into Frapper, they’re activated according to the emotional tags recognised by the system in the text. The significance of this method is that no pre-recorded animations are needed, allowing any character in The Muses of Poetry to recite any poem with the corresponding emotional expressions being generated in real-time.

Another feature of Frapper is a crying simulation and eye-redness, which we applied for moments when the emotional content of a poem is registered as ‘very sad’. Figure 1(a) shows Calliope with tear simulation while reading a poem. Figure 1(b) shows her waiting state, active until someone talks to her.

Figure 1a. Facial Expression during animation: sadness

Figure 1b. Facial Expression during animation: waiting state

A major highlight of the Agent Framework, which we have used for our character, is the automatic lip-synch with a text being spoken. To achieve this within Frapper, we used seven different phonemes – A, O, E, CH, I, F, M – which are defined as the smallest segmental unit of sound employed to form meaningful contrasts between utterances ¹. These base phonemes are represented in short animations of corresponding mouth movement. Therefore, when the text of a poem is converted to speech using a Text-To-Speech tool, it also outputs information about the corresponding phonemes, which are then linked to the animations in Frapper.

Figure 2 shows the node structure of Frapper and how the nodes corresponding to the phonemes (top right) and the facial animations (bottom right) are interrelated. This structure makes the entire system flexible enough to maintain and improve.

Figure 2. Node structure of The Muses of Poetry in Frapper

Interacting with The Muses of Poetry

Interaction with integrated system, portrayed as Calliope, is through voice recognition. A user speaks three words aloud, to which the system then replies with an emotional poem.

‘The Unwritten’ by Jane Williams

However, given the relatively small pool of poems we have so far, we cannot allow a user to say any word, as it might be the case that a suitable poem based on those requested cues is not found. Yet. Because of this, we decided to show a user a list of selectable words in this prototype version, which are major cue words with a higher frequency of appearance across the current poem library. This set of most common words is displayed within Calliope in the shape of a cloud, as shown in Figure 3.

Figure 3. Word cloud

Poem selection is performed using an algorithm that at first considers all words of all poems existent in the system, displaying the most common words overall in the cloud. When the first word is picked, the algorithm looks for all poems that contain that word, ruling out the poems that do not. It then displays the most frequent words in this subset, shrinking the cloud further and allowing the user to choose a second word. The process is then repeated, narrowing the number of possible words to select from next a second time. At last a poem including all the chosen words is selected and recited.

Regarding the human interaction itself, our objective is to establish a natural communication between a user and an character in The Muses of Poetry, free of conventional devices like microphones or wires. Hence, we have chosen Microsoft’s Kinect software for voice recognition. A user simply stands in front of the system and speaks.

Emotions, poetry … to infinity and beyond

The Muses of Poetry is an on-going project that continues with the support of poets and 3D artists who are contributing to make it as artistic and appealing to a variety of audiences as possible. Our primary goal is to create a bi-directional movement between artists and audience.

‘Garlic’ by Jo Langdon

People who engage with the installation receive an intimate approach to poetry by interacting with a virtual character who expresses the underlying affective content of the poems. In addition, poets from all around the world could have the opportunity to make their work known by providing a poem to be emotionally read in the Muses of Poetry.

What has been described and shown in this article is the result of our first Muse prototype which serves as the basis to further develope from its foundation. The next steps are directed to create a more intelligent semantic analysis of the poems, to design new and more abstract characters that allow a greater engagement with the system, to improve the rhythm and melody in the poetry reciting, and to add artistic and design detail that will improve the countenance in all emotive states and more fluid recitations. New concepts, new poems and new technology (perhaps, too, a bit of old technology, old poems and old concepts) will be our tools and our motivation to continue developing this innovative project.

Acknowledgments

The Muses of Poetry is a project funded by the Innovationsfonds Kunst of the Ministry for Science, Research and Art in Baden-Württemberg. Thanks to all the members of the R&D Department at the Institute: Volker Helzle, Tatiana Pushkareva, Simon Spielmann, as well as to the staff of the Filmakademie who is helping us in the development of the project. Also, many thanks to the poets associated with Cordite who kindly gave us their poems to be part of The Muses of Poetry, and Kent MacCarter for his endless motivation to continue with this work.