The voice of art

The voice of art

The voice of art

IBM & Pinacoteca Museum


72% of Brazilians have never been to museums nor art institutes because they find them inaccessible. For the launch of Watson in Brazil, IBM collaborated with Ogilvy to create “The Voice of Art” project, an interactive guide that allowed people to have conversations with art pieces housed at the Pinacoteca of São Paulo Museum. Through an exclusive app, Pinacoteca’s visitors could talk to seven pieces of the collection, asking them whatever they wanted, the way they wanted. The response was given by Watson Artificial Intelligence through his Brazilian voice in real time.


Role: User experience manager
Company: Ogilvy & Mather agency
Client: IBM
Project: The Voice of Art

The Challenge

● Design a solution for a broad audience aged from 6 to 70 years old.
● Introduce the app in the context of museum visitation without interfering with the regular flow of visitors.

My role

I led the User Experience Team on designing the app that made possible to talk to all works of art.

We did research, studied the audience, prototyped based on the results of our findings, tested a lot and constantly iterated during the process. We spent 5 months from the beginning of studying to the App launch.

Main tasks

1. Lead the team by organizing steps, UX and methodologies to ensure quality in each aspect.
2. Plan and define product scope.
3. Plan and perform the user research process to understand the needs of the museum’s audience.
4. Lead the process of UX design, contemplating the creation of intuitive product flows, definition of interaction models, user’s task flows, wireframes and interface specifications.
5. Lead the visual design execution prioritizing user experience and IBM brand.

Project Schedule

We‘ve developed an action plan that splitted the work into:
1.User Research
2.Insights and ideas for the app
5.Iteration for final solution

User Research

The biggest challenge of this project was designing a product that could serve a broad audience from 6 to 70 years old, since we’re talking about all visitors of the Pinacoteca museum. To begin with, we did a qualitative research by interviewing 25 people, a considerable amount based on the extensive age range.

The goal was to understand users’ behavior regarding the visiting experience at the museum, by mapping their journey and needs. Furthermore, we’ve studied their behavior while interacting with daily apps so that we could design an experience as familiar as possible.


We had the following main lessons:

Difficulty in understanding art

Users aged 40 to 70 years old feel a great barrier to go to the museum, believing that understanding art is very difficult.
They feel more comfortable when some help is offered such as audio guides. But it needs to be offered at the beginning of the visit, or else they just give up.

For the younger audience, such as children from 6 to 12 years old, the visit is an exploration experience. Visiting the museum is not difficult to them, even without being able to fully understand art. But they do get bored with pictures and they prefer interactive art pieces they can touch or understand its meaning more easily.

Behavior inside the museum

Even with very different age groups, a common preference among adults are the indications on where to start the visit, making them feel a little less lost.
Children’s visits followed the order of the rooms, even without a logical order of the exhibition.

Use of the app

In general, the most popular apps in Brazil are for chatting, social networking, location guides, bank services, streaming and gaming.
Only the most distant age groups, such as children from 6 to 12 years old and elderly from 50 to 70 years old, have a big difference between them.

Data analysis and metrics

The museum splits its audience into 2 types: Educative Audience and Total Audience. Educative Audience refers to students and Total Audience is the common visitor.

In 2016, one year before the project, the numbers were:

Educative Audience: +42K people
Total Audience: +325K people

In 2017, the expected audience was 415K people

The weekends are the most visited days of the museum, averaging 2,000 visitors on Saturdays and 1,000 visitors on Sunday. On weekdays, the average drops to 700 people, increasing to 1,000 only during holidays. Visitation rates vary between 11 AM and 4 PM (data based on Total Audience).

Visitation rates of Educative Audience occurs during the week (without counting vacation time), varying between 10 AM to 12 PM in the morning and 2 PM to 4 PM in the afternoon.

Source: Pinacoteca Museum.

This data mainly helped us plan the days and hours for prototype tests regarding a crowd number close to reality.

Design Sprint

We’ve mapped the following key insights by the end of research:

At glance

The solution should be offered to public at the entrance of the museum, specifying the purpose to help understanding the art pieces in a simple way and free of charge.


Location guidelines are welcome within the museum and they need to be didactic as it is common that the audience has never been there.


Make clear to the audience that they could ask art pieces anything in any way, just like a common conversation.


Make clear they could talk to the art pieces alone, without the help of anyone.

No distraction

The audience is used to apps with specific tasks and few distractions on the interface because they are easier to use, like WhatsApp, Youtube, Skype and Netflix.

The solution

The strategy was putting all Watson’s cognitive power into a mobile app available to visitors of the Pinacoteca Museum. This application uses several Watson APIs to:

  1. Recognize visitors’ questions
  2. Decode the intent of those questions
  3. Deliver specific answers

In addition, we’ve incorporated a system at the museum building that recognizes the visitors’ presence in front of the art pieces to begin the interaction in the app.

It’s a new tool that allows visitors to have a completely new experience by turning the museum into a place of technological experimentation.


This phase was divided into two fronts.

1 – Design of app user experience: the experience while interacting with the art pieces should be intuitive and easy to talk to and to explore.
2 – Rethink the experience in the physical location: mapping of messages and instructions so the audience could find the art pieces and be guided by the museum.

App user experience

Considering all learnings in the research process, we developed the wireframe with 4 main intentions:

1. Create a great onboarding process
2. Locate and recognize the art piece in the museum
3. Facilitate interaction with the art piece
4. Evaluate Watson’s answers to improve its cognition

A great onboarding was crucial since our public needed a didactic experience, so we addressed the app’s functions in a quick and easy way.

Art Piece Location

The first wireframe was based on the user location inside the museum so users could find the art pieces and start the conversation.

Art Piece Recognition

When approaching the art pieces, a screen opened up to quickly recognize the artwork. We also designed it primarily on portrait mode because it was more comfortable to visualize the artworks on screen.

Questions Recognition

The main goal was to encourage the user to talk to the art piece, so we’ve highlighted the question button right on the interface. The main screen also had easy access to map and help section.

Mapping messages

We’ve made a map of messages that guided the audience inside the museum. These messages were sent to the device by proximity through beacons. But, if users passed through the same beacon again, the instructions were not repeated a second time.

We’ve classified the beacons into 3 functions:

Location indicator beacons: These were activated within 20 meters of a room containing an interactive art piece. Users listened to (or read) the instructions on where they were and which art pieces were nearby.

Art pieces beacons: These beacons were activated within 5 meters from the artwork. An introductory text was launched inviting users to talk to it.

Gateway and exit beacons: These beacons were activated 2 meters away from the sensor and thanked for the visit.


We’ve tested and iterated the app each week for 2 months. The message map and the quality of Watson responses were also tested (this helped improving the understanding of artificial intelligence).

The main lessons we learned regarding the app usage were:

People struggle to interpret maps and blueprints, because they are much more visual. Telling them the floor number, the room name and showing a picture of the art piece was enough to locate themselves.

Followed by:

• In Brazil, people hardly go alone to museums. They like to talk to someone during visitation to share their perceptions, so the app shouldn’t become a barrier between them.
• They prefer not to follow scripts. They like to tour as they please.
• They prefer to choose when to start the conversation and when to stop it at any time.

The main lessons we learned regarding the message map launched by beacons:

• Visitors don’t follow ordered directions and as they may be coming from different sides, informations like “turn right, turn left” are very confusing in this context.
• People like to be reminded when they are already in the room that there are cognitive art pieces there. They chose to enter this room and the information comes in form of a “tip” without disturbing their visitation.
• But they don’t need to be reminded every time they enter a room that there are cognitive works. They understood the message on the first time.
• It’s positive when the work presents itself to them at first, but to keep repeating the same information is annoying.

Final Solution

The main change was that the app wouldn’t be based on a map any longer. So by taking off the museum map, the app became much easier to use. We also continued to prioritize one action at a time (onboarding, locating the artwork, talking with the artwork, evaluate the answer).


In the final solution, we’ve kept onboarding with the same simplicity of the beginning.

Finding and recognizing the art piece in the museum

We changed the app in order to work accordingly with the floor the user were at. For instance, if the person was on the first floor, the app showed which galleries were there and which cognitive art pieces they would find. The user could also swipe the screen to see all galleries on that floor.

Talking to the art piece

On the question screen, the “question button” gained more relevance. We removed other buttons since tests’ results proved it to be unnecessary.
We added a tab so users could access all art pieces whenever they wanted. If they weren’t close enough to the art piece, the button was disabled.

Evaluating Watson's responses

By the end of each question, we added a questionnaire directing Watson’s training team to improve quality on responses. This team was composed by IBM members and museum curators.




increase on the number of Pinacoteca’s visitors weekly. Due to its huge success, the project was extended for two extra months.

6+ awards

Including mobile and cyber interaction design awards.

4 ★

The app was highly rated by museum users getting 4 stars on a scale from 1 to 5.

$3.3 million in earned media value

More than 50 local and international PR stories, including a 4 minute segment on Brazil’s leading TV news program; highest organic reach (+1M) and engagement rate (37% – our average is 3%) in the history of IBM Brazil’s Facebook page; +6M views on YouTube and Facebook, being +280K organic views.

But more than that, it has already delivered business results: 4 new client opportunities for IBM including another big museum, proving that the results were positive in different spheres.


Mobile Category Bronze Winner

Cyber Category Silver Winner

Gold Winner

London International Awards Bronze Winner

Global Association for Creative Advertising & Design Awards – Graphite Pencil

User Experience Design Team: Fernando Souza, Flávia Goulart, Cadu Pereira
Creative Director: Marcio Fritzen, Guiga Giacomo
Copywriter & Art Director: Leandro Neves & Frederico Gasparian
Account Supervisor: Chad Cathers

Next cases


Itaú Unibanco

Evolution and operation of the bank's main application


Volkswagen New Up

Pitch project for the launch of Volkswagen's new car in Brazil