Intendo
public sentiment captured and visualised through AI, speech recognition, and physical computing
About this project:
Context:
SVA / Fall 2016
Collaborators:
Type:
- Product Design
- Interaction Design
- Physical Computing
The Black Box
Intendo is a device that leverages artificial intelligence to interpret human language in terms of emotions.
Working with my classmates Janel Wong and Tyler Gumb, I designed, prototyped, and wrote the code for Intendo.
The project's goals are two-fold:
- Educate people about how AI works. In a world increasingly reliant on AI, knowing that what we said can be interpreted not only for what it means but also for how the AI thinks we are feeling is essential to remain in control (or at least in awareness) of what you are communicating to AI.
- Measure the general emotional landscape of a group of people. You interact with Intendo by speaking to it, and it responds with an emotional analysis of what you said. The dominant emotions are represented by 5 thermal printers, one for each emotion. The emotion interpreted the most often by the AI will be visible by the physical length of the paper hanging from the corresponding thermal printer.
Inspiration
Our initial concept was of a device the user wishes into and receives a unique token based on your wish. We imagined use in a public space. We wanted the piece to be whimsical and artistic.
We experimented with the effects of heat on thermal paper using cigarette butts and soldering irons, thinking we could use a heat pad in the device to produce a unique pattern for each users receipt. The intensity of heat required to achieve this was quite high so we decided to explore other possibilities for creating this unique token.
Initial speech recognition in Processing
We initally got speech recognition working in Processing using this method where the speech processing was done by a google API in chrome and sent to Processing via Websockets.
For an initial proof of concept and to get a feel for the interaction for whatever wish the user said the printer responded by randomly printing from an array of responses we prepared.
One key component of the interaction we spent a lot of time considering was indicating to a passerby how to interact with the device. In this first iteration we put an IR sensor and coded the printer to print a phrase asking for wishes whenever someone stepped in front of the device. We thought there was room here for some magic and to give the impression of the device being alive and having a personality.
Iteration
The use of heat within the device to effect the thermal paper of the receipt printer started to seem less likely to be able to accomplish. There were many concerns with how the paper left the printer and lining it up exactly to be subjected to a high enough level of heat for long enough and in a way that produced unique results each time seemed a little too much for our time frame, so we started considering alternatives. One consideration for a more practical application of the device was as a means to collect public sentiment data on something, so a device to encourage customers or visitors to provide feedback. We considered the possibility of collecting the responses in a database for analytics to be done later.
We were still debating what token—if any—the user should receive after interacting with the device. One thought was they receive an output that was instructive on how your speech was interpreted and classified by the computer. The idea being to encourage people to interact with the device to learn about sentiment analysis and receive a token on how their speech was analyzed.
Enter Watson
Working off this concept and the idea that the responses would be grouped by their overall sentiment we came up with a new form for the device. We thought to replace the IR sensor initiated printing phrases asking for wishes with a LED that changes color and fades in and out to entice the user to press the button. When the user presses the button to initiate the recording and speaks when they release the button the recording is sent to IBM's Watson Sentiment Analysis API which will categorize the statement. The device then prints your statement on one of five sheets of cascading paper corresponding to the sentiment that it is grouped by.
The appeal of this form for us was that the five cascading sheets serve as a sort of physical bar chart immediately visible when a visitor comes up to the device of the overall sentiment of other users.
Josh Zheng's Medium posts (AlchemyLanguage Sentiment Analysis in Python and How To Build a Candy Machine With Feelings) were extremely helpful for us as we started to explore and setup the code for our device to get the text from the speech and then get the sentiment from that text.
We set up the five printers and tested coding the python to route the phrase to one printer at a time.
Once that was working reliably, we built two wooden enclosures: one for the input containing the button and microphone, which also housed the printer which delivered the token for the user; and one for the output of the 5 printers corresponding to the 5 emotions.
Installation
We tidied up the wiring and mounted Intendo in a public space (the lobby of our department) for testing.
