An augmented reality learning game

HCI Masters Project @ SJSU



In grad school, Zheng Cheung and I were assigned to design an app that loosely integrated the idea of a “pet.” Although the “pet” constraint focused us in a general direction, there was a lot of freedom. We could have designed anything:

Next-gen Tamagotchi

Connected saltwater fish tank

A virtual cat assistant named Kitty Purry

…but that’s not where our research led us.

Finding Our Way

Zheng and I wanted to focus on mobile and design something fun for kids, which was tricky, because we didn’t know how kids used mobile devices or what apps they liked. Fortunately, there’s a lot written on the topic. We also interviewed over a dozen families to learn what parents thought about their kids using mobile apps and how they monitored them.

Play outside or read a book

Parents favor mobile apps to watching TV, because games require their kids to solve problems and some teach things interactively. However, nothing beats a good book or being a kid outside.

Let’s do it together

Parents are excited to play with their kids, just not on a mobile device. Some devices are too small and most apps aren’t designed for local co-op.

It’s my turn

Kids often interact with the devices alone.  When they do play “together,” it’s one child watching the other or one child sharing a moment of success.

Don’t underestimate me

Younger children learn how to navigate apps like we learn to navigate a new city – by landmarks. The same demographic also doesn’t know when they make a mistake or over look a feature.

Go Go Go

Older children read better and pick things up quicker, though most don’t let silly things like instructions slow them down.

Easily distracted

In general, a child’s attention is divided across multiple tasks. Younger children toggle between tasks more frequently out of fear their going to miss out.


We gathered a lot of insights with our research and we couldn’t wait any longer. This was the week we had to pick a product direction. I was hoping for an “Ah-ha!” moment to settle my anxiety, and I wasn’t expecting one at Trader Joe’s. Yet there I was, knee deep in dried mango samples, holding a paper shot of coffee and trying to avoid children with my cart. There were kids going everywhere looking for something on the shelves. It wasn’t food. It was a frog, and when they found it a store clerk rewarded them with mini ice cream cones.

“What if kids used a tablet to follow a series of virtual clues that lead to the frog?”

At the time, Yelp’s monocle was only the one legitimate example of augmented reality. Every other app in the App Store made it clear that the hardware and/or technology wasn’t game day ready.  Fortunately, we were only tasked to design a concept, not build a POC. And we were excited to explore this concept from the perspective of our users.

Designing with (their) perspective

Our research results proved there were a lot to consider with regards to what to build and how to build it. Kids are wild and they have parents with preferences and influence. If the product wasn’t something kids desired and could use, then we’d have to go back to the drawing board and test another solution. If the parents didn’t like it, then we knew whatever we made wouldn’t make it past their App Store passwords. To put our best efforts forward we developed personas based on the 12 families we interviewed, and we tried several design activities centered around those personas to bring empathy to our process.


There were limitations to our persona research. For one, we didn’t have the resources or time to conduct a high volume survey with a qualifying sample. Also, we weren’t experienced enough to conduct in-depth ethnographic research. Consequently, we didn’t know how the market was segmented or the exact characteristics of those segments. However, we did interview 12 families and did our best to match the personalities we met with proven profilers, like Myers Briggs, and focused on four personas.

“Look, I’m a mom. Of course I don’t want my kids to be iPad junkies. But they do all kids of things. They go outside a lot – which makes them tired and helps me.”

“I didn’t think I’d be like this, but I only give my kids organic food and I read to my daughters every night.”

“Mom says my shoes don’t match, but I think clear color matches everything… I like doing stories the most!”

“I’m going to play sports like skateboarding or baseball when I grow up… Reading makes me sweat, and I don’t like it so much.”


Stephanie has 3 children: Aden (12), Jacob (8), Claire (6).  The kids have school, then after school care where they complete homework and play a little, they are home by 5:30 when Steph makes dinner.  The kids have the evenings pretty free.

Sebastian’s family has been growing “like crazy.” He has two daughters (5 and 1). Although he is very busy with work, at home he spends a lot of time with daughters doing things they like to do. Recently, he’s introduced the girls to gardening and cooking the food they grow.

Elizabeth (7) is in the 2nd grade at a public school. Her mom works part time and picks her up from school everyday. Elizabeth has a younger sister, Amber (4). Amber likes to watch Elizabeth play angry birds and subway surfers on her iPod touch. Mom or Dad always helps her with her homework and sometimes reads her a story.

Logan (6) is in the 1st grade, has soccer practice after school on some days and plays with kids in the neighborhood when he doesn’t. Logan has one sibling – an older brother (9) who he looks up to and competes with. Logan’s Dad usually does physical activities with Logan like wrestle or play baseball. His Mother occasionally reads him stories before bedtime.



Pretty savvy with computers, but her husband is the “know it all.”


Extensive experience with computers and internet, and likes new gadgets.


Elizabeth has been playing games on the PC since she was 5 and now she has a 4th generation iPod touch.


Logan plays flash games on the computer. He has a Wii and Nintendo 3DS, and he plays with  his friend’s Dad’s extra tablet.


• Limits her kid’s iPad usage.

• The sound of Angry Birds drives her crazy.

• Kids download whatever they want.

• Daughter wants to do more with her than the boys.

• Enjoys her downtime when kids are occupied.

• Doesn’t know of any kids educational apps.

• Would like more time to spend with the family.

• Prefers his kids to play games with more educational value.

• Has picked every game his daughter plays.

• Doesn’t consider iPad usage as “quality time.”

• Will share her iPod with her sister (when prompted), but prefers she watch her play.

• Mom says she can’t play on her iPod, watch TV, or go on the computer all the time.

• Favorite game involves animation programming.

• Loves all her stuffed animals – she’s a collector.

• Her and her friends at school can play with electronics during “computer time.”

• He’ll sit for a while if what he’s doing is entertaining.

• He wants to do things better than his brother.

• He hates reading, but enjoys math.

• Most things he does: chores, cleaning, and after school work all require him to have an incentive.

• Wants to make time for his friends.

The big vision

We learned that parents want their kids doing meaningful activities where they can creatively, socially and physically engage. And we learned kids want the same thing. Our vision was to leverage the iPad, its camera and sensors to transform common spaces, like a living room, into something mysterious and exciting that demands exploration.

Story mode places kids at the center of a narrated adventure, which sets a scene in classic story book fashion then prompts them to explore their surroundings to continue the journey. Along the way, mini games and riddles must be solved to “turn the page.”

Quest mode is featured for instances when kids can’t move around, like a car ride. It’s a side scrolling gaming experience that allows kids to travel through a digital world, solving problems and completing objectives. 

Friends is a kid safe MMO set in a world parallel to Quest mode, where kids can socialize and compete with one another.

In the store parents and kids can find new stories, characters, games and other items.

Narrowing our focus

Our vision for the product was large and we knew we couldn’t develop the entire concept before our deadline, so we put all our efforts into Story mode and started asking a lot of questions.

To get the answers we needed, we began hypothesizing an ideal experience through interaction flows and wireframes. As our ideas became increasingly concrete, we tried several design activities to better understand our persona’s needs and desires and bring empathy to our process.

Interaction flow

Before we committed anything to Photoshop, we mapped out the story interaction flow on a whiteboard. In addition to diamond decisions and action boxes, we loosely sketched each screen as we debated page elements and micro interactions.

In effort to get kids invested in the story, we wanted to lead them through a few fun activities to customize the characters and make the story their own. We settled on 4 onboarding steps:

1. Create an avatar as the main character

2. Customize a side kick

3. Choose the story for the characters to appear in

4. Warm up with basic stretches

Wire framing

To clarify features and determine if page elements were a good fit, we quickly wire framed all the screens in the interaction flow. Wireframes forced us to look objectively at the app’s ease of use and consider content first. Additionally, the wires proved we needed to apply skeuomorphic styling to invoke more emotion from our users.

Empathy & Scenario maps

In effort to establish a deep, personal empathy with our users we leveraged empathy mapping to consider what our personas thought, felt, said and did as they used our wireframe prototype. Additionally, scenario mapping helped us think about the kids, their tasks and the sort of user experience we wanted to provide.

Key considerations


Story setup should feel like it’s part of the game. Kids won’t see it as a barrier if it’s fun.

Quick challenges

Challenges should progress the story, not distract from it. Keep them short.

Appropriate content

Reading level should be considered when the story is being told (bigger words for bigger kids).

Dynamic difficulty

Detect if challenges are too hard or too easy – if kids fail a lot they’ll get frustrated and the story won’t progress.


Kids of all ages enjoy being read to. Kids can’t walk and read as they are exploring spaces.


When kids do well, tell them and encourage them to achieve more.

My Story Playtime

Adventure is all around you.

Make your own story

Kids can play as the main character in every adventure, and create an unlimited number of imaginary friends to do just about anything together.

Start the journey

When the stage is set, the story begins. Kids are introduced to their role and the characteristics of their imaginary friend are woven into the plot.

Experience it in real life

Transform any room in the house, collect clues and solve mysteries. Augmented reality enables the story to play out wherever you are, and makes imaginary friends real.

Learn with games

Mini-games help kids complete stories. Kids learn mathematics, music, art, reading comprehension and more. Difficulty is determined by a child’s profile age and will change according to skill level.