Jia, Language Learning through AI-Generated Chinese Stories

Overview
As a project for graduate school, we were tasked with building a start-up from scratch. Our team developed Jia, a web-based platform aimed at enhancing Chinese language acquisition for 2nd generation children aged 3 to 6. Jia’s goal is to improve exposure to the language at home by fostering frequent conversation between the parent/s and child/ren.

  • As a researcher, I conducted desk research and semi-structured interviews to understand the main problems within the space.

  • As the lead designer, I strategized which features to include, designed wireframes, mid-fidelity prototypes, and oversaw the development of the platform. I also created the overall visual branding.

  • As an entrepreneur, I created the demo and pitch deck which I and two others presented to 15 venture capitalist firms to secure funding

What was my role?

Collaborators
Software Developers: Wen Qui, Sylvia Wang, Keith Koon & Mauricio Guerrero
Business: Marcel Gremaud & Andre Stone, Nikhil Goyal
Design: Margaret Hsiao

Duration
4 weeks

PROBLEM SPACE

Background of the Problem

Studies show that although proficiency in their heritage language (HL) is vital to the well-being of Chinese American families and child development, it is declining in 2nd generation Chinese children. The main barriers to the child’s heritage language proficiency are:

  1. Undervaluing their heritage language

  2. Limited exposure to both explicit (e.g. traditional Chinese instruction and conversation at home) and implicit forms of socialization (e.g. reading material, media, participation in ethnic communities, and relationships with peers)

  3. The difficulty level of the Chinese language

View Research Report Here

RESEARCH

Understanding Why Exposure to the Chinese Language is Limited

I conducted surveys and semi-structured user interviews with (8) parents who migrated to the United States and (2) 2nd generation Chinese Americans to understand the barriers in Chinese language learning and teaching in the United States. I asked them to tell us about the challenges they experienced, the tools and methods they used to either teach or learn the Chinese language.

I created an affinity map to identify the main barriers in language learning and teaching in Chinese American households.

  1. Traditional methods (i.e. Chinese classes, textbooks, etc.) were both scarce and unengaging for the children.

  2. Absence of consistent, structured, and repetitive teaching at home was due to constant code switching when children could not express themselves fluently.

  3. Children lacked the motivation to learn Chinese because they were trying to assimilate into the US culture and had no Chinese peers or communities to converse with.

Through generative research, I uncovered 3 main challenges:

We recognize that although many parents enroll their children in traditional Chinese schooling, it is only one part of the child’s learning. Without sufficient reinforcement from parents at home, or natives to speak with, the child will not be able to communicate in Chinese. Therefore, we chose to reorient our challenge statement to helping parents foster consistent teaching and conversation at home.

3 main challenges View Full Report Here

CHALLENGE STATEMENT

How might we help Chinese immigrant parents teach Chinese at home to their 3 to 6 year old children?

With this new challenge statement in mind, I developed 4 design principles to help our team with design and business decisions. These principles were formulated based on the the 3 primary challenges identified during user interviews.

4 Design Principles. See Full Definitions Here

IDEATION

Exploring Possible Solutions

I facilitated a brainstorming session with my team who each came from different fields. Recognizing that not everyone in the group was comfortable with drawing, I encouraged them to express their ideas in writing instead. I believed that including everyone and leveraging their various perspectives would be beneficial for the brainstorming process. We brainstormed 15 ideas and down selected to 3 using our design principles as a metric. See Ideation Suite Here

Top 3 Ideas:

These ideas were chosen because they incorporated all 4 design principles

  1. Digital picture book - interactive digital picture book that allows parents and children to learn together through Chinese stories

  2. Co-Chinese Games - Co-op Chinese games that allow children to work on tasks together with their parents, fostering conversation in Chinese.

  3. Chinese dinner conversations - A set of digital flashcards with prompts that could aid dinner time conversations

About the Final Idea: Learning Chinese Through Personalized Stories

The personalized digital storybook teaches Chinese through stories that parents and children craft together. It uses machine learning to weave in details that they input into the platform such as character name, genre, etc. The AI model then generates the corresponding images while incorporating appropriate vocabulary words for the child’s learning level. With each collaborative session, the vocabulary words are rearranged and repeated strategically to enhance retention and learning.

We chose this idea because of compelling research that supports the effectiveness of storytelling as a language learning technique. The key insights from our research can be summarized as follows:

  1. It enhances vocabulary and helps children understand how to use language in specific contexts.

  2. It Increases oral and communication skills

  3. Motivates children to learn language because of its engaging storylines

Additionally, our competitive analysis revealed that no products or services leverage collaborative storytelling between parent and child as a means to teach language.

MVP DEFINITION

Given the time constraints, the highest impact features were selected for the minimum viable product

Our main features aim to solve the 3 problems uncovered during user interviews–lack of motivation, unengaging and unstructured teaching methods. The features are as follows: 

  1. Personalized stories generated with AI. We included this feature because stories with the child’s interests in mind are more engaging and would increase motivation.

  2. A progress tracker to encourage consistent teaching at home 

  3. Related activity recommendations to encourage structured discussion and conversation after the storytelling session.

I then mapped those features on a user flow and made wireframes.

TECHNICAL TESTING

Testing the Technical Viability of the Product

To populate the platform with personalized stories, we planned to utilize AI tools such as ChatGPT, a powerful language model for story generation, and DALL-E for image generation. Recognizing the significant role these tools play, we decided to test their viability. However, during testing, we discovered certain limitations in the technology's creativity:

  1. Subpar quality Chinese stories
    To ensure the stories effectively motivate children, they must be engaging. Currently, the stories generated by ChatGPT lack liveliness and require adjustments by our Chinese team member. As a temporary solution, we are considering using Amazon Turks or hiring creative writers to make the stories more captivating. However, we acknowledge that this is only a short-term fix, and as we scale the product, we will need to enhance the AI model further.

Generated Chinese stories with vocabulary / keyword inputs

English translation of generated Chinese story

2. Generating stories and images take time
The generation of stories and images is not as instantaneous as initially anticipated. It currently takes several minutes for ChatGPT and DALL-E to create stories with corresponding images. To address this, we propose generating only a limited selection of 2 to 3 stories at any given time. Additionally, the stories would need to be generated in advance prior to the session.

3. Mismatched imagery from Dall-E
When creating a story, the images should have a uniform look. However, the images from Dall-e and Craiyon don't have the same drawing styles. Our software developer on the team said that we would need to articulate precisely what kind of drawing style we want per story.

Images generated from Dall-E

Moving Forward Despite Initial Technical Problems

Despite the initial setbacks, we were confident in the potential of the technology to create personalized and instant stories for language learning. Although we couldn't make the necessary adjustments to the AI model in time for the demo due to time constraints, we anticipate that our software developers can fine-tune and adjust ChatGPT's parameters to enhance its creativity and the quality of story generation. Techniques such as modifying training data, adjusting hyperparameters, and implementing reinforcement learning can be employed to achieve better creative output from ChatGPT given sufficient time for development.

OUTCOMES

After mapping out the user flow, I built a mid-fidelity prototype that I presented to 15 venture capitalists. The prototype I presented included the following features:

Created an Initial Demo for Venture Capitalists 

  1. Onboarding screens that learn about the parent and child so the AI model can create appropriate learning sessions for them

  • It collects the interests of the child and their favorite genre so the stories are both engaging and motivating.

  • Language level is collected so the appropriate vocabulary words are reshuffled in each story

  • The frequency and time allotment is collected so Jia knows the duration of each learning session and how often to notify the parent.

2. Personalized and AI generated stories incorporates interests and details of the child and vocabulary words appropriate for their language learning level.

  • AI reshuffles the words in subsequent stories until they are mastered by the child. The vocabulary words are color-coded:

    • Red - first time encountering the word

    • Orange - encountered but not yet mastered

    • Green - mastered the word

  • Word mastery is assessed at the end of the story by the parent. They just need to input whether the child had a hard time reading or remembering the word

3. Recommended activities after the story help induce discussion which aim to increase word retention and oral communication skills. All activities are short, illustrated and easy to follow.

4. The progress tracker shows a calendar of reading sessions, the words learned on a specific day, and the activities that took place. This ensures structured and consistent language learning for the child.

IMPACT & NEXT STEPS

Thoughts and Feedback After the Demo

Overall, the demo went well and the VCs were very pleased with our idea and business. They also gave us a couple of comments on how to move forward with our idea:

  1. They had asked us why we had not launched yet and we said that we wanted to do further testing. However they replied and said that It’s important to be first to market even if the product is not perfect yet. Testing and iteration can still be done during the beta phase.

  2. Our market is larger than Chinese immigrant families, it can be applicable to any family that want their child to grow up in a bilingual household

  3.  Identify how we are testing engagement and measuring learning over time.

Called Back for a 2nd Meeting with A Venture Capitalist

Surprisingly, after the demo presentation, we were called back for a second meeting with Trilogy, one of the venture capitalist firms at our pitch demo. They were interested in understanding how we were moving forward and gave us a couple of resources we could use for the development of Jia.

As a 5-member team, we’re currently moving forward with two types of user testing:

  1. Evaluative test of our current mid-fidelity prototype

    • Objective: The objective of the evaluative test is to understand if the flow and functionality of the current prototype is intuitive

  2. A one-page website with a 24-hour story generator

    • Procedure: One-page website that has a text field collecting parents information. They will then receive their AI-generated story after 24-hours and can use it for their child. Follow-up interview with parents after.

    • Objective: To test the value of AI-generated stories as a teaching technique and to gauge traction from the parents.

REFLECTIONS

What Would I do Differently?

  1. Early rapid prototyping and evaluative testing with the target audience
    What made this project difficult was we were unable to settle on a problem space until the last month of the quarter. This left us with very little time to do good research and test with participants. However, if I had more time, I would have liked to test even with rapid paper prototypes

  2. Work more closely with developers to identify best image generators
    Tinker and play with DALL-E or other image generating platforms to create better quality images. Understand the constraints and how the quality of these images can be improved.

  3. Run an ethnographic study and interview SMEs in the language learning space
    Conducting research on how parents currently teach their children and how language is taught in traditional Chinese institutions would have provided valuable insights to inform the design of our prototype. Observing these teaching methods could have guided us in creating a more effective solution. Additionally, interviewing subject matter experts (SMEs) would have helped us gain a deeper understanding of the vocabulary and appropriate learning levels for different age groups.