Be a Summer Learning Hero! Share resources, earn rewards with Mrs Wordsmith! Sign up now

 

Successfully added to your Shopping Cart:

X
Add items to your cart to receive free shipping.

Grand Total (0 items)

€0,00

Continue shopping

A test of Word Tag:  Education games 
for at-home child & adolescent populations | 2023

Abstract


In the summer of 2022, we recruited twenty-seven families to participate in a project to measure the efficacy of an educational game. The game was offered to these children on iOS and android platforms. The scientific goals underscore the advantages of educational games, including identifying if children can learn while being intrinsically motivated to participate in an educational activity. Children in our research study showed over a full letter grade improvement in vocabulary testing across approximately twenty-eight days.

Most children reported wanting to play this game every day for various reasons, including fun, I prefer this to homework, and I like racing and collecting.The implications of this research suggest that while extremely difficult to balance fun with learning, games like the one in this study show promise for further study, development, and implementation in both home and classroom settings.

METHODOLOGY

Summary

Twenty-seven families were contacted who had children between seven and twelve to participate in a vocabulary builder game study. Nineteen children from those families were included in the final analysis after excluding those who did not complete the required measurement activities (pre or post) or failed to start the study. Four children failed to complete the pre-test requirements and were disqualified from participating in the experiment due to vacation priorities or other plans. Children were asked to participate in a game study over four weeks during the summer break. 

No children in our cohort attended summer school, but seven children were homeschooled in year-round school with light schooling during the summer. The children were asked by the researchers to play an average of fifteen minutes a day and provide usability feedback to the research team in written and interview format. Children were given a vocabulary and word familiarity test to understand what Lexile level they should be placed into to start the game. Over four weeks, the research team did virtual meetings with each family, collecting feedback about the product, including usability and sentiment data.

Experimental Manipulation (The Game)


The game is played on iOS or Android devices (phone format) no more than four years old. At the time of the experiment, the game was considered in the development phase and was not feature-complete. The game gave the child control of a humanoid animal avatar (fox) that would explore virtual environments, complete word tasks, study, collect, and compete in races and other activities. The words were introduced in a word tagging or spray-painting format, with the main avatar interacting with walls in a graffiti-style interaction. Each word was encountered up to 8 times following a spaced repetition schedule.

The encounters with each word involved an activity focusing on the pronunciation and spelling of the word, a synonyms or collocations quiz, and a sentence game requiring the player to select the right word to complete a sentence among four options. The words included in the game were selected based on wordlists from popular exams and popular fiction books as well as words recommended by academics specializing in vocabulary teaching (e.g.Biemiller, 2010). The words were also assigned a difficulty score based on which they were split into different levels of difficulty. The difficulty score assigned to each word came from theLexile Word Bank databaseof words commonly appearing in textbooks for K-12. The pre-tests performed by the research team ensured that each child would be enrolled to the right level of the game and would therefore be presented with words that would match their reading ability, without being already known, nor too challenging to learn. The research team was interested in studying the learning of thirty words presented in the game over the course of four weeks. Although the game contained more words, the research team limited the current study to the first thirty words that would complete a rotation through interactions with the game.

Measures

Children were given a thirty-question pre, and post-test with the words and answers scrambled to help prevent test-retest improvements. The tests were given approximately thirty days apart after the experiment measurement phase of four weeks had concluded. A word familiarity test was provided in the pre-test phase to help align children to one of five available Lexile levels in the game: 

While the researchers were hopeful to have balanced Lexile levels, seven children moved Lexile levels after the study started providing us with unequal Lexile levels. Lexile scores and assessments are commonly used throughout the United States and Lexile reading assessments are offered as part of official state assessments in 21 States.

Engagement Data

The research team working with developers were able to estimate the average playtime of 484 minutes or twenty minutes per day over the course of twenty-four days. The engagement was also collected from the parents in the form of two interviews, and these feedback sessions provided us with data that gave us some confidence that most children were playing four to five days a week. Based on parent feedback and the available server data, we estimate most children had play sessions of thirty to forty minutes per day.

Efficacy results

All data were analyzed for violation of statistical assumptions. The maximum possible score on the pre-test and post-test is thirty (30) words correct. The available range of movement is the difference between the maximum possible score (30) and the pre-test words correct average which was 21.74 for our group (30 – 21.74 = 8.26 available range) (see figure 1).

Overall, a large effect size was found, Cohen’s d = 1.13 (considered very large) when comparing pre-test (21.74 words correct) to post-test (25.16 words correct) scores. The average gains were 43% when comparing the available range of movement (8.26 words) to the actual positive learning (3.42 words improved). (see figure 1)

Fifteen children showed gains with an effect size of Cohen’s d = 1.93 (considered huge) when comparing pre-test (21.93 words correct) to post-test (26.53 words correct) scores. The average gains were 58% when comparing the available range of movement (8.07 words) to the actual positive learning (4.6 words improved). (see figure 2)

Three children showed negative gains, and one child showed no movement for four children with negative or no gains. These children had an effect size of Cohen’s d = .39 (considered small to moderate) when comparing the Pre-test (21 words correct) to post-test (20 words correct). The available range of movement was an average of nine words, with the actual movement being the loss of one vocabulary word. (see figure 3

Efficacy Results Discussion

Summary

80% of the children reported enjoying the game by providing verbal feedback and statements and answering a four or a five on a five-point Likert scale measuring game fun. The game does have a significantly large effect size when comparing pre and post-test scores showing that many children (78%) who play this game will have vocabulary gains when they can master the controls and have an interest in the game’s mechanics and design. Those children who didn’t show performance gains were reported to be having issues controlling the product or navigation issues which resulted in lower product affinity scores on a five-point Likert scale (not enjoying the game is a 1,2, or 3 on a five-point fun Likert scale).

Analysis of Words Learned and Words Wrong

No trend exists with the words gained from the pre-test to the post-test with distractors. It was also important to provide a minimum of twenty days between the pre and post-test to avoid test-retest validity issues. However, the research team did see a trend in the incorrect terms, which was limited by the relatively small n within each Lexile group. For example, words such as campaign, passion, central, conflict, humid, content, aggressive, and superior were commonly answered incorrectly in pre-test and post-test measures.

Usability Results

User Sentiment

Fifteen children in our study described the gameplay as fun verbally and on a five-point Likert scale as either a four or five (1 = not fun at all to 5 = extremely fun). These children looked forward to playing the game daily throughout the study. The most identified positive traits of the product were its art style and artwork, gameplay including racing and collecting, and its ability to be played after the educational activities were done for the day.

Four children reported they didn’t enjoy the game, and the most common complaint (four children) was a lack of gameplay experience in the home environment using twin-stick virtual thumb controllers on a touch screen. The children who reported this finding had the lowest overall gameplay in terms of time and accomplishments and reported negative sentiments towards the product from day one due to their perception that the control scheme was too difficult. Three of these children who didn’t enjoy the game were also in our group with no gains or losses. Only one child who reported a three on the five-point fun Likert scale was in our group with gains.

Play frequency

The average play per child was four hundred and eighty-four minutes over twenty-eight days. Seventy percent of this gameplay time was accomplished in the first two weeks. The study's play frequency decreased as the cohort exhausted the game's content. Many of the children were driven to collect items available in the game. Once these items were attained, the children reported they wanted more to collect and would probably continue to play as the game added more activities and items.

A summary of the positive finding is as follows:

1. Most children want to play it

a. Collecting is fun

b. Getting rewards is fun

c. Exploring is fun

d. Skating, Parkoring, and Competing is fun

2. Parents like seeing their kids want to play

Lessons Learned

A group of parents (five) from the positive sentiment cohort raised a mild concern with a moderate tone in our interviews (meaning the parents casually mentioned this) that the gameplay might be too fun, and the children were continuing to play after the educational activities were accomplished for the day. These parents were concerned that this gameplay time could be allotted to other educational activities. Although only five parents were concerned with this, our overall sample size suggests this might be an issue with any product that attempts to be both a game and educational.

Two of the youngest children in our study expressed difficulty with navigation. These younger children had difficulty exploring the virtual environment and finding their way in and out of side missions.

All parents expressed the need for a parent report that speaks the parent language with summaries of daily, weekly, monthly, and yearly accomplishments that allowed the parents to quickly assess the child’s progression. Some parents stated that a simplified parent report would help with the parents engage the student with word reinforcement strategies outside the game, such as a discussion about vocabulary over dinner.  Research shows that children whose parents are involved in their education, perform better than children whose parents are less involved (e.g. Lara and Saracostti, 2019). Parental involvement is especially beneficial when it comes to language abilities (Hill and Craft, 2003). At the same time, other studies report that parents are not adequately involved in their child’s education because of lack of time due to work or raising multiple children (Baker et al., 2016). Thus, an informative and easy to follow progress report that parents could glance at quickly would facilitate parental involvement which in turn could further reinforce children’s vocabulary performance.

A summary of the lessons learned is as follows:

Negative Findings

1. Parent report needs to be designed to speak the parents’ language

2. Some parents are skeptical about the game being too much fun

3. Play frequency went down in the second interview

a. Some children had collected everything they wanted and were looking for new loot

b. Many children asked for new levels, characters, contests

4. Some Children can’t figure out how to play

a. Some children can’t figure out the controls

Usability Results Discussion

Overall, the game was reported as being quite easy to use, with the main feedback being on further simplifying the controls when fast, aggressive movement is needed (parkoring and jumping) and further refining the features around navigation and exploration so that younger players can have an easy and efficient wayfinding experience and highly memorable navigational framework. The parent report feedback from the parent was simple, focusing on speaking the parent’s language and presenting information in an easily digestible format for at a glance usage.

Summary

Most children aged six plus with moderate gaming experience and/or experience with digital devices will enjoy this game and show learning gains. Children who play other games similar to Minecraft or Roblox enjoy this product the most. Children under six with minimal or no gaming experience may have control issues and navigation issues but enjoy the game. Parents’ feedback is occasionally diverse – some think their kids are going to the game anyway, so they may as well get something out of it. Some parents think it’s too much fun and waste time wandering and playing. Most Kids picked up new vocabulary, but modifications to the presentation of words need adjusting (audio features and ability to return to a word). The kids would like to see more in the shop and more worlds to explore. Parents would like a simplified and efficient means of monitoring from afar via an emailed or easily accessed report of words learned, missed, or attempted. If word tag’s team goal was to create a game that distracts the kids with fun while learning new words in an intrinsically motivating experience for children, they have succeeded. However, some parents will always have an underlying philosophy of anti-gaming which this game at odds with a segment of the population.

Shawn Stafford, Ph.D, Applied Experimental and Human Factors Psychology Specialist, Professor and Head of Research Full Sail University, 

Dr. Tammy Stafford, Ed.D, M.Ed., Literacy Specialist, Adj. Professor University of Central Florida