Anne Teravainen-Goff | August 2018
Pilot Evaluation:
Mrs Wordsmith Narrative Journey
Children have increased their overall knowledge of vocabulary by 50% when taking part in the Narrative Journey programme (increasing from 8.2 points on average before taking part in the programme to 12.3 points following the programme). Children were better able to identify synonyms and word pairs for Narrative Journey words after taking part in the programme.
Analysis of children’s writing data suggests that children incorporated more Narrative Journey words but fewer synonyms in their writing after taking part in the programme. This might suggest that children have learned more of the words and want to use them in their writing. In addition to increased vocabulary, after taking part in the programme, more children felt that pictures help them to remember new words and that if they know a lot of words, they can improve their writing. Children who scored fewer marks in the pre-test increased their knowledge of vocabulary more than their higher-scoring peers. Their score saw a significantly greater increase in the word pairing task but not in the synonym task, suggesting that the word pairings might have been particularly beneficial to the lower-scoring children.
Introduction
Mrs Wordsmith is a vocabulary programme that combines data science and creativity to show children the words they need for school and life. Hilariously illustrated by the Emmy award-winning artists behind Madagascar, this is a research-driven vocabulary product aiming to make learning vocabulary manageable and engaging. Developed with leading literacy experts, The Narrative Journey illustrates storytelling words for 6-13 years, with the aim to boost their reading and writing age. The words are grouped into six themes: Character, Weather, Taste and Smell, Action, Emotion and Settings. Each set includes a Word of the Day workbook and display stand, Word Pair cards and an illustrated dictionary. The classroom set also includes a teacher’s guide, which provides additional data and teaching ideas for each word.
This pilot evaluation explored the benefit of the Narrative Journey on children’s vocabulary learning and their attitudes towards it within the classroom. The purpose of this evaluation was also to pilot the vocabulary test based on the Mrs Wordsmith product and the attitudinal questions so that they can be used to conduct a larger-scale evaluation of the product in the future.
Methodology
The National Literacy Trust was commissioned in April 2018 to conduct a pilot evaluation of the Narrative Journey used in four Oasis Academy schools in Bristol over the summer term of 2018.
Instrument
The vocabulary test was created in collaboration with the Mrs Wordsmith schools team and the National Literacy Trust. The aim was to create a variety of tasks that measure children’s knowledge of vocabulary included in the Narrative Journey before and after taking part in the classroom-based vocabulary programme. The test was based solely on the character words in the pack to make learning all 48 of them over one term more manageable.
Three vocabulary tasks were created, using words chosen based on the Mrs Wordsmith team’s expertise and experience to ensure that the task provides a reasonable challenge without being too difficult.
IMAGE TASK:
Children were asked to match words to corresponding Mrs Wordsmith images from four multiple choice options. Eight words were included in the image task, chosen based on Mrs Wordsmith team’s expertise and experience to ensure that the task provides a reasonable challenge without being too difficult.
SYNONYM TASK:
Children were asked to match a word to a word with the same meaning. Again, children worked with eight words and chose answers from four multiple choice options.
WORD PAIRING TASK:
Children were asked to match a total of eight words to words that often go with them in a sentence from a range of four multiple choice options.
In addition to the vocabulary test, the instrument included attitudinal statements about vocabulary learning. These were asked at the beginning of the test and measured on a five-point scale ranging from “strongly disagree” to “strongly agree”.
The test was administered by teachers using a Google form. An initial coding error with the image task led to a loss of 90% of the data for this task in the pre-test. Although the error was fixed as soon as it was noticed, the significant loss at pre-test meant that this aspect of the instrument could not be assessed in this evaluation.
Children completed a writing task before and after the programme, which was then returned to Mrs Wordsmith team for analysis. Pupils were given several prompts made up of words from the Narrative Journey, and chose from a variety of topics. Statistical methods were then used to assess whether children had used more words from the Narrative Journey character pack, their synonyms or collocations after taking part in the programme.
Sample
181 children filled the pre-test and 157 carried out the post-test. 122 responses from three schools matched in the pre- and post-test based on children’s initials and date of birth. Demographic information, such as gender and free school meal uptake, was not collected as part of the pilot and data from the writing task before and after the programme were only available for 54 children.
Analysis
The vocabulary test items were coded as “correct” and “incorrect” in both the pre- and post-test. The correct answers were awarded one point and incorrect answers no points. A maximum vocabulary score of 16 was available as there were a total of 16 items in the synonym and the word association tasks. This score was used to analyse the difference in children’s vocabulary knowledge before and after taking part in the activities.
As well as looking at the overall vocabulary score, each of the two tasks, the synonym and word association tasks were also analysed separately. As both had eight words, the maximum score was eight points in each task. The analysis also explored the percentage of children who got each item correct in pre- and post-test. The attitudinal questions scale was coded from 1 (strongly disagree) to 5 (strongly agree) and the mean difference was explored between pre- and post-test for all the items individually. All findings presented in this report are statistically significant at the conventional 0.05 level.
Findings
The data show that children have improved their knowledge of vocabulary over the course of the summer term. Before taking part in the programme, children scored an average of 8.2 points out of 16 in the vocabulary test. After taking part, they scored an average of 12.3 points; a mean difference of 4.1 points. This suggests that after the end of the programme, children had increased their vocabulary by 50%.
SYNONYM TASK:
Children scored statistically significantly higher in the synonym task after taking part in the programme. Their average score increased from 4.3 in the pre-test to 6.2 in the post-test, a mean difference of 1.9 points. This indicates that children have increased their knowledge of the character words by 44%.
Table 1 presents the percentage of children who answered each item correctly in the pre- and post-test. It shows that while there is a statistically significant increase in identifying the synonyms correctly for most words, some have seen a particular increase. For example, the words “vacant” and “gargantuan” show a particular increase of children knowing their meaning at post-test. However, the results also show that children were already quite familiar with certain words such as “sheepish” and “impeccable”, and the percentage of children who identified synonyms correctly for “cunning” and “bumbling” was not significantly different between pre- and post-test.
WORD PAIRING TASK:
The word pairing task also showed improvement in children’s vocabulary. Their scores increased on average by 2.2 points, from 3.9 at pre-test to 6.1 at post-test, an increase of 56%.
Table 2 shows the percentage of children pairing each word correctly in pre- and post-test. While all words have seen an increase in the number of children identifying the word pairs correctly, there is a particularly large increase with words such as “bloodshot” which has seen an 82% increase in the number of correct answers and “shrivelled” which has seen a 72% increase in correct answers between pre- and post-test.
WRITING TASK:
Analysis of children’s writing data showed that overall, children used more words from the Narrative Journey after taking part. On average, children used 3.9 words before and 5.3 words after taking part (see Table 3). This increase is particularly interesting as the words were available to children in the writing task prompt both in the pre- and post-programme writing task. This might suggest that children learned to use the words accurately during the programme and hence incorporated more in their writing.
However, children used fewer synonyms of these words after the programme activities (4.0 vs. 1.7). This might suggest that children favoured words included in the Narrative Journey set over their synonyms and used them at the expense of other words. Word collocations were not used by children before or after taking part, which may be due to the nature of the task.
Changes in attitudes towards vocabulary learning
After taking part in the programme, more children felt that pictures help them to remember new words better and that if they know a lot of words, they can improve their writing. As Figure 1 shows, before taking part, just over half the children felt that pictures help them to remember new words better, compared to over three in five after taking part. Similarly, 73.8% of children agreed that knowing a lot of words will make them better writers before taking part compared to 80.4% after taking part.
In contrast, while there also was an increase in the number of children at post-test who agreed that learning words is fun, this difference was not statistically significant. Interestingly, there was a drop over time in the number of children who agreed that knowing a lot of words can make them a better reader, but again, this difference was not statistically significant. The percentage of children who agreed they find it hard to learn new words was similar in pre- and post-test.
These findings are particularly interesting as more children started to think that knowing a lot of words can improve their writing, but they did not feel the same about reading. It is possible that the writing tasks have helped more children to see the benefit of learning vocabulary. Similarly, it is interesting that while children did not necessarily see learning as more or less fun after using the Narrative Journey, more of them feel that pictures help them learn new words. This might indicate that the novel aspect of using pictures for learning vocabulary was perceived by children as useful and contributing to their learning.
Did any children in particular benefit?
In order to assess whether there were any groups of children who benefited from the Narrative Journey in particular, the sample was divided into “lower” (n= 50) and “higher” (n= 57) -scoring children based on their overall score in the pre-test.
The results show that children who scored lower in the pre-test increased their knowledge of vocabulary more than their higher-scoring peers. Lower-scoring children increased their overall score by 4.6 points (from 5.3 to 9.9) and higher-scoring children increased their overall score by 3.3 points (from 10.9 to 14.2). Interestingly, while lower-scoring children had a significantly greater increase in their word pairing score than their higher-scoring peers (2.7, from 2.3 to 5.0 vs. 1.6, from 5.4 to 7.0), there was no difference in their scores in the synonym task. This might indicate that the word pairings have been particularly beneficial for the lower-scoring children to start learning new words, but they might have not have completely understood the meaning of the words yet.
Conclusion
The findings of this pilot evaluation indicate that children who used Mrs Wordsmith’s Narrative Journey increased their knowledge of vocabulary. This is particularly true for children who started with a lower vocabulary score. In addition to increased vocabulary knowledge, more children feel that pictures help them learn new words and that knowing a lot of words makes them better writers.
Next steps
While the findings suggest that children have learned more words, the small sample size and the absence of a control group mean that it is not possible to attribute the improvement solely to the programme and determine whether the images in particular have made a difference in children’s vocabulary learning. In particular, future evaluation should focus on establishing whether children have actually acquired the meaning of the word from the pictures and can use it in a sentence, rather than simply matching the letters and the image.
Future research could also focus on exploring whether the Narrative Journey is particularly beneficial for one group of children based on demographic factors such as gender and socioeconomic status. It would also be useful to test whether picture-based vocabulary learning can be used to improve poorer readers’ skills.