Introduction
In the last forty to fifty years, research has been performed, and dynamic discussions have taken place, in the field of L1-education in many countries. In discussions on the development of writing and linguistic understanding in school, previous research has pointed out the importance for teachers as well as students to use metalanguage to analyze, discuss and assess aspects of form as well as content in different text types (Gebhard, Chen & Britton, 2014; Macken-Horarik, Sandiford, Love & Unsworth, 2015; Myhill & Newman, 2016; Schleppegrell, 2013). For example, Chen and Myhill (2016) argue that language to talk about language and the process of concept formation (in this case the formation of a formal metalanguage) is a premise for developing metalinguistic understanding. However, previous research on, and assessment of, early reading and writing development has to a large extent focused on formal aspects of reading and writing, such as code breaking and spelling (Skoog, 2012). In a Swedish context, neither in teaching practices, nor in the curriculum, have aspects of how linguistic resources are used to create content been stressed to the same extent (Liberg et al., 2012).
In terms of how linguistic resources can be used, there are two main ways of commenting on texts: by means of formal language, i.e., technical language including grammatical and writing-related metalanguage, and by means of informal language, i.e. everyday language (Chen & Myhill, 2016; Jesson et al., 2016; Myhill & Newman, 2016). It has furthermore been discussed that important dimensions of teachers’ knowledge are often a tacit set of criteria necessary for holistic assessments of texts and these tacit criteria could, when verbalized, be discussed as parts of a formal as well as an informal metalanguage (Matre & Solheim, 2016). Macken-Horarick (2008, p. 47) takes this discussion even further as she argues that teachers could need a functional grammatics with metalinguistic concepts that are on the right level and possible to use without being trained as a linguist, “a metalanguage that is too general or too specific will not serve; we need one that is functional, stretchable and good for teachers to think with […]”.
Macken-Horarick et al. (2015) show that teachers tend to be insecure in their metalinguistic knowledge (which they name LSK, Linguistic Subject Knowledge). This makes holistic and content based assessments of student texts difficult, as shown by e.g., Myhill & Watson (2014). They argue that teachers with low grammatical knowledge have a tendency to communicate generalized principles for writing that students can find hard to operationalize in a meaningful way. With an increasing knowledge of a grammatical metalanguage, teachers are shown to change their view upon and approach to the role of grammar in teaching and assessing writing (Love & Sandiford, 2016; Macken-Horarick et al., 2015).
Based on the studies above, we assume that limitations of scope and use of metalanguage can lead to teachers’ underestimating or misjudging students’ literacy abilities. On this view, initiatives where teachers gain knowledge of, and confidence in, using technical metalanguage are of interest. The main aim of this article is therefore to investigate whether professional development workshops that introduce a model for discussing content-based aspects of students’ early school writing can have an impact on how teachers discuss texts, particularly in terms of their use of formal language. To explore this, we pose the following research question:
1. Are there any differences in how a group of primary teachers discusses the content, structure and form of two informational texts (written by two primary students) before and after participating in professional development workshops – particularly in terms of when and how they use formal metalanguage?
Further, since previous studies have focused on formal metalanguage, little is known about how teachers use informal metalanguage, as well as whether this use differs from how they use formal metalanguage. We therefore pose a second research question related to our first research question:
2. How is formal and informal metalanguage used in the discussions of the content, structure and form of the two informational texts before and after the professional development workshops?
The second research question involves a discussion of the functions and qualitative differences between formal and informal metalanguage that were observed in the study.
The study was carried out within the larger project Function, content and form in interaction. Students’ text making in early schoolyears1. In this project, the aim has been to develop a model for analyzing content-based aspects of students’ early school writing and thereby establishing a metalanguage to talk about such aspects of text. Of importance for the larger project has therefore also been to examine how a metalanguage to discuss such linguistic resources can be used to enrich teachers’ ways of looking at and assessing students’ texts.
Method and Materials
Design of the study
In order to find out more about how teachers talk about content, structure and form issues with formal as well as informal metalanguage, the study was designed as an exploratory study using teachers’ text discussions before and after professional development workshops as comparison points. With this design, we can acquire new insights of the use and function of formal and informal metalanguage as well as how teachers discuss content, structure and form in early school writing (RQ2). By comparing discussions before and after professional development workshops, the study can also give insight into the development of how the teachers use formal as well as informal metalanguage (RQ1).
Professional development workshops
14 primary teachers participated in the study. A letter of invitation to participate in professional development workshops concerning early school writing was distributed to all principals of elementary schools in one large municipality in Sweden. Among many interested, 14 teachers from 14 different schools were randomly selected to take part in the study, all of them teachers in grade 1–3. Since the teachers were randomly selected, they had varying experience of teaching and assessing writing. This was seen through a short survey where we collected information on number of years as a teacher as well as how well-known the different presented resources were for the teachers. The teachers participated in 6 professional development workshops. During the first and last workshop they were asked to discuss two narrative and two informational texts written by 7–9-year-old students, in groups of three teachers (and one pair). These discussions were supposed to focus on strengths and weaknesses in the texts as well as on comments that they would give to the student who had written the text. Instructions for the text discussion were given orally as well as in writing. Each text was discussed for 15 minutes, and the instructions as well as the texts discussed were the same for the first and last text discussion. The choice of using the same texts in the two discussions made it possible to compare teachers’ talk on these two occasions. Although commenting on the same texts twice can affect the result, the alternative to present new texts was excluded since the characteristics of texts give different opportunities for text discussions. Over two months passed between the introductory and closing text discussion. During these months, the participants discussed 16 other texts in connection with the workshops. In sum, we concluded that the advantages of using the same texts in the conversations before and after the professional development exceeded any impact on the result.
The four workshops were each organized around a separate theme. Participants were introduced to resources (and accompanying metalanguage) inspired by SFL-based text-analysis (Systemic-Functional Linguistics) as well as other resources well established within text analysis, e.g., concerning vocabulary. The theoretical framework for the resources presented was thereby mainly found within a social semiotic perspective. According to Halliday (1978) and Halliday & Matthiessen (2004), the semiotic systems that we live by are considered to form a meaning resource. It is from this meaning resource that we choose when we articulate and structure meaning. By these choices, certain aspects are put in the background or completely excluded while others are foregrounded and emphasized. In this respect, the selected language forms are highly significant.
The social semiotic perspective provides a well-developed theoretical framework and metalanguage for detailed analyses of different dimensions of meaning-making in students’ texts. Within this tradition, a number of linguistic resources have been pointed out as relevant for the teaching of writing. In the list below, these resources are formed into four groups (see Humphrey, 2017). The four groups were the basis for the four professional development workshops (workshop 2–5), and also guided the analysis of teachers’ talk2:
Resources to express and enrich content with a focus on resources such as word choice, word variation and vocabulary ranging from everyday to academic language (e.g., Beck & McKeown, 2007; Liberg, 2014; Ravid, 2006). Examples of concepts discussed during the workshop: word choice, word variation, everyday language and academic language, nominalizations.
Resources to expand and develop content in terms of expansions as discussed by for example Halliday & Matthiesen (2004) and af Geijerstam (2006). Examples of concepts discussed during the workshop: expansions, reformulations, extension, pre- and post-modifiers.
Resources to create a coherent content in terms of thematic coherence (e.g., Folkeryd, 2014). Examples of concepts discussed during the workshop: text structure on a macro and micro level, thematic organization, cohesion.
Resources to create a relationship with a reader with a focus on evaluative language use as discussed for example within the framework of Appraisal (Folkeryd, 2006; Martin & White, 2005). Examples of concepts discussed during the workshop: evaluative words, dialogue, explicit and implicit evaluation, sensory expressions.
Lectures on these resources were followed by recorded group discussions of student texts (different texts from the first and last text discussion). As a scaffolding structure for the discussions, the teachers were given handouts with the central metalanguage (keywords as seen in the list above) discussed during the lecture3. They were also provided with guiding questions such as the following (when discussing resources to create a coherent content): Is there a clear main theme? Is there one or several micro themes or subthemes? Do the subthemes fit under the main theme, or is a fragmentary content created when the subthemes deal with different content?
The texts discussed
In this study, the focus is on the informational texts that the teachers discussed during the initial and final discussions. Since strengths and limitations in the student texts affect potential text discussions, the texts chosen for discussion in the first and last discussions represented a variety of ways to express content. Both of the informational texts in focus for the present study were written during the fall semester in grade 3 in two different schools in Sweden. Text 1 was a relatively long text about space (464 words), without subheadings or separated paragraphs. The vocabulary used to express ideas consists of everyday language as well as more academic language (e.g., gravity, weather satellites, the Big Bang). The content is mostly coherently organized in subthemes such as the creation of Earth, Earth’s gravity, planets, satellites and space travels. These subthemes are furthermore expanded on, creating new subthemes such as space travels being discussed in terms of the space dog Laika or the astronaut Fuglesang. There are plenty of explicit as well as implicit evaluative language resources being used (e.g., the space is cool, In space there are 10 000 parts of blown up satellites!), thus potentially getting the reader engaged in the content.
Text 2 was written about the Iron Age. This text is shorter (198 words) and the information is organized under 12 subheadings dealing with topics such as animals, food, weather, housing, clothing, etc. The vocabulary used to express these topics consists of everyday language (with the exception of words such as thralls, cloak, runes), correctly spelled and formed as complete main sentences. Although rich in subthemes that all fit under the main theme about the Iron Age, most of the topics are not elaborated on to any greater extent (e.g., Food. One could eat meat. One could bake bread.). There are examples of expanded content (e.g., If you were rich and died they could kill animals and thralls because they believed in an afterlife and then it could be good to bring some things) but mostly the text resembles a list of facts. Very few evaluative language resources were used in order to create a relationship with the reader.
Analysis
As previously mentioned, the same texts were discussed during the first and last workshop in order to investigate the effect of the professional development workshops on the way teachers talked about the informational student texts. These discussions were transcribed and analyzed qualitatively as well as quantitatively. As a first step in the qualitative analysis, all instances (words or word groups) from the transcripts dealing with the central themes for the workshops (the different types of resources described above) were extracted and categorized according to type. In addition, instances of teachers’ comments about correctness in spelling and sentence formation were added as a fifth dimension of resources. In a subsequent step, the excerpts were also marked for being expressed through formal or informal metalanguage and whether the chosen metalinguistic expression was supported with a clarifying example from the text. The coding of the excerpts into themes as well as the coding of formal and informal metalanguage was performed by the four researchers in the research group. The same researchers also designed and conducted the professional development workshops. The fact that the same researchers conducted the workshops and coded the transcripts could have affected the results to the extent that the researchers had a pre-understanding of what to find in the discussions. On the other hand, since the study was designed to gain new insights into the use of formal and informal expressions a well as exploring how teachers talk about content, structure and form using five dimensions as analytical framework, this pre-understanding was also an advantage in the analytical process.
Inter-rater reliability was investigated by individual ratings of three transcripts from the group dicussions, by the four researchers in the research group. A calculation of the results showed a Fleiss kappa-value between 0.66 and 0.84 for the five dimensions. According to Landis and Koch (1977) a kappa-value between 0.61–0.80 indicates “substantial agreement” while values between 0.81–1.00 indicate “almost perfect agreement”.
Results
The quantitative analysis shows no significant difference in the total number of metalinguistic comments before and after the workshops, 305 instances before the workshops as opposed to 326 instances after. In other words, the teachers did not talk more about the linguistic resources used in the student texts after the workshops (such as how students express and enrich content or how they create a coherent content). What is also immediately noticeable is that when teachers talk about the different dimensions of linguistic resources, they use both formal and informal metalanguage before as well as after the workshops. When for example commenting about how students express and enrich content, they might use expressions such as word variation and colloquial language (formal metalanguage) as well as many words or not super correct words. However, formal metalanguage replaces informal metalanguage to a noticeable degree. While 52% of the total number of instances is expressed through formal metalanguage before the workshops, this number increases to 60% after. The number of informal expressions subsequently decreases from 48% before to 40% after. This redistribution of metalanguage is shown in Figure 1 below.
As can be seen in Figure 1, teachers’ talk mostly about creating a coherent text before the workshops and also about formal aspects such as spelling and punctuation. After the workshops, the largest change has to do with the teachers talking less about spelling and punctuation. This is an expected result considering the long tradition within the Swedish school system to focus on formal aspects. It is also expected based on the content of the workshops which did not foreground linguistic resources to create a formally correct text. The largest increase of instances had to do with words and expressions for creating a relationship with the reader, e.g., the use of evaluative language or dialogue. However, informal metalanguage is still used to the same extent as formal metalanguage after the workshops when talking about relationship to the reader. A possible explanation to this emerged in interviews with the teachers where they expressed that they were less familiar with such resources. A likely interpretation of the result is therefore that the workshop helped the teachers recognize resources in the text that can create a relationship with the reader, but the more technical ways of talking about these resources might not yet have been internalized.
In order to further develop the discussion of formal and informal metalanguage, it is of interest to exemplify qualitative similarities and differences. In Table 1 examples of formal as well as informal metalanguage are given, as used by the teachers when they talked about the four different types of resources.
| Resources to … | Formal metalanguage used | Informal metalanguage used |
|---|---|---|
| express and enrich content | word length, word choice, word variation, colloquial language, compound words, word packing, academic language, nominal phrases, nominalization | good words, many words, many sentences, rich language, their own words, not super correct words, difficult words, casual wordings |
| expand and develop content | descriptions, develop, connectives, conclusions, comparisons, explanations | filled with details, sparse with information, brief, repetitive, stick with it a little bit longer |
| create a coherent content | course of events, chronology, context, macro/micro theme | haphazardly, with flow, confusedly, clearly, in the wrong order, with just one sentence about each thing |
| create a relationship with a reader | adjectives, reinforcement words, omniscient narrator, graduation, appreciation, evaluative language | appeals to the reader, creates an image in our mind, expresses passion, makes the reader curious, personal touch |
As seen in Table 1, formal as well as informal metalanguage is used to talk about all four resources. The functions and qualitative differences between formal and informal metalanguage will be further discussed in the following and concluding section.
A widened approach to student writing
It can be concluded from the study that the workshops resulted in widened teacher discussions of different aspects in texts where they used formal as well as informal metalanguage. Interestingly, it can be noted that in many cases where certain passages in the student texts were identified and commented on using informal metalanguage before the professional development workshops, the same passages were foregrounded by the teachers using a formal metalinguistic expression after the workshops. One such example is when the student’s choice of words in the text about the Iron Age, was commented on as being in his or her own words in the pretest, but as the student using colloquial words in the posttest. Another example is how the teachers talked about the coherence of content in the text about space. Before the workshops, coherence of content was talked about as with flow, whereas teachers pointed out how the text achieved coherence by organizing several micro themes under a general macro theme after the workshops. In this case, the use of formal metalanguage added precision to what actually happened in the text that creates flow.
In some cases we could see that the workshops resulted in a reevaluation of aspects in the student texts. Thus, they did not only result in the teachers noticing new aspects. One example of a reevaluation was found as teachers discussed the following extract from the student text about space:
And then a lot of large rocks flew around in space and they bumped into each other and created larger rocks which then created earth! It would have been cool to see the Big Bang!
Before the workshops, one teacher said that: “This doesn’t belong in an informational text. The student needs to be told not to include personal opinions in this way”. After the workshops the same extract was commented on by the same teacher as: “Although this doesn’t really belong in an informational text it could be that we can see an engaged child through the text”. In this case, informal language was used in both cases to identify the phenomenon, but the view on how the student choice of wordings affected text quality had changed. This could be interpreted as an effect of discussions during the workshops about how evaluative language use can say something about the writer’s involvement in the content and also potentially create a relationship with the reader.
Although it has been shown that formal metalanguage can make comments more specific, this is not the only way to achieve this specificity. In some cases, informal metalanguage can also serve this purpose. Through the informal metalanguage, there is a tendency that textual aspects are evaluated to a larger degree for example through expressions such as good words, rich language or not super correct words. In these examples the precision of what exactly makes the language rich or not correct is less than what could be expressed through a formal metalanguage mentioning the extensive use of adjectives or the lack of subject specific academic words. In other examples however, the use of informal evaluative metalanguage can be rather precise as to how the text affects the reader, although not using formal metalanguage and not pinpointing the expression in the text that causes this feeling. Such examples can be when the teacher says that the text appeals to the reader, creates an image in our mind, expresses passion, makes the reader curious, personal touch.
Finally, it could also be argued that there are no clear cut borders when the degree of precision of expressions is concerned. Instead, we could talk about a continuum of precision, possibly ranging from (1) using isolated informal metalanguage that identifies phenomena, to (2) exemplifying the phenomena and using informal metalanguage, to (3) using formal metalanguage with added examples and explanations, and finally to (4) (possibly) using metalanguage of both types with examples and explanations. A fabricated example illustrates the steps on the continuum:
1. The use of evaluative language in a text could be commented on using only informal language such as the text contains feelings.
2. An example from the text could be added to clarify how feelings are expressed in the text such as the text contains feelings as when it says in the text “the boy was angry”.
3. The same passage could also be commented on by using formal metalanguage such as there are expressions of negative affective evaluative language in the text.
4. Formal metalanguage could be used in combination with an example such as there are expressions of negative affective evaluative language as when it says”the boy was angry”.
Finally a combination of these metalinguistic comments could be used: This text is filled with feelings as when it says”the boy was angry”. These expressions of negative affective evalutative language really create an image in my mind as a reader and makes me want to read further and find out what caused the anger.
However fictive, similar examples have been found in the study, although not all used by the teachers in relation to one specific aspect in the student text examples. That leaves us with the question of what type of metalanguage teachers need.What this study has shown is that the workshops on different resources to create content in text resulted in the teachers foregrounding new dimensions in text, such as how the student creates a relationship to the reader. They also discussed these dimensions using a formal metalanguage to a larger degree, thereby commenting on affordances and limitations in the student texts in a more specific way. The study furthermore showed that textual dimensions were also identified through the teachers using informal metalanguage, and that this informal metalanguage also could be functional in expressing for example a relationship to the reader.
Our study has shed light on similarities and differences between teachers’ use of formal and informal metalanguage and the affordances and limitations of both. We have shown that teachers use a more informal metalanguage that can be functional and good to think with in order to improve student writing (see Macken-Horarik, 2008). We have also shown that the use of a formal metalanguage (together with examples from the student texts) makes teachers’ comments about student writing more specific, thereby expressing a metalinguistic knowledge that can facilitate holistic and content based assessments of student texts (see Myhill & Watson, 2014). A widened approach, including informal as well as formal metalanguage, to discuss metalanguage as a prerequisite for teaching and assessing writing is necessary in order to encourage a language development that prepares children for a changing text landscape.
References
- Beck, I., & McKeown, M. (2007). Increasing Young Low Income Children’s Oral Vocabulary Repertoire through Rich and Focused Instruction. The Elementary School Journal, 107(3), 251–271. https://doi.org/10.1086/511706
- Chen, H., & Myhill, D. (2016). Children talking about writing: Investigating metalinguistic understanding. Linguistics and Education, 35, 100–108. https://doi.org/10.1016/j.linged.2016.07.004
- Cope, B., & Kalantzis, M. (Eds.). (2000). Multiliteracies: Literacy Learning and the Design of Social Futures. London: Routledge.
- Dixon, J. (1975). Growth through English. Set in the perspectives of the seventies (3rd edition). London: Oxford University Press for the National Association for the Teaching of English.
- Folkeryd, J. W. (2014). Hjärtan, hjul och hävstänger – Innehåll i elevers sakprosatexter på lågstadiet. [Hearts, weels and levers – Content of students’ expository texts at lower school]. In P. Andersson, P. Holmberg, A. Lyngfelt, A. Nordenstam & O. Widhe (Eds.), Mångfaldens möjligheter – litteratur- och språkdidaktik i Norden. Göteborg: Elfte nationella konferensen i svenska med didaktisk inriktning. Retrieved from oai:DiVA.org:liu-141971.
- Folkeryd, J. W. (2006). Writing with an Attitude – Appraisal and student texts in the school subject of Swedish (PhD dissertation). Institutionen för lingvistik och filologi, Uppsala, Sweden.
- Gebhard, M., Chen, I., & Britton, L. (2014). “Miss, nominalization is a nominalization:” English language learners’ use of SFL metalanguage and their literacy practices. Linguistics and Education, 26, 106–125. https://doi.org/10.1016/j.linged.2014.01.003
- af Geijerstam, Å. (2006). Att skriva i naturorienterande ämnen i skolan [Writing in science-oriented subjects in school]. (PhD dissertation). Institutionen för lingvistik och filologi, Uppsala, Sweden.
- Halliday, M. (1978). Language as social semiotic. The social interpretation of language and meaning. London: Edward Arnold.
- Halliday, M., & Matthiessen, C. (2004). An Introduction to Functional Grammar. London: Arnold.
- Humphrey, S. (2017). Academic Literacies in the Middle Years: A Framework for Enhancing Teacher Knowledge and Student Achievement. New York; London: Routledge.
- Ivanič, R. (2004). Discourses of Writing and Learning to Write. Language and Education 18, 3, 220–245.
- Jesson, R., Fontich, X., & Myhill, D. (2016). Creating dialogic spaces: Talk as a mediational tool in becoming a writer. International Journal of Educational Research, 80, 155–163. https://doi.org/10.1016/j.ijer.2016.08.002
- Landis, J., & Koch, G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
- Liberg, C. (2014). Att tala om innehåll och ämnesstämmor i elevtexter i tidiga skolår. [Talking about content and topic voices in student texts in the early school years]. In R. Hvistendahl & A. Roe (Eds.), Alle tiders norskdidaktiker: Festskrift til Fröydis Hertzberg på 70-årsdagen (pp. 141–156). Oslo: Novus Forlag. Retrieved from oai:DiVA.org:uu-237239.
- Liberg, C., Folkeryd, J. W., & af Geijerstam, Å. (2012). Swedish – An updated school subject? Education Inquiry, 3(4), 477–493. https://doi.org/10.3402/edui.v3i4.22049
- Love, K., & Sandiford, C. (2016). Teachers’ and students’ meta-reflections on writing choices: An Australian case study. International Journal of Educational Research, 80, 204–216. https://doi.org/10.1016/j.ijer.2016.06.001
- Luke, A., & Freebody, P. (1999). Further notes on the Four Resources Model. Reading online. Retrieved from https://pdfs.semanticscholar.org/a916/0ce3d5e75744de3d0ddacfaf6861fe928b9e.pdf
- Macken-Horarik, M. (2008). A “good enough” grammatics: Developing an effective metalanguage for school English in an era of multiliteracies. In C. Wu, C. Matthiessen, & M. Herke (Eds.), Proceedings of the ISFC 35: Voices around the world, 35th ISFC Organizing Committee, Sydney (pp. 43–48).
- Macken-Horarik, M., Sandiford, C., Love, K. & Unsworth, L. (2015). New ways of working ‘with grammar in mind’ in School English: Insights from systemic functional grammatics. Linguistics and Education, 31, 145–158. https://doi.org/10.1016/j.linged.2015.07.004
- Martin, J., & White, P. (2005). The language of evaluation: Appraisal in English. London: Palgrave Macmillan.
- Matre, S. & Solheim, R. (2016). Opening dialogic spaces: Teachers’ metatalk on writing assessment. International Journal of educational Research, 80,188–203. https://doi.org/10.1016/j.ijer.2016.07.001
- Myhill, D. & Watson, A. (2014). The role of grammar in the writing curriculum: A review of the literature. Child Language Teaching and Therapy, 30(1), 41–62. https://doi.org/10.1177/0265659013514070
- Myhill, D., & Newman, R. (2016). Metatalk: Enabling metalinguistic discussion about writing. International Journal of Educational Research, 80, 177–187. http://dx.doi.org/10.1016/j.ijer.2016.07.007
- Nordlund, A. (2016). Berättarteknik i elevberättelser från tidiga skolår. [Narrative technique in students’ narratives from early school years]. Forskning om undervisning & lärande, 4 (2), 46–67.
- Ravid, D. (2006). Semantic Development in Textual Contexts during the School Years: Noun Scale Analyses. Journal of Child Language, 33(4), 791–821. https://doi.org/10.1017/s0305000906007586
- Schleppegrell, M. J. (2013). The Role of Metalanguage in Supporting Academic Language Development. Language Learning, 63, 153–170. https://doi.org/10.1111/j.1467-9922.2012.00742.x
- Skoog, M. (2012). Skriftspråkande i förskoleklass och årskurs 1. [Using written language in nursery class and class 1]. Örebro: Örebro University.
Fotnoter
- 1 The project Function, content and form in interaction. Students’ text making in early schoolyears was financed by the Swedish Research Council.
- 2 In the analysis of teachers’ talk a fifth category concerning formal aspects of text was added, as seen in figure 1.
- 3 During the workshops before and after the professional development no such supporting structures were used.