Using GenAI tools to refine EAP assessment

By Laura.Duckett, 31 October, 2024
View
There are several tools available to help English for academic purposes assess students’ writing skills. Here is some guidance on how to use a selection of them
Article type
Article
Main text

Students’ essay writing and digital literacy skills have become popular topics in English for academic purposes (EAP) assessment research in recent months. Before the emergence of ChatGPT and other large language models, EAP practitioners like ourselves would pilot the writing coursework task and refine the brief by writing sample essays. When using artificial intelligence-generated sample essays, we can gauge their key features to refine the writing briefs. 

Creating an EAP writing brief in Magic School AI 

The Magic School platform provides AI-powered and fully customisable tools for educators. It features its own AI chatbot, Raina, inspired by leading large language models from OpenAI, Anthropic and Google.

Raina has the ability to customise and generate teaching-related materials. It provides a range of custom prompts after the user’s initial prompt to help with the rewriting process and to offer supplementary resources. After several rounds of revisions, we used Raina to generate an EAP writing brief on the topic of the advantages and disadvantages of AI in higher education. Figure 1 shows AI-generated examples of marking criteria for the essays.

Image
Figure 1. Raina (chatbot) in Magic School AI
Figure 1. Raina (chatbot) in Magic School AI

 Using Prepostseo to generate sample essays

Prepostseo is an online search engine optimisation and content writing toolkit. Its key features include a plagiarism checker, paraphrasing tool, grammar checker and text summariser, among other factors. We chose it because it shares some of the same features as Grammarly and QuillBot and used it to generate sample essays to show us what AI-generated essays look like.

Image
Essay Generator in Prepostseo
Figure 2. Essay generator in Prepostseo

As you can see in figure 2, the platform is easy to navigate. It offers 19 essay types which cover descriptive, persuasive, problem and solution, cause and effect and compare and contrast options. 

It only took a few seconds to generate three sample essays of short, medium and long lengths. The main similarities between them were the structure and inclusion of uncited information. They all followed a block structure with body paragraphs about the advantages and drawbacks of artificial intelligence in higher education. There was information used as supporting detail in one of the generated essays that was irrelevant to the essay topic.

It is worthwhile to examine AI-generated sample essays so that we can better identify AI-generated content and include more requirements about essay structure, source integration and logic into our essay-writing briefs. By using GenAI to refine these briefs, we can refine assessment, ultimately building students’ language and critical thinking skills

Tip for generating writing brief with Magic School AI

Identify the learning objectives: what specific skills or knowledge do you want the students to demonstrate? Are the learning objectives aligned with the whole module learning outcomes? Understanding this will help you craft a prompt that aligns with the identified learning outcomes.

Be clear and specific: use short and simple sentences and include the expected word count, structure requirements, reference system, source number and type and any specific requirements or guidelines you think students should follow. 

Provide clear assessment marking criteria: include information about how the writing task will be evaluated, such as the grading rubric or assessment criteria. These documents could be uploaded as attachments into Magic School AI. 

Describe your identity: Raina, the Magic School AI chatbot, can better understand and generate the end product if you let her know your identity. For example, “I am an EAP teacher from a Chinese university teaching year two undergraduate students. The writing task topic is ‘Benefits and drawbacks of integrating GenAI in higher education’. Students will be asked to write a discursive essay that discusses both sides of this topic in a relatively balanced manner. The word count is 300-500. The writing brief should include the essay topic, task description, format requirements, and structure suggestions. I attach a sample writing brief.”

Keep refining prompts: the first attempt might not generate satisfying results, so you need to be prepared to make multiple attempts to tweak and customise the brief by gradually inputting more specific instructions. Another possible way to keep refining your prompt is to rely on Raina’s automatically generated prompts for you. Each time, several prompts will be automatically generated to help users brainstorm possible instructions to improve the prompts (figure 3). You can also click on one provided prompt to generate further content in the chat box.

Mengqi Hu and Rui Xu (Tiffany) are language lecturers at the English Language Centre in the School of Languages at Xi’an Jiaotong-Liverpool University.

If you would like advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the Campus newsletter.

Standfirst
There are several tools available to help English for academic purposes assess students’ writing skills. Here is some guidance on how to use a selection of them

comment1

THE_comment

1 week 1 day ago

Reported
True
User Id
3642584
User name
AZRAEL
Comment body
Xu sensei 👍👍👍
jwt token
eyJhbGciOiJIUzUxMiJ9.eyJuYW1lIjoiQVpSQUVMIiwiZW1haWwiOiJxZXFlMzYzMzQyMTA2QDE2My5jb20iLCJpZCI6IjM2NDI1ODQiLCJpYXQiOjE3MzAzNTEyMzYsImV4cCI6MTczMjk0MzIzNn0.rq-6jmDhuplfYgmJW28mwy82NOf5imE0B8vE9XWko84tICXQkUrnPR-iHzQWnMv-Kv4_J3n_enAfdBu5sqBoXw
Reviewed
On