We’d like to thank Craig Hurwitz, an instructor in the Financial Technology Master of Engineering program in the Pratt School of Engineering, for this guest post.
ChatGPT has been a topic of great discussion in academia, particularly about how to prevent its unauthorized use in classes. However, students can benefit from understanding how to use generative artificial intelligence (AI) as a tool to save time and improve performance on writing assignments. Craig Hurwitz, an Executive in Residence in the Pratt School of Engineering, asked the graduate students in his “Emerging Trends in Financial Technology (Fintech)” course to generate a first draft of an essay with ChatGPT’s help. His working assumption was that when students enter the workforce they will have access to, and the ability to use, generative AI for productivity purposes. He wanted to experiment with ChatGPT in his course to give students a first-hand look at how to use generative AI.
For the assignment, the class was instructed to read a case study and each student chose a Fintech approach to help solve a particular challenge mentioned in the case. Their written assignment was a 750-word Executive Summary convincing the instructor (playing the role of Venture Capitalist) why he should consider meeting to discuss a potential investment.
The instructions called for each student to create a first draft using ChatGPT. After generating the draft, they imported the draft into Microsoft Word and turned on “track changes.” They edited the initial draft and submitted the final draft with “track changes” still turned on. This gave the professor the ability to see the draft generated by ChatGPT, the changes that were made to that draft, and the final product submitted by each student. The range of unedited final output ranged from about 20 percent to over 90 percent. Following the assignment, students completed a survey about their experiences on the assignment. This post summarizes the student reactions to using ChatGPT.
AI as a Work Skill
Students viewed this assignment as important training since they expect that they will be using AI in the workplace. There were many comments about how this assignment was a learning opportunity on how to use an AI chatbot effectively and how this is going to be a valuable skill when they graduate and take on full-time roles in the workforce. As one student explained, “I really enjoyed working on this assignment. I believe what you let us do is what will happen in real life. Completing this assignment is good practice for us to use AI to improve our work.”
One specific skill the students gained is how to write effective prompts in ChatGPT. Many students realized that developing an appropriate prompt is an iterative process and is the key to a successful experience. Almost all students embarked on this exercise with an attempt to use a single prompt to create the entire essay. However, most students found that the output was too general, and in most cases, too repetitive. The most common reaction to this was to submit to ChatGPT specific follow-up questions using descriptive, succinct text designed to create output for a particular paragraph, or more specifically, a bullet point. Several students commented on the importance of training ChatGPT to take on a particular role prior to the process of prompting it to generate any written output. “I’d place more emphasis on training the AI to adopt a certain persona, instead of spending a lot of time trying to figure out the right prompt,” one student commented in the post-assignment survey.
Writing Efficiency
Every student claimed that using an AI chatbot to help generate a first draft was helpful. Ninety percent of students said that using an AI chatbot was a time saver. One student remarked, “For every minute you spend on ChatGPT you are saving ten minutes, but be sure to give the AI the precise information you want it to work with and the risk of mistakes is minimal.”
Twenty-one out of 36 students in the class who speak English as a second language felt another benefit. Every one of them indicated that they already had experience using ChatGPT before this assignment. A common refrain was that using an AI chatbot helped them write as if English were their native language. However, as one student cautioned, “The grammar the AI used was perfect, but I found the language to be vague and unsophisticated.“ This comment points to students’ general understanding of ChatGPT’s limitations.
Limitations
While students found ChatGPT useful, there was significant commentary about limitations of this generation of the technology. Limitations generally fell into one of three categories: 1) incorrect sources, 2) outdated sources, and 3) the output only being as good as the prompt. “I found that ChatGPT generated information that was correct, but with incorrect citation,” one student noted. Over half of the class found that ChatGPT generated incorrect information and only two of 36 expressed confidence in the sources cited by ChatGPT. Students understood fact-checking is an essential part of using ChatGPT for writing, since source integrity is a big issue. In addition, since the students’ assignment was to highlight emerging trends in Fintech, they could not rely on ChatGPT’s older data (Sept 2021). Note that students used the GPT-3.5 model for this AI-assignment; that was before GPT-4.0 was released, when the ability to cite sources improved somewhat, but relying on the sources provided by GPT remains problematic.
Many students commented that they enjoyed working on the assignment, but they are aware of the ethical considerations such as intellectual property, and appreciated that guardrails are necessary. Students understood using ChatGPT can only be one step in the writing process. “The bot shouldn’t be used as a final response, but merely as a way to understand a topic or brainstorm ideas. ChatGPT’s ability to synthesize sources and condense information is great, but shouldn’t replace our own filters and ability to do that same job ourselves.”
Conclusion
Generative AI capabilities are advancing quickly. It is a productivity tool that will likely gain vast acceptance and usage in the workforce. Training students how to use the technology appropriately and effectively is an important life skill. To those worried about whether the output generated by ChatGPT is the work of the tool or the student, there is no substitute for doing the basic research, having a clear understanding of what it is that you want to communicate, and knowing, in advance, the written structure you expect to receive from ChatGPT. From the perspective of a professor who spent most of his career in industry, it was rewarding to see how quickly students adapted to the technology and optimized it, and how this led to essays produced more efficiently, and in many cases, of higher quality than they might have otherwise achieved.
Interested in learning more? Consider reading our further resources on AI in education.