ChatGPT: Brazilian colleges prepare to adapt test format; You understand

Tavares copied and pasted the questions into a ChatGPT box, typed the answers and then placed them on a platform that estimates the score using Item Response Theory (IRT). “I found the system’s performance impressive on the humanities and language tests. I think the good performance occurred because more often than not the answer to the question was in the statement itself, or it’s a paraphrase of the same statement,” he says.

“I don’t think this is a flaw in the enemy, because it is designed to evaluate people. When we talk about human thinking, paraphrasing means understanding what has been read, building a mental model of it, and then using other words to describe the same thing. In ChatGPT’s mathematical operation, it is found The answer is by checking which words are most likely to be part of the same group of words as in speech, which is quite different,” he adds.

“On a math test, it becomes very clear: ChatGPT is incapable of doing algebraic syllables, even the simplest ones. The equivalent to this, on a humanities test, is that ChatGPT tends to make mistakes when the answer depends on understanding a broader context than the context explicitly described in speech.” , he says.

Obviously, the man-made technology has limitations, which ChatGPT isn’t shy about admitting (there are a number of caveats that “the system may occasionally generate incorrect or misleading information and produce offensive or biased content”). “He is not online, so he is not able to talk about the issues that are happening today, and his knowledge base has a strict deadline, which is 2021,” says Carlos Rafael.

In New York, some schools have been completely categorical and ChatGPT has been banned from use. However, the professors Estadão interviewed don’t think this is the way to tackle the problem and prefer a less hard-line approach. It is the road of no return. Technology is developing. The only constant is change. I talk a lot about the Darwinian principle: only the strongest survive. So we have to think about this course of action,” says Flávio Marques Azevedo, coordinator of the Information Systems course at ESPM.

Ignoring new technology can also mean missing out on an opportunity to improve teaching and learning. “I recently read an article where a professor at a university in Ecuador did an experiment, not using ChatGPT, but the university created an app that was a bot, like Siri or Alexa, that was used as a tutor for a student. They divided the class into two groups. Half of the students didn’t use the bot and the other half Use it Marcos Vaco, director of communications and marketing at FGV, said the combination I used during the program had 20% higher performance.

In this sense, the researchers interviewed Estadao List a number of benefits of technology, for both students and teachers, such as: marking tests; work as a “teacher” or “assistant teacher”; help with boring tasks (converting an Excel spreadsheet to PDF, for example); simplify search and write processes; Refine generating bibliographies and helping to think of questions for tests. They also point out risks, such as a lack of accuracy, not quite right answers, and the presence of payout biases. After all, AI is trained by humans (the platform itself assumes this, although it strives for it). reflect “prejudice” and “inequality”, with small social groups underrepresented from their sources).

On accuracy, ESPM’s Carlos Rafael commented that in the tests he conducted, he noticed that for simple questions he had a success rate of 100%, but when the complexity is medium, the story is already different. “It does something very, very dangerous,” he says, “and gives an answer that’s not wrong, but it’s not 100 percent correct.”

“I ran a test where I asked ChatGPT to determine whether a geometry intersects another. There are three possible cases: the two geometries that don’t have an intersection; I got the code right in 2/3 of the cases. It speaks when it is not and when it collides, but not when it is inside. That is, if the student takes this code and performs a superficial test, without effort, at some point he will make a mistake “, Add.

“The biggest danger is that people will completely trust what he offers. There must be confirmation,” Facco sums up.

The danger that is perhaps the cause of the greatest panic in the United States is related to academic ethics and plagiarism on the part of students. Notably, this is a problem that predates the technology in question and relates to behavior and the way students see their role in education.

However, the catch is that it is much more difficult to detect the version when using ChatGPT. “As far as I know, spoofing programs don’t catch ChatGTP,” adds Carlos Afonso Souza, professor at Uerj and director of the Institute of Technology and Society (ITS Rio), who tested the texts prepared by the AI ​​on the platform it usually uses.

However, the teachers interviewed before Estadao He doesn’t seem afraid of introducing technology. They consider that more “content-oriented” and “just explain” disciplines will suffer the most, but point out that it is indeed possible to circumvent plagiarism with ChatGPT. In a more practical way, they point to the need for professors to become aware of the technology and its potential, and to think about assessments that require more than content, with an emphasis on problem-solving and analytical and critical comparison. On the other hand, they also assume the possibility of resuming oral assessment activities (directly in class or through video recording), in addition to essays.

Souza embodies an activity he usually imparts to students in law classes: the analysis of jurisprudence. “I ask them to find four judicial decisions on a topic and the job is to summarize the case and at the end compare the decisions,” he says. The professor decided to test how ChatGPT interacts with this task: he gave four decisions and asked to compare them.

“He does not make the comparison as if he were a graduate student, but what he does is interesting, because he always gives an introductory text on the subject with certain concepts, with certain definitions that are generally correct, which is a failure on the analytical part,” he says.

I joked on Twitter that Messages at Home has been turned off, but then it won’t necessarily be off. Those who would simply ask you to elaborate on a concept, to sum up a court decision, that’s over, Chat does. So you have to go further, you need to foster a critical, comparative, and analytical viewpoint in the student, preferably one that is not found on the Internet,” he summarizes.