#research-methodology

#research-methodology

AI is not just another research tool; it is redefining what research is, how it is done and what counts as an original contribution. Universities are mostly unprepared for the scale of disruption, with few having comprehensive governance strategies. Many academics remain focused on the failings of early generative AI tools, such as hallucinations (confidently stated but false information), inconsistencies and superficial responses. But AI models that were clumsy in 2023 are becoming increasingly fluent and accurate.

Its eclectic nature. I am trained in a style of research that places value in the faithful documentation and analysis of the actual practices of experts in certain technical fields. Sometimes this line of work runs the risk of being too removed from the immediate advancement of such fields. (The rather polemical analogy is "as useful as ornithology to birds," but I'd like to think of it as a kind of ethnography of certain specialized expertise.)

Researchers Deng et al. (2025) recently asked: Does ChatGPT enhance student learning? They conducted a meta-analysis including 69 studies published from 2022 to 2024. Looking at the overall effects across all studies, they concluded that ChatGPT enhances academic performance, affective-emotional states, and higher-order thinking propensities, while also reducing mental effort. However, Weidlich et al. (2025) identified several flaws in the meta-analysis.

In comparing the replication data analysis to the original study, we noted a significant decrease in effectiveness, from close to 60% to nearly 40%, indicating potential variability in treatment outcomes.

The grounded theory methodology is suitable for building theory in fields where little is known, making it ideal for investigating AB testing in BPI.

To enhance construct validity, we implemented strategies such as pilot experiments for data labelling agreements and consensus involvement to mitigate personal bias.

In evaluating LLM-generated mutations, we designed metrics that encompass cost, usability, and behavior, recognizing that higher mutation scores don't guarantee higher quality.