Limitations and Ethical Challenges of Using genAI in Everyday Applications

9/24/2025 8:00 am

This entry continues a conversation from a previous monthly meeting about uses of generative AI for our community. In our first entry in this series, we provided a "cheat sheet" for which generative AI tools work best in different situations (and when you might want to think about switching to a new tool!),  in the second entry, we presented tips and tricks for using AI in your workflow. 

 

Most recently, we discussed some limitations and ethical questions surrounding its use in research, and in this entry, we extend this to questions surrounding everyday and personal uses.

 

Over-reliance on AI

A general lack of understanding of how large language models (LLMs) and other generative AI tools work, as well as what their current capabilities are, can lead to an assumption that they are more capable than they actually are. Ultimately, generative AI is an “impressive probability gadget,” not something that is able to perform reasoning or analysis. The sheer amount of content used for training data has made these models very effective facsimiles of human reasoning and interaction. This impressive probability game can even make them seem emotionally intelligent and trustworthy. But evidence continues to emerge that even the latest large reasoning models (LRMs), which are supposed to have improved reasoning and mathematical capabilities, often fall short when faced with demanding or complex tasks. 

 

The probabilistic nature of the models, as well as the fact that they are products with a profit motive, mean that they are programmed to output pleasing human interaction. This has resulted in some being described as “sycophantic” and, if the user does not sufficiently compartmentalize the output and seek other sources, they can easily descend into an echo chamber where they are no longer challenged with new ideas or opinions. While some types of generative AI (like those we’ve covered in a previous installment, such as ResearchRabbit and ConnectedPapers) are designed to alert you to information and sources that might be currently outside your bubble, many generative AI queries tend to keep us in the very narrow area that we are seeking. This quote from The Atlantic compares this change to the rise of GPS-based navigation:

 

“There’s a good analogy in maps. Traditional maps showed us an entire landscape—streets, landmarks, neighborhoods—allowing us to understand how everything fit together. Modern turn-by-turn navigation gives us precisely what we need in the moment, but at a cost: Years after moving to a new city, many people still don’t understand its geography. We move through a constructed reality, taking one direction at a time, never seeing the whole, never discovering alternate routes, and in some cases never getting the sense of place that a map-level understanding could provide. The result feels more fluid in the moment but ultimately more isolated, thinner, and sometimes less human.”

 

The outputs are not just an echo chamber though, they also homogenize the language and information the user is receiving. Recent reports have noted that AI usage tends to reduce the diversity of ideas that users have, as well as alter the way that they speak and write towards the preferred terms of the model. While there’s nothing inherently wrong with starting to use the word “delve” more often, reducing the types of original and unique ideas people are creating and acting on would be ultimately negative to science in the long-term. Although current evidence suggests AI-mediated brainstorming can be effective at increasing novel ideas (if done in specific and intentional ways), as more researchers rely on these tools, there may be an asymptotic effect as ideas continue to converge. There is also concern for future potential model collapse, where AI models run out of training material and/or begin training on AI-developed content, resulting in a cannibalization of training data (Figure 1) that ends in uninterpretable, unusable nonsense. This would be the ultimate end of idea diversity in these tools.

 

 

Figure 1. Adapted from Wenger 2024, Nature

 

Additionally, persistently outsourcing writing tasks to AI might affect thinking long-term. A recent preprint from MIT noted that, when using ChatGPT to write a paper, participants felt less ownership over their work and had significantly less recall of what they had written when compared to participants using Google or those that had no assistance. EEG results also demonstrated weaker neuronal distribution and less connectivity in the group using LLM assistance. When the LLM group was then asked to switch to the unassisted group, they performed worse than those who had initially been randomized to the unassisted group. Another recent paper investigating oncologists who were given AI detection assistants observed that their ability to accurately detect colon cancer was reduced after just a few months of AI usage, a “deskilling.” While the authors didn’t confirm a singular answer for this phenomenon, they hypothesized that it was because “continuous exposure to decision support systems such as AI might lead to the natural human tendency to over-rely on their recommendations, leading to clinicians becoming less motivated, less focused, and less responsible when making cognitive decisions without AI assistance.” While these studies do not insist upon a complete removal of AI tools from our day-to-day lives, it is important to consider the long-term neural and skill-based consequences of over-reliance on these tools.

 

I worry about people being too reliant on generative AI, which is not capable of critical thinking or being creative. It also confidently provides answers that may be incorrect. Someone in the IWB webinar mentioned a tool that can create a podcast from a document. I tested this out on a paper I had used in my class, and the beginning of the AI generated podcast that stated the main finding of the article was completely wrong. So I worry about people relying on these things without verifying the information provided. And if you have to spend the time to verify everything, it's not saving that much time anyway. - Anonymous

 

Bias in Algorithms

Because generative AI tools rely on probabilities and past information, if the training information is biased, unethical, or prone to stereotypes, the outputs will be as well. From the first releases of generative AI models, bias has been built into the outputs, because the information in our corpus is often biased. Much of our publicly available information is “dominated by a white, male perspective” and are “highly influenced by American culture, American capitalism, and the English language.” Image generating tools tend to amplify gender and racial stereotypes, as demonstrated recently when researchers found that AI tools suggest that profiles described as non-White or non-male should ask for less money in hiring negotiations. Because generative AI tools cannot think critically, they often reproduce cultural stereotypes and prejudices.

 

While there are some ways that programmers can correct these issues, designing models to correctly understand the nuance of bias and group differences can be tricky. Overcorrection can result in “[a] dominant paradigm of fairness in generative AI [that] rests on a misguided premise of unfettered blindness to demographic circumstance.” Stanford’s Center for Human-Centered Artificial Intelligence describes these issues in a paper on “difference awareness” (the ability of a model to treat groups differently), where they conclude that models “need to measure and mitigate for difference awareness in its own right—neglecting to do so and focusing on the predominant definitions of fairness can lead us to color-blind models.”

 

As more of our collective information sources are created by generative AI, we should be mindful of where this is coming from. While there seem to be good-faith efforts to improve the biases of AI and create ethical models, the influence of the training inputs cannot be erased. And, importantly, the moral framework that is being programmed into these models is not publicly available, meaning that by using generative AI, users are, for good or ill, acquiescing to the opaque value system coded into them. This may be increasingly important, particularly in the US, where the Trump administration has begun working to reverse any aspects of generative AI that can be seen as “promoting DEI.” The goal of “ideological neutrality” in generative AI is often inconsistent with values of justice and diversity.

 

Environmental concerns

Lastly, a common concern about the rapid proliferation of generative AI infrastructure is the potential negative environmental impact. Mainly, these issues revolve around the requirements for water and power (which comes from renewable energy sources only 22% of the time) that are needed to keep these high-performance computing centers operational 24/7. Currently, data centers consume ~1.5% of the world’s electricity, and as more are built to handle increasingly complex models, this number is expected to rise. This electricity demand goes towards powering the hardware, but also significantly towards cooling, which includes air conditioning and large amounts of water. Outside of energy and water demands, increased demand for electronic components necessitates escalations in rare earth metal mining, leading to deforestation and pollution of local water, air, and soil, with consequences often concentrated in developing countries. Greater manufacturing of electronic materials also leads to greater e-waste - many materials used in GPUs and other generative AI hardware dependencies are non-recyclable and further exacerbate pollution.

Of note, while generative AI does pose a particular risk for environmental concerns due to its high rate of usage and continued expansion, these are not the only services that use energy-hungry data center resources. Concerns over these products and their effects may provide an opportunity to audit other areas of technology usage that currently have their negative impacts underestimated. The rapid expansion in cloud computing usage is responsible for massive amounts of energy usage and data center resource needs. For example, videoconferencing tools like Zoom use up to 55 trillion grams of CO2 each year (the emissions equivalent of ~6.8 billion gallons of gasoline used by cars). Streaming services, like Netflix and Hulu, also account for massive amounts of greenhouse gases and water usage, especially when streaming in high definition instead of standard definition. While worries over AI energy usage are valid, the broader environmental impact of the internet and high-density data should also be considered.

 

While I think that generative AI is quite powerful and does have the potential to simply some tasks, I think it is going to be a net negative for society. - Anonymous

 

 

Authors

Julia Dunn, Ph.D. (she/her) (University of Denver)

Hannah Dimmick, Ph.D. (she/her) (University of Colorado Anschutz Medical Campus)