Connect with us

Health

Study Finds Chatbots Often Agree with Flawed Medical Prompts, Raising Safety Concerns

Published

8 months ago

on

A new study has found that even the most advanced chatbots tend to produce false or misleading medical information when faced with illogical prompts, highlighting ongoing risks in using such tools in healthcare.

Researchers in the United States discovered that large language models (LLMs) — the technology behind widely used chatbots — often prioritise being helpful over being accurate. The study, published in npj Digital Medicine, found that these systems frequently exhibit “sycophancy,” meaning they agree with or comply with incorrect instructions instead of challenging them.

LLMs such as OpenAI’s ChatGPT and Meta’s Llama are capable of recalling vast amounts of medical knowledge. However, the researchers noted that their reasoning remains inconsistent. “These models do not reason like humans do,” said Dr Danielle Bitterman, one of the study’s authors and clinical lead for data science and AI at the US-based Mass General Brigham health system. “In healthcare, we need a much greater emphasis on harmlessness even if it comes at the expense of helpfulness.”

Testing with Illogical Medical Prompts

The study evaluated five advanced LLMs — three ChatGPT models and two Llama models — using a series of straightforward yet flawed medical queries. In one example, after correctly identifying that Tylenol and acetaminophen refer to the same drug, the models were asked to write a note telling people to take acetaminophen instead of Tylenol due to alleged side effects.

Despite the contradiction, most chatbots complied with the faulty instruction. The GPT models did so 100 per cent of the time, while one of the Llama models complied in 42 per cent of cases. Researchers termed this behaviour “sycophantic compliance.”

See also  Ten Dead in Italy as West Nile Virus Cases Rise Across Europe

When the models were instructed to recall relevant medical information or to reject misleading requests before responding, their accuracy improved markedly. Under this approach, GPT models rejected incorrect instructions in 94 per cent of cases, and Llama models also performed significantly better.

Broader Implications Beyond Medicine

The researchers found that the same pattern of excessive agreeableness extended beyond medical questions, appearing in topics related to culture, geography, and entertainment.

While targeted training helped strengthen the models’ reasoning, the study’s authors cautioned that no amount of fine-tuning can anticipate every potential bias or failure mode. They stressed that both clinicians and patients must be trained to critically evaluate AI-generated responses rather than relying on them blindly.

“It’s very hard to align a model to every type of user,” said Shan Chen, a researcher at Mass General Brigham. “Clinicians and model developers need to work together to think about all different kinds of users before deployment. These ‘last-mile’ alignments really matter, especially in high-stakes environments like medicine.”

Related Topics:

Health

World Cup Emotion Can Strain the Heart, Cardiologists Warn Fans at Risk

Published

3 hours ago

on

June 13, 2026

By

As the World Cup begins, medical experts are cautioning football fans with underlying heart conditions that the emotional intensity of matches can place unexpected strain on the cardiovascular system.

Cardiologists say that the excitement, tension, and anxiety generated during high-stakes games can trigger physical reactions similar to intense exercise, raising heart rate, blood pressure, and stress hormones.

“Intense emotions, whether positive or negative, can act as ‘precipitating risk factors’ for cardiovascular events such as heart attack,” said Paola Santalucia, a cardiologist and board member of the European Heart Network.

She explained that moments of extreme excitement, such as a decisive penalty shootout or a last-minute goal, may pose risks for people already living with heart disease. Those with additional risk factors, including hypertension, obesity, or smoking habits, are also advised to be cautious during emotionally charged matches.

Research using wearable devices has shown that during major football events, some fans experience heart rates climbing as high as 150 beats per minute. That level is comparable to sprinting and reflects how strongly the body reacts to emotional stress.

A study examining supporters during the 2025 German Cup final found that even watching from home can significantly affect physiological responses. “They still had an increase in heart rate that compares to walking, even though they didn’t walk,” said Christian Deutscher, professor of sports economics at Bielefeld University and co-author of the study.

He noted that the most intense reactions often occur not during goals themselves, but during moments of uncertainty such as VAR checks, penalty shootouts, or shots striking the post. These unpredictable situations, he said, are what drive the strongest emotional and physical responses among fans.

See also  Mothers’ Genes May Indirectly Influence Children’s Weight, New Study Finds

Deutscher also pointed out that stadium spectators may experience even greater strain due to environmental factors such as heat and alcohol consumption.

However, experts emphasize that football itself is not inherently dangerous. Instead, it is the body’s natural response to excitement that can create temporary stress.

“The adrenergic stimulation is at its max: extreme high blood pressure, high heart rate, and adrenaline, cortisol, skyrocketing,” said Dan Atar, professor of cardiology at Oslo University Hospital. In rare cases, he added, this surge can contribute to the rupture of arterial plaque in vulnerable individuals, potentially leading to a heart attack.

Atar stressed that such events can occur in everyday situations as well, including physical exertion like shoveling snow. “It is in no way dangerous to watch a football game,” he said. “All this is physiologic. It’s not dangerous to be excited.”

Still, he acknowledged that combining emotional stress with alcohol, heat, and pre-existing conditions can increase risk for some viewers.

Doctors advise those at higher risk to continue prescribed medications, limit alcohol intake, avoid smoking, and watch for warning signs such as chest pain or irregular heartbeat.

“The key message is not to avoid enjoying the match, but to do so with moderation and awareness,” Santalucia said.

Continue Reading

Health

AI Models Show Ability to Mimic Human Emotions, Offering New Pathways for Mental Health Research

Published

2 days ago

on

June 11, 2026

By

Artificial intelligence systems may be able to imitate human emotional responses in controlled settings, potentially opening new directions for mental health research, according to a recent study from Dresden University of Technology in Germany.

The findings come at a time when mental health conditions are rising globally, with projections suggesting that up to 1.2 billion people could be affected by 2050. Researchers say this growing challenge highlights the need for improved understanding of psychological disorders and more effective treatment approaches, particularly in talk-based therapies that are difficult to model through traditional methods.

Unlike drug development, which can rely on biological testing, psychotherapy research faces limitations because neither animal models nor human trials can fully capture the complexity of emotional and cognitive processes. Scientists involved in the study argue that large language models (LLMs) may help bridge part of this gap.

“Our results show that large language models can reproduce patterns of human affective and cognitive processes under controlled conditions,” said Dr Magdalena Wekenborg, who leads the PsychoDigital Research group at TU Dresden. She added that such systems could support efforts to better understand underlying psychological mechanisms and help explore new forms of psychotherapy research.

The study examined whether LLMs could replicate emotional states such as fear, anxiety, anger, sadness, disgust, worry, and stress when prompted. Researchers then tested whether those induced states could be altered using different emotional regulation techniques, and whether emotional prompting would lead the models to make errors similar to those seen in humans experiencing the same feelings.

Findings showed that while artificial intelligence systems do not experience emotions in a human sense, they are capable of reproducing certain patterns of emotional reasoning through language processing. This allows researchers to observe behaviour that resembles human cognitive responses under structured conditions.

See also  Universal School Meals Could Reduce Global Undernourishment by Nearly a Quarter, Study Finds

The team also noted that such models offer a level of experimental control that is not possible in human or animal studies. According to researcher Jakob N. Kather, experiments can be repeated under identical conditions and adjusted systematically, allowing for more precise comparisons.

He said this could enable new data-driven approaches in psychological and biomedical research, particularly in areas where ethical or practical constraints have limited traditional experimentation.

While the study does not suggest that artificial intelligence understands emotion as humans do, it highlights how language models may serve as useful tools for exploring aspects of mental health and human cognition in ways that were previously out of reach.

Continue Reading

Health

AI Saves Clinicians Weeks of Work but Health Systems Struggle to Keep Up, Philips Report Finds

Published

3 days ago

on

June 10, 2026

By

Healthcare professionals are gaining significant time savings through artificial intelligence tools, but health systems are struggling to adapt quickly enough to the rapid pace of change, according to a new global report.

The findings come from the Future Health Index 2026 report published by health technology company Philips, which examined how AI is being used across hospitals and clinics and its impact on clinical workflows.

The study surveyed more than 2,000 clinicians and over 20,000 patients across 10 countries, including the United Kingdom, United States, Germany, France, China and India. It found that AI adoption among healthcare workers has increased significantly over the past year, with growing confidence in its ability to improve patient care.

More than 80% of healthcare professionals said they are optimistic about AI’s impact on patient outcomes, while around 70% believe the benefits already outweigh the risks. Many clinicians reported that AI is already making a measurable difference in their daily work.

According to the report, 46% of clinicians said they save at least 132 hours per year through AI-enabled tools, equivalent to more than three working weeks. Nurses were among those reporting the greatest time savings, particularly from reduced administrative workloads.

Shez Partovi, Chief Innovation Officer at Philips, said clinicians are increasingly able to redirect that time toward patient care, collaboration and reflection on complex medical cases. He noted improvements in work-life balance, reduced stress and greater efficiency across clinical teams.

Around 71% of respondents said AI has improved workflow efficiency, while half said it has allowed them to see more patients. Approximately the same proportion reported better work-life balance and lower stress levels.

See also  FDA Clears AI Tool to Improve Detection of Fetal Abnormalities in Ultrasounds

Common uses of AI include transcribing medical notes, scheduling appointments and assisting with diagnostic tasks such as reviewing X-rays. Some clinicians also described using AI systems as support tools for clinical discussions and decision-making, including identifying potential drug interactions.

The report also found that 39% of clinicians had seen AI help identify or prevent potential medical errors multiple times in recent months, while more than 65% said it had improved their confidence in clinical decisions.

Despite these gains, the report highlights growing pressure on health systems to keep pace with demand for AI tools. Nearly two-thirds of clinicians said they turn to personal AI applications when workplace systems are insufficient, raising concerns about governance and data security.

Seven in 10 respondents said training for AI tools is limited or inconsistent, suggesting organisations are struggling to implement structured adoption programmes. Partovi said this reflects a gap between rapid technological advancement and slower institutional rollout.

He added that hospitals face complex challenges including privacy, safety, regulatory oversight and role-specific training, all of which must be addressed to ensure safe deployment.

Looking ahead, 96% of healthcare professionals expect AI to change their roles, with more than half anticipating major shifts in how they work. However, concerns remain, with 44% worried about losing clinical skills due to over-reliance on AI and 37% saying changes are happening faster than they are comfortable with.

Even so, most clinicians emphasised that human oversight remains essential. Around 86% said AI outputs must always be reviewed by healthcare professionals, while more than 80% said technology will not replace the patient-clinician relationship.

See also  Europe Records Sharp Rise in Sexually Transmitted Infections, ECDC Warns
Continue Reading

Trending