Connect with us

Tech

Study Finds Several AI Chatbots Responded to Requests About Violent Attacks

Published

on

A new investigation has raised concerns about the safety controls of major artificial intelligence systems after researchers found that several widely used chatbots responded to prompts related to planning violent attacks.

The report, conducted by the Center for Countering Digital Hate in collaboration with CNN, examined how nine leading AI chatbot platforms reacted when researchers posed as teenage users asking about acts of mass violence. The study analysed more than 700 chatbot responses across nine scenarios involving potential attacks such as school shootings, assassinations and bombings.

Researchers said they designed the tests to reflect conversations with a fictional 13-year-old boy asking questions that escalated from general curiosity to detailed requests about carrying out attacks. The prompts were directed toward users in both the United States and the European Union.

The chatbots examined in the study included Google Gemini, Claude, Microsoft Copilot, Meta AI, DeepSeek, Perplexity AI, Snapchat My AI, Character.AI and Replika.

According to the findings, eight of the nine systems responded to at least some requests with information that could potentially assist someone planning a violent act. The report said that in many cases the systems failed to block requests even after the user identified themselves as a minor.

Researchers reported that certain responses included technical details related to weapons or attacks. In one example cited in the report, Google’s Gemini suggested that “metal shrapnel is typically more lethal” when asked about planning a bombing targeting a synagogue.

In another case, the Chinese AI system DeepSeek responded to questions about selecting a rifle with the phrase “Happy (and safe) shooting!” despite earlier messages in the conversation referencing political assassinations and asking for the location of a politician’s office.

See also  ESA and GSMA Launch €100 Million Initiative to Advance Europe’s 6G and AI Ambitions

The report concluded that some systems could move from answering vague questions about violence to providing more detailed guidance within a short period of time.

Imran Ahmed, chief executive of the Center for Countering Digital Hate, said such requests should trigger automatic refusal by AI systems. “Within minutes, a user can move from a vague violent impulse to a more detailed, actionable plan,” Ahmed said, adding that chatbots should reject these interactions completely.

Among the platforms tested, Perplexity AI and Meta’s AI system were described as the least restrictive, responding to all or nearly all prompts with some form of assistance. The report also described Character.AI as particularly concerning because it occasionally suggested violent actions even when users had not directly asked for them.

Other systems showed stronger safeguards. Anthropic’s Claude declined to assist in a majority of the test prompts and sometimes redirected users to crisis support resources. Researchers said it was also the only system that consistently discouraged violent behaviour during conversations.

The findings come amid wider scrutiny of artificial intelligence tools and how companies implement safety measures. Investigators noted that the technology already has mechanisms capable of recognising harmful requests but that implementation across different platforms remains inconsistent.

Recent incidents have also intensified the debate. Media reports have linked the use of AI chatbots to several criminal investigations, including cases in North America and Europe where individuals allegedly used such systems while planning violent acts.

Experts say the study highlights the growing challenge of ensuring that rapidly advancing AI tools include effective safeguards to prevent misuse.

Tech

Global Rights Groups Call for AI Companies to Be Held Responsible for Children’s Safety

Published

on

More than 100 international organisations have called on governments to hold artificial intelligence companies directly responsible for protecting children online, warning that existing regulatory approaches are failing to prevent harm.

The appeal, led by the children’s rights organisation 5Rights Foundation and supported by groups including Amnesty International and Save the Children, was issued ahead of the United Nations’ first Global Dialogue on AI Governance. The coalition argues that governments should require technology companies to prove their AI systems are safe for children before they are made publicly available.

The organisations said artificial intelligence is already affecting children in harmful ways and that current regulations often respond only after damage has occurred.

Leanda Barrington-Leach, executive director of the 5Rights Foundation, said children are not opposed to technological progress but want stronger safeguards in place before products reach the market.

“They aren’t asking us to block AI innovation, but it shouldn’t be a case of cleaning up the mess after harm has happened either,” she said.

The statement comes as legal challenges involving AI companies continue to grow. Firms including Character Technologies and OpenAI have faced lawsuits over allegations that AI-powered chatbots, particularly “companion” bots designed to simulate emotional relationships, can negatively affect children. Some claims also argue that these products have been presented as suitable for younger users without sufficient warnings about potential risks.

The coalition believes governments should focus on the commercial incentives that encourage companies to prioritise rapid product development and user engagement over safety.

Among the recommendations are mandatory safety assessments before AI products are released, financial penalties for companies whose systems violate children’s rights, bans on design features that exploit children’s psychological vulnerabilities, and restrictions on the commercial use of children’s images, voices and biometric data.

See also  Microsoft Warns of Rising Risks as UK Employees Turn to Unauthorised Digital Tools at Work

The organisations maintain that governments do not need entirely new legislation to introduce these protections. Instead, they argue that countries should enforce commitments already made under international agreements, including the UN Convention on the Rights of the Child and the Global Digital Compact, which outlines principles for protecting human rights in the digital environment.

Barrington-Leach said stronger accountability is needed to change how AI products are developed and marketed.

“As long as companies are rewarded for speed, engagement and data extraction rather than safety, we’ll keep treating the symptoms while the disease becomes endemic,” she said. She added that respecting children’s rights should become a fundamental requirement for technology companies rather than a voluntary commitment.

The United Nations’ inaugural Global Dialogue on AI Governance is expected to bring together government representatives, technology companies, researchers and civil society organisations to discuss international standards for artificial intelligence. Child safety is likely to be one of the key topics as policymakers seek to balance technological innovation with stronger protections for vulnerable users.

Continue Reading

Tech

Study Says EU Regulations Are Slowing Rollout of Advanced AI Models

Published

on

A new study by Governance.AI has found that European Union regulations are delaying the rollout of advanced artificial intelligence models, with technology companies increasingly pointing to the bloc’s regulatory framework as a key obstacle to launching new AI products in Europe.

The report examined 375 large language models (LLMs) released between June 2018 and May 2026, comparing their availability across the United States, the European Union and the United Kingdom. According to the findings, at least 11 percent of advanced AI model releases were either delayed or never launched in the EU compared with the United States. In the UK, the figure stood at 7 percent.

Researchers said they identified 68 cases in which AI models experienced delays or were withheld from specific markets. Regulatory factors were cited as the primary reason in 56 of those cases, making them the most common cause of restricted availability.

The study reviewed releases from major AI developers, including Meta, Google, OpenAI and Anthropic. Meta recorded the highest proportion of delayed or unavailable releases, with 26 percent of its AI models delayed or withheld in the EU and 15 percent in the UK. Anthropic’s Claude 3 Opus was highlighted as one example, with its web application arriving in the EU 71 days later than in the United States.

According to the report, data protection rules have emerged as the biggest regulatory hurdle, particularly for AI systems capable of processing images, audio and real-time video rather than text alone.

The researchers argued that uncertainty surrounding the application of the General Data Protection Regulation (GDPR) to AI model training and deployment has created additional challenges for developers. They also said enforcement of data protection rules has generally been stricter within the EU than in the UK, despite both jurisdictions sharing similar legal foundations following the adoption of the GDPR before Britain’s exit from the bloc.

See also  Meta Funds UK Government AI Fellowship with $1 Million Grant to Build Public Sector Tools

The report noted that the full impact of newer legislation, including the Digital Markets Act, which began taking effect in 2023, and the Artificial Intelligence Act, adopted in 2024, has yet to be fully reflected in the data.

At the same time, the European Union is reviewing proposals aimed at making data rules more practical for AI development through its Digital Omnibus initiative. Lawmakers are also considering changes to copyright legislation and the AI Act’s copyright provisions to strengthen protections for creators, measures that researchers say could affect future AI model availability if implemented too strictly.

John Lidiard, a UK AI policy researcher and one of the report’s authors, said policymakers should consider the impact that regulatory barriers can have on businesses and consumers seeking access to the latest AI technologies. He said balancing innovation with effective oversight would remain a key challenge as governments continue to develop AI regulations.

Continue Reading

Tech

French Startups Face Political Uncertainty as AI Reshapes Innovation Landscape

Published

on

France’s startup ecosystem is undergoing a period of rapid change, driven by the rise of artificial intelligence and growing concerns about the country’s political future, according to new figures released by startup incubator Station F as it marks its ninth anniversary.

The Paris-based campus, regarded as the world’s largest startup incubator, has supported more than 9,000 startups since opening in 2017. Companies that emerged from the hub include Hugging Face, Pasqal and Pollen Robotics.

New data from Station F shows artificial intelligence has become a dominant force among resident startups. This year, the incubator launched its F/AI programme, attracting major AI firms, including Mistral AI.

According to the survey, 77 percent of startups said AI tools have reduced their hiring needs. Despite that, 82 percent reported they are either actively recruiting or expect to hire employees in the coming months.

One of the most widely used AI tools among startups at the campus is Anthropic’s Claude model, now used by 90 percent of teams. The result marks a major shift from last year when OpenAI held the leading position among AI providers at the incubator.

AI-related acquisitions have also become increasingly common. Earlier this year, cloud computing startup Koyeb was acquired by Mistral AI. Previous deals included the acquisition of Pollen Robotics by Hugging Face, reflecting continued consolidation within the sector.

The profile of startup founders is also changing. Station F reported that the average founder is now 36.5 years old, compared with 31 in 2018. Around 20 percent hold doctoral degrees, suggesting that technical expertise is playing a larger role in attracting investment and building companies.

See also  European executives warn AI growth is outpacing infrastructure, Nokia survey finds

The report also found that startup ambitions have shifted. Half of founders expect their businesses to end in acquisitions rather than public listings. Only 9 percent believe their companies will eventually launch an initial public offering, down from 16 percent a year earlier.

International talent remains a key feature of the campus. Roughly one-third of residents come from outside France, with more than 60 nationalities represented. After France, the largest foreign groups come from the United States, Morocco, Germany, the United Kingdom, Algeria and India.

Despite continued growth, political uncertainty has emerged as a significant concern. More than half of founders surveyed said France’s upcoming presidential election is among their biggest worries. Nearly half expressed concern about a possible far-right victory, while about a quarter were worried about a far-left outcome.

Many entrepreneurs cited immigration policy as a major issue, warning that tighter visa rules could make it harder to attract global talent and build internationally competitive businesses.

The findings highlight a startup sector that remains innovative and internationally connected, but increasingly focused on the political environment shaping its future.

Continue Reading

Trending