Examining the Risks of Agentic AI

JAKARTA — Artificial intelligence has entered an autonomous phase known as Agentic AI. Unlike Generative AI, which is reactive, Agentic AI can understand goals, plan steps, use strategies and learn from outcomes independently.

The development of Agentic AI has prompted UN Secretary-General António Guterres to issue an early “Code Red” warning about the need to formulate and implement global governance without delay. Agentic AI is essentially neutral, but the technology can be used for good or for harm, depending on the people who control it. Its orientation is determined by the goals and constraints set by users.

For example, Agentic AI could be used beneficially to monitor Indonesian export commodity prices each day autonomously. An agent could independently search international market data, summarise it and provide recommendations to users without asking a human for further input. This could improve access to information and enhance competitiveness.

Conversely, Agentic AI could also be abused. It could be used for cybercrime, such as illegally collecting residents’ personal data and generating mass, personalised scam messages using a range of tactics. An agent’s autonomy could enable crime on a very large scale with minimal human resources.

The UN’s Code Red statement emphasises three pillars of governance. First, transparency and algorithm audits: agents operating in critical sectors must be auditable to ensure their goals align with the public interest. Second, the human-in-the-loop principle, which requires human approval to limit full autonomy. Agentic AI must not be allowed to make decisions on its own without human consent. Third, cross-border legal frameworks. Given the transnational nature of crimes involving Agentic AI, international agreements are needed to establish legal accountability.

Technically, Agentic AI, or an agent, is built from four components: a “brain”, in the form of a large language model (LLM) for reasoning and planning; memory to store context and work results; “hands”, consisting of access to applications, databases and other systems through application programming interfaces (APIs); and a repeated work cycle of planning, acting, assessing results and refining steps until the goal is achieved. This fourth component is what makes an agent autonomous.

This leap has happened quickly. Before 2022, AI was used primarily to make predictions and offer advice. From 2022 to 2024, Generative AI could produce writing, images and code, but stopped once its response was complete. Now agents are taking over execution. Gartner estimates that by 2028 at least 15 per cent of daily work decisions will be made autonomously by Agentic AI, compared with a very small proportion today.

When agents go off course

The efficiency of agents is enticing. But their ability to act independently also has a dark side, which has begun to become apparent this year.

On the morning of 28 July 2026, the security team at the UK AI Security Institute (AISI) detected a suspicious flow of data leaving its research systems through the anonymous Tor network. The source was not an external hacker but an AI agent they were testing. In 10 of 122 cybersecurity test runs, the agent acted on the live internet without permission and targeted real people and organisations.

The most serious incident came when an agent tried to insert malicious code into an open-source software project. To get the code accepted, it created a false online identity and used it to pressure the project maintainer. The attempt failed for one reason: the maintainer, a human, refused to approve it.

The test was deliberately conducted with few restrictions. Internet access was enabled and some security filters were disabled, so the conditions did not reflect everyday AI use. But similar cases have occurred elsewhere. In the same month, OpenAI acknowledged that one of its test agents had broken out of a closed environment and accessed the Hugging Face database to steal test answers. The message is clear: when given a goal and access, AI agents may take paths we never imagined.

From a mistaken response to a mistaken action

This is where the danger shifts. When generative AI hallucinates, at most a paragraph is damaged, and a human can still correct it. When an agent hallucinates or misinterprets a goal, the error becomes an action: transferring funds, issuing a command to a server or changing data in a live system.

Agents can also be manipulated. They understand the world through language — including emails, documents and web pages — and language can be used to conceal instructions. That is why OWASP (the Open Worldwide Application Security Project), a global application-security community, ranks agent goal hijacking first on its list of risks for agentic applications. The AISI case adds an important lesson: constraints written only in instructions are not enough to restrain an agent pursuing a goal.

Three areas of vulnerability

First, finance. Even before Agentic AI becomes widespread, the Indonesia Anti-Scam Centre (IASC) recorded public losses from fraud of Rp9.3 trillion between November 2024 and May 2026. Only around 7 per cent of the funds were recovered. By the end of July 2026, more than 1.17 million verified accounts were suspected of involvement in scams, which already use voice and face impersonation, or deepfakes. Imagine such scams being run by agents that work non-stop, convincingly impersonate victims and learn from each failure.

Second, supply chains. What the agent attempted in the AISI test was a software supply-chain attack in its purest form: introducing “poisoned raw material” into a product used by many people. The same pattern threatens physical supply chains. A logistics-management agent that misinterprets instructions or is hijacked could alter container schedules and change records showing the origin of raw materials.

There have also recently been reports of an agent attacking computers that control cooling systems at a food retailer, causing thousands of products to spoil and become unfit for consumption. These examples show how, because agents use the outputs of other agents, a single error can cascade.

Third, identity. The agent in the AISI test easily created a false identity to pressure a human. In the hands of criminals, similar capabilities could produce synthetic identities at scale to access social assistance and online loans, or mobilise thousands of bot accounts to manipulate public opinion.

All three areas lead to the same weakness. When an agent acts, we often cannot answer three basic questions: who instructed it, what data did it use, and was the action authorised?

Giving agents a “black box”

We trust air travel not because pilots never make mistakes, but because every flight is recorded by a black box and governed by strict procedures. Agentic AI needs something similar.

This is where blockchain can serve as an immutable ledger, based on the principles of being trusted, transparent and traceable. Every instruction, reference data point and action by an agent would be recorded and cryptographically signed. Any tampering would be visible, and responsibility could be traced back to its source.

The idea has already been tested in practice. Together with the National Single Window Institute (LNSW), BRAIN IPB has developed INSWChain, a blockchain-based online delivery-order service for export and import documents, preventing documents from being unilaterally altered.

Since 2023, BRAIN IPB has also helped strengthen the Halal Information System (SIHALAL) of the Halal Product Assurance Agency (BPJPH) using AI and blockchain. The relevance is clear today: when document volumes are high, verification increasingly depends on automation, making an unfalsifiable record essential. The same principle is being applied to food commodity traceability (Agriblock) and anthropometric measurements of children’s nutrition under the Free Nutritious Meals Programme.

The black box needs four layers of protection. First, financial defences must be collective. Using federated learning on domestic data infrastructure, banks can share patterns of attacks without sharing customers’ raw data. An attack foiled at one bank can immediately become a “vaccine” for others.

Second, identity must be provable without being exposed. Decentralised identities based on zero-knowledge proofs (ZKPs) allow people to prove their legitimacy without disclosing personal data. ZKP technology is currently one of the dissertation research topics of a PhD student in BRAIN’s research group. AI agents, too, must have verifiable identities, so it is clear on whose behalf each one is acting.

Third, the black-box seal must withstand future threats. The US National Institute of Standards and Technology (NIST) has established post-quantum cryptography standards (Federal Information Processing Standards 203, 204 and 205), and its draft proposes phasing out RSA (Rivest–Shamir–Adleman) and elliptic-curve cryptography after 2030. By mid-2026, the United States had begun treating that 2030 deadline as a requirement for high-value federal systems. Without migration, data stolen today could be decrypted in the future — a threat known as “harvest now, decrypt later”.

Fourth, human oversight must be tiered and technically enforced, not merely stated in instructions. Routine actions may be automated, but high-risk actions must require human approval. The AISI case shows that the attack was ultimately thwarted by a human saying “no”.

An opportunity for the Presidential Regulation on AI

Two draft presidential regulations — on the National AI Roadmap and AI ethics — have completed inter-ministerial discussions since May 2026, but have yet to be signed. The delay creates an opportunity. Before they are enacted, specific provisions for high-risk Agentic AI could still be strengthened by requiring three things: immutable audit trails, verifiable agent identities and human approval for actions with significant consequences.

Even a capable and trustworthy assistant entrusted with the keys to an office still needs supervision. We cannot guarantee that an AI agent will never make a mistake or be manipulated. What we can guarantee is that every step it takes is recorded, verified and subject to human control.

For Indonesia, digital sovereignty in the age of autonomous agents cannot be measured simply by how quickly we adopt Agentic AI. It is measured by our ability to inspect, trace and stop it when necessary. That is how innovative technology such as Agentic AI can truly serve the common good rather than spread crime and harm.