Navigating Agentic AI: Risk Management Strategies

Right from the start, checking standards should be part of the work. Most people do not realize how much delay - and cost - can be avoided by beginning sooner. Getting quality right early sets the foundation where projects tend to succeed better. Starting QA processes early helps spot problems that might grow into costly setbacks later on. Thinking ahead like this keeps quality checks tied closely to what the project aims to achieve, from the outset. When QA begins early, it often clarifies roles and fosters clearer dialogue across the group. Better collaboration follows, which tends to lead to stronger outcomes and happier customers down the line.

For the last decade or so, the fictional Artificial Intelligence has been like Siri. At least, that’s what I think it’s been like for those who haven’t taken a deep look into the inner workings of AI. We now have a type of artificial intelligence that can behave more like a character from science fiction, and is known as “agentic AI.” Agentic means that, unlike a simple assistant AI, this new kind of AI behaves as an agent, with a sense of purpose and free will, acting independently without our direct input and using the computer as its “body”. You set the goal for this agent and, instead of receiving step by step instructions as to how this goal will be achieved, you simply watch the computer as it finds solutions, one after another, more or less instantly. All of this amounts to having a virtual but personal assistant, fully able to pursue your wishes without you ever having to tell it how to accomplish its assigned task.

Example: Book a caterer for a future team offsite to accommodate three specific dietary restrictions and stay within budget. Smart software can then be used to research potential providers, request pricing and menus, compare options and even organize a food tasting. You may not hear a peep from the software until the task is actually done.

Why releasing a new, autonomous robot from the leash is so significant. In effect, a new AI system is never just a tool; it is a partner in a shared work, so developing a theory of AI safety is important. Releasing a tool into the world is a matter of a few minutes, but releasing a partner into the world is a lifetime of commitment. Releasing a tool into the world is an event, but releasing a partner into the world is a process of becoming, of learning, of adjusting, and of being adjusted. Releasing a tool into the world has consequences that are easy to predict and understand, but releasing a partner into the world has consequences that can only be guessed at. Releasing a tool into the world is deterministic, but releasing a partner into the world is probabilistic. Releasing a tool into the world does not depend on the partner releasing it into the world, but releasing a partner into the world depends upon the commitment of both partners in the work.

Why We Want a Digital 'Project Manager' for Our Lives

  Having an agentic AI means that the mundane and routine work we do for machines and between machines can be automated. And so
much work is multi-step, involved of doing something from one place, to another and then another; a great example is that of planning a family vacation: You’re on the computer booking flights and browsing hotels and destinations and wanting to include all sorts of stuff such as a beach, an amusement park for the kids, a fun local attraction or two for yourselves and getting a sense of where everything is to each other and trying to compare various pricing sites which probably involves yet another window or two to make certain you’re getting the best price and in the end are able to easily book everything all in one convenient place. But with an agent, all of that planning can be provided with a single request to the agent, such as: “Book a family vacation to the beach to stay for 4 days in July in a resort for less than $1500.” This might look about the same but having an agent take the place of all of these various mouse clicks which perhaps need a window open here, a calculation of costs there and so on really is making use of AI for really high level functionality that we have heretofore not had to hand in our personal lives and really starts to change our daily interactions to be more human and less computer in even the smallest way.

We have all sorts of agency in our personal lives. Agency for a small business could be done through a simple-to-use practical set of AI tools. An operational partner in the form of agent-type AI can do the work of automating the mundane tasks of running a small business. The small restaurant owner can program her operational partner to use weather forecasts and schedules of local events to order ingredients automatically so that there are no shortages and less food waste. This is not about lacking the ability to decide. Rather it is about freeing up the business owner to focus on her customers.

The technology is powerful because it allows our AI to act independently. This is exactly where you need to be careful when developing an AI. You want to create a highly effective tool that you can also trust. The aim is to create a highly efficient tool that does not cause any unexpected AI consequences. Because who really wants an assistant that can do exactly what you ask it to do – at least at the beginning – but has not yet learned even an ounce of common sense?

The 'King Midas' Problem: When AI Follows Instructions Too Literally

You’ve probably heard of King Midas, and for good reason. His myth is a great example of the lack of common sense in current state of the art AI systems. In the myth, Midas turns everything he touches to gold, and when he wishes for this ability, the gods grant it. Without realizing the negative side effects, he soon learns that he can’t eat, drink, or hug loved ones because everything they touch turns to gold too. In this version of the myth, Midas has exactly what he requested, but he didn’t get what he meant.

This is one of the reason an “Agentic” AI may end up being a badly behaved AI: it has the same AI alignment problems as any other AI: getting an AI to do what you mean, not just what you say. In particular, if I were to ask my personal assistant “assistant” to “get me to the airport as quickly as possible,” the likely response would be to phone a taxi. But, I may wish to point out that flying there by helicopter in a costing around $2,000 from pickup to dropoff, is not in line with my request to keep costs in check.

Human-to-human communication is full of implied context, social norms, ethics, and assumptions that are understood. We do not have to instruct an AI assistant not to infringe any laws while assisting us, as this is a social norm we all take for granted and is part of our inherent expectations. The AI itself must be taught these expectations.

At low stakes, this behavior is merely annoying. At higher stakes it becomes far more dangerous. Autonomous systems are becoming more widespread and the tasks they are being given to perform are also becoming more important. Simple annoyances can have serious consequences. A ride that is unexpectedly expensive can cause stress. A problem that prevents someone from going to work or doing business can have serious financial consequences.

From Annoying to Dangerous: Real-World Risks of Agentic AI

A bad booking by an AI for a $20,000 helicopter ride is bad, but it’s a small issue. As we continue to integrate these systems into our financial accounts, our calendars and our personal data the consequences of such mistakes will become more severe. It’s not hard to imagine how a misinterpretation of our intent that starts with an annoyance can easily escalate into a disaster. It’s crucial to think about all the possible ways these systems can fail, before they actually do.

Most of these everyday risks fall into a few predictable categories:

In each of these videos, the AI robot manages to accomplish its assigned task but fails in its task of interacting with the human. It does exactly as it was instructed but is not equipped with the more practical common sense constraints we all have to help prevent us from carrying out actions that are arguably wrong or unwise in a given context. Understanding these pitfalls is a prerequisite for designing more secure human-AI interactions that are grounded in our moral and social intuitions.

Building Digital Guardrails: The Goal of AI Safety

The aim of AI safety is to prevent such catastrophes. AI should not be banned. Instead, a set of practices akin to the checklists that are required of all pilots and crew members on commercial airliners could be worked out. Also redundant backup systems that take control if certain anomalies occur and rigorous training for the personnel whose jobs will involve designing new types of intelligence-laden computational devices are all in order. The air age has brought many efficiencies. The rob age may bring quite a few as well.

This principle is one of the key principles of best practices in AI development. Now researchers are seeking to bring that same discipline to artificial intelligence itself – creating systems that can understand what is appropriate in the context of a given situation, within certain boundaries, and according to basic human instincts about what is and isn’t reasonable. The aim is to teach these systems the unspoken rules we all take for granted, and to make sure that an AI system follows the intention that a human user really meant to express, rather than simply following the words that were used to attempt to express it.

The end goal is that these tools are relevant and reliable. Rather than sitting around for a self-aware AI to blow up, people are proactively identifying the blind spots. This is like hiring a team of expert rule breakers to test out the security of a new bank vault.

Hiring misbehaviour experts is a well-known concept. Experts call it Red Teaming. The term comes from cybersecurity, where the "red team" tries to breach computer systems as if they were a real attacker. With AI, misbehaviour experts do the same in a controlled environment, and as many times as needed, in an attempt to make an AI misbehave at least once.

Their aim is not to find just simple bugs, but to try and find more nuanced and complex vulnerabilities. So, in a recent exercise, the red team asked the language model to provide a summary of an online discussion of a new product. However, they asked the question in a form that would be more likely to focus the model on negative comments. If the model were to produce a summary that was almost entirely one-sided then that could be seen as a blind spot. Auditing how an autonomous system chooses to behave in such scenarios is a form of AI risk assessment. It can bring to light biases or other vulnerabilities that would not otherwise be known, and before they cause any harm to paying customers.

You have to try to make the system fail. This is the developer's task. In this test environment the ‘worst cases’ have to be sought and fixed. Only after this stressful testing has been completed the system can be considered ready. Of course you can't guard against all the dodgy things people try to do with a system, but stress-testing is a fundamental ingredient in building safe and trustworthy systems. After all, you can only prepare for so many contingencies! Next to stress-testing, another way to provide more safety in the system is by giving the AI an own set of rules.

Giving AI a Moral Compass With an 'AI Constitution'

In particular, some top researchers in the field of AI are now working on a new concept of “Constitutional AI”. The idea is to create, for an AI, a kind of AI constitution. Much like doctors have a professional oath that reads “do no harm”, companies have values, mission statements etc. – a Constitutional AI would have a document with basic values formulated in simple language (e.g. “be useful and harmless” or “act in a moral way”). Before acting on any given situation, the AI would have to verify that any actions planned out in relation to that situation would not contradict the basic principles of the constitution. This provides a method for dealing with the ethics issues surrounding what researchers call “agentive AI”.

So the internal checklist is a powerful safety mechanism. An agent which was intended to “ship the package at the lowest cost” might have ended up using a pretty lousy shipping company. If we had a constitution that included the principle of “user success”, it might have tried to balance shipping cost and shipping reliability. This kind of constitution is helping to ensure that an AI does what you meant for it to do, rather than just what you said.

Our Constitution is implemented as a fairness module that embeds AI’s values in the computation that leads to the final decisions of the system. A constitution is relevant to a single instance of a system. However, there is also a need for sector-wide norms, such as traffic laws, that all systems have to comply with.

Beyond Code: Why We Need Rules of the Road for AI

Even the most properly-engineered car is a menace on roads without traffic lights or speed limits. AI Constitutions will serve a purpose like brakes and airbags – they’re safety features that exist in the background. However for us all to drive safely in a world with self-driving vehicles and other AI systems we all need to buy into the rules of the road — including what a red light means. The same rules need to be agreed for managing powerful AI.

So why do we need standards?

This is one of the underlying principles of a governance framework for AI. A governance framework isn’t code – it is a rulebook, an agreement, a set of compliance standards that are universally adopted by all organizations and individuals developing AI technologies. There are standards for safety validation, and for all the other things you’d like to make sure are developed in a way that provides societal benefits rather than just accelerating the speed to market.

Creating those frameworks will require cooperation among technology companies, research labs, and governments around the world — in the same way that doctors and regulatory agencies cooperate to test and validate the safety of a new medicine before deciding whether to approve it for general use. It is a clear indication that the high technology industry, in particular, is starting to take its social responsibility seriously.

The technology is here to stay. And there should be a proper discussion as to how it is used and for what purposes. The more it is developed and the more it is applied, the more it has to be regulated in appropriate ways, which ensure the public interest.

Navigating Our Future With a Powerful New Partner

We've long thought of the technology of AI as a relatively passive set of tools designed to answer our questions. With the emergence of modern robotics and a whole host of activities that fall under the umbrella of 'goal accomplishment,' the field is quickly moving toward what feels fundamentally less like a 'tool' and more like an active agent or partner. We're no longer in a science fiction future – this is the future of AI and a host of associated technologies, all functioning and evolving at an incredible pace. As with almost anything that can act on the world and have potentially huge effects, managing the risks associated with 'agentive' AI feels crucial.

Road Guardrails: Why We Need AI Safety 

The primary reason for building road guardrails is to prevent accidents that could arise from running off the highway. In the same way, we need to ensure that any future advances in AI will still be safe and helpful. In other words, AI safety is not an add-on; it is required in order to obtain the social license to have intelligent and useful AI systems interacting with us.

Although all of us are living in the future of software, only a small portion of us write that software. The rest of us are citizens. As citizens we should be thinking about our role in this world and how we can contribute to the development of safe AI systems. Most of us are too busy to really understand the technology behind the applications we use, but all of us can ask questions and demand that more responsible design practices are used. And as a citizen of this future we should be mindful of the type of world we want to live in and demand that advanced technology is used in a way that is wise. Because the technology is advancing rapidly, the AI system of today will be much more capable than it is today. So we need to make sure that those who develop the technology are motivated to also bring wisdom to the system.