The AI agent explosion: Who is managing the digital workforce?

Artificial intelligence is no longer simply a tool employees use to assist with everyday tasks. Across enterprises, AI is evolving into a digital workforce capable of accessing information, executing workflows, and making decisions with varying degrees of autonomy.

From customer service and IT operations to finance and human resources, organisations are adopting AI agents to improve efficiency and streamline operations. However, as these systems become more capable, an important question emerges – who is responsible for managing them?

According to Gartner research, the average global Fortune 500 enterprise could have more than 150,000 AI agents by 2028, compared with fewer than 15 in 2025. Yet only 13-percent of organisations believe they have appropriate AI agent governance mechanisms in place.

While these figures reflect forecasts and survey findings, they highlight a growing concern: enterprise AI adoption may be advancing faster than the governance structures needed to support it.

The rise of an invisible digital workforce

Unlike traditional enterprise software, AI agents can operate independently within defined parameters, accessing applications, processing information, and initiating actions without constant human intervention.

An AI agent in finance might reconcile transactions or process payment requests, while another in human resources could manage onboarding or access employee records.

Although these capabilities offer operational advantages, they also introduce risks. As departments deploy agents independently, organisations could find themselves managing numerous autonomous systems with limited visibility into their activities.

This phenomenon, known as AI agent sprawl, resembles shadow IT, but with an important distinction: AI agents do not simply process information; they can act on it.

A real-world example is Swedish fintech company Klarna, which introduced an AI-powered customer service assistant in 2024. Within its first month, the assistant handled 2.3 million conversations, accounting for two-thirds of the company’s customer service chats, according to Klarna. The company also reported faster resolution times and fewer repeat inquiries.

While Klarna’s assistant is not necessarily representative of today’s more autonomous AI agents, its deployment demonstrates how quickly AI can assume responsibility for customer-facing operations. As enterprises move beyond conversational assistance toward agents capable of executing transactions and accessing sensitive systems, the need for clearly defined permissions, human oversight, and accountability becomes increasingly important.

Who is accountable when AI makes a mistake?

As AI agents take on greater responsibilities, accountability becomes increasingly complex.

If an agent processes an incorrect financial transaction or exposes confidential information, who bears responsibility? The employee who initiated the task, the department deploying the agent, or the organisation itself?

While the answer depends on the circumstances, delegating decisions to AI does not eliminate organisational accountability.

The NIST AI Risk Management Framework provides a foundation for addressing these concerns. Its four core functions – Govern, Map, Measure, and Manage –  emphasise defined responsibilities, continuous risk assessment, and appropriate oversight.

For enterprises, governance must extend beyond initial deployment to how agents operate, what decisions they make, and how their activities are monitored.

Managing AI identities and permissions

Traditional enterprise environments assign employees digital identities and access permissions. AI agents require similar controls, particularly when interacting with multiple business applications and databases.

Without adequate safeguards, organisations risk creating networks of agents with excessive or poorly documented privileges.

Gartner identifies agent identity management, permission structures, and lifecycle controls as essential to addressing AI agent sprawl.

Enterprises must establish clear ownership, define access privileges, monitor activities, and regularly review permissions.

Established cybersecurity principles, including least-privilege access and zero-trust security, can help reduce these risks. Higher-risk activities, such as authorising payments, may also require human approval.

The growing challenge of Shadow AI

As AI tools become more accessible, employees and departments may deploy agents without formal IT or security approval.

While this encourages innovation, it can create fragmented environments where organisations lack visibility into their automated systems.

Gartner emphasises balancing governance with innovation rather than relying solely on restrictions.

Maintaining a centralised agent inventory, establishing deployment policies, and conducting regular access reviews can strengthen oversight without unnecessarily limiting adoption.

Importantly, governance must extend beyond IT to include cybersecurity, compliance, legal, and business leadership teams.

Can enterprise governance keep up?

The rise of agentic AI represents a shift in enterprise automation. Until recently, the conversation focused on what AI could generate, analyse, or recommend. Increasingly, it concerns what AI can independently execute.

For CIOs and enterprise leaders, this means establishing visibility into their digital workforce, defining accountability, and ensuring mechanisms exist to intervene when something goes wrong.

The challenge is not necessarily how many AI agents an organisation deploys, but whether it can manage them responsibly.

Ultimately, successful adoption will depend on more than technological capability or deployment speed. It will require governance, security, and accountability structures capable of supporting increasingly autonomous operations.

The digital workforce is expanding. The question now is; can enterprise governance keep up?