When Software Starts Making Decisions, Whose Values Does It Follow?

For decades, enterprise software has helped us make decisions.

We built sophisticated systems to capture data, automate transactions, measure performance, identify patterns, and present information to the people responsible for running businesses.

I spent much of my career in enterprise SaaS, including helping scale Cornerstone OnDemand. We built systems capable of serving tens of millions of users, managing complex processes, and providing extraordinary visibility into organizational performance.

But there was an important boundary.

The software could tell you what was happening.

A human still had to decide what to do about it.

A customer health dashboard might indicate declining adoption. A sales dashboard might reveal deteriorating pipeline conversion. A financial system might flag an unexpected variance.

The software calculated, organized, and presented information.

The human interpreted it, exercised judgment, and acted.

Agentic AI is changing that fundamental relationship.

We are moving from systems that help us understand what is happening to systems that can determine what should happen next and, increasingly, execute decisions within the authority we give them.

That is an extraordinary shift in the evolution of enterprise technology.

And it raises a question that deserves far more attention than it is receiving:

When software starts making decisions on behalf of an organization, whose values is it following?

The difference between what is and what ought to be

One of the most thought-provoking concepts explored in the AI Strategy & Builder Intensive course from the Harvard Data Science Initiative comes during a live session led by Professor Stephanie Dick, titled Ethics, Moral Reasoning, and Organizational Change in Agentic AI.

Professor Dick examines the work of Scottish philosopher David Hume, who identified a fundamental distinction between what is and what ought to be.

That distinction has significant implications for how we design agentic operating models.

We can know an extraordinary amount about a situation without that knowledge alone determining the right course of action.

Facts describe reality. They do not independently establish what we should value or prioritize.

Consider a practical example.

Imagine an AI agent responsible for monitoring customer retention across a portfolio of enterprise accounts.

It analyzes product adoption, support tickets, executive engagement, usage patterns, contract terms, and external market signals.

It identifies a customer at significant risk of churning.

The agent has assembled the is.

But what should happen next?

Should it automatically offer a substantial discount?

Should it protect profitability and accept the possibility of losing the customer?

Should it escalate the situation to an Account Manager?

Should it prioritize this customer over another account with a smaller contract but greater strategic potential?

The data may help predict the consequences of each option.

It cannot, by itself, determine which consequences the organization ought to prioritize.

That requires a decision about values, objectives, obligations, and acceptable tradeoffs.

And once an agent has authority to act, those decisions can no longer remain entirely implicit.

Making implicit decisions explicit

This may be one of the most consequential organizational implications of agentic AI.

Historically, an experienced employee could navigate ambiguous situations using judgment, institutional knowledge, and an understanding of the organization's culture.

A relationship manager might know when to make an exception. A foundation executive might understand why a particular organization deserves additional consideration. A customer success leader might recognize that a technically healthy account is actually at risk because of an executive relationship.

These decisions often happen without anyone formally documenting the principles behind them.

But when we delegate portions of that work to autonomous systems, we have to translate at least some of those principles into explicit instructions, decision criteria, permissions, and escalation rules.

In effect, we are being asked to pre-compute aspects of organizational judgment.

Not every possible decision can be anticipated. Nor should every decision be automated.

But the act of designing an agent forces us to confront questions that our existing operating models may have avoided.

What outcomes are we optimizing?

What rules are non-negotiable?

What exceptions are permitted, and who has authority to approve them?

When should an agent stop and ask for human judgment?

What happens when two organizational priorities conflict?

And who is accountable when the system acts?

These are not merely technical design questions.

They are questions of organizational governance.

You cannot buy your organization's ethics

The issue becomes even more interesting when we consider the build-versus-buy decision.

Most organizations will not build their own foundational AI models. They will use commercial models, agent platforms, and specialized solutions from technology providers.

That makes economic and operational sense.

But what exactly are they buying?

Large language models learn patterns from enormous amounts of human-created information.

That information reflects different cultures, historical experiences, social norms, assumptions, and ways of reasoning.

Models are also shaped by training methods, developer choices, and safety policies.

They are not culturally or ethically neutral simply because they are software.

The Harvard course explores research suggesting that AI models can reflect some cultural perspectives more strongly than others, particularly those prevalent in Western, English-speaking societies.

During my tenure at Cornerstone OnDemand, I built and led the company's Asia-Pacific business, establishing operations across seven countries.

One of the most important lessons from that experience was that you cannot simply export a successful American operating model into another market and expect the same outcomes.

The technology might be the same. The product might be the same. Even the business objectives might be the same.

But the cultural context in which people make decisions, build trust, communicate, negotiate, and conduct business can be profoundly different.

What constitutes effective communication in one country may be perceived as unnecessarily confrontational in another. How decisions are made, how authority is respected, how relationships develop, and how organizations establish trust can vary considerably, even between neighboring markets.

We had to understand those differences and adapt how we operated while maintaining the core values and objectives of the company.

Now consider what happens when we introduce autonomous AI agents into those same environments.

If a foundational model has learned patterns of reasoning that disproportionately reflect Western cultural norms, what happens when it begins making decisions or communicating on behalf of an organization operating across Asia-Pacific, the Middle East, or other global markets?

Will it recognize the cultural nuances that influence how a customer relationship should be managed?

Will it understand when direct communication is appropriate and when a more relationship-oriented approach is necessary?

Will it interpret urgency, authority, trust, and business priorities in ways that align with local expectations?

Or will it unknowingly impose assumptions that are entirely reasonable in one culture but counterproductive in another?

An agent can be technically correct and culturally wrong.

And when that agent is empowered to act, the consequences extend beyond an awkward interaction. They can affect customer relationships, commercial outcomes, employee experiences, and an organization's reputation.

This is not an argument against using foundational models globally. It is an argument for recognizing that cultural context must be an intentional part of agentic operating-model design.

Just as we adapted our human operating models across Asia-Pacific, organizations will need to consider how their AI agents should behave across different markets, stakeholders, and cultural environments.

That may require localized policies, contextual instructions, market-specific workflows, appropriate human oversight, and ongoing evaluation of how agents actually behave.

It also requires recognizing that cultural norms are not uniform within any country and cannot be reduced to a simplistic set of national stereotypes.

That is why I believe the build-versus-buy question needs to expand.

It is not simply:

Can we build this more economically than we can buy it?

It is also:

Can we govern what we buy so that it behaves consistently with our organization's values, obligations, and operating model, across every market in which we operate?

Buying a capable model does not mean outsourcing accountability.

Audit the decisions, not just the code

Traditional software audits often focus on security, compliance, code quality, access controls, and operational reliability.

Those remain essential.

But agentic systems introduce additional questions.

Does the agent consistently apply the organization's stated policies?

Does it treat comparable situations consistently?

Does it produce different outcomes for different populations, regions, or stakeholder groups without a legitimate reason?

Does it recognize uncertainty?

Does it escalate situations outside its authority?

Can a human inspect the evidence supporting its recommendation and identify material limitations?

And can the organization reconstruct what happened when something goes wrong?

This calls for more than a technical audit.

It requires examining how the entire system behaves within its real-world organizational context.

A governance framework should make the organization's choices visible and testable.

Not because an audit can guarantee that an agent is ethical.

It cannot.

But because an organization cannot responsibly govern decisions it cannot observe, explain, or challenge.

The next generation of operating models will encode organizational values

As AI becomes more capable, it will become tempting to measure progress primarily by the number of tasks or workflows we can automate.

That would be a mistake.

The more consequential question is how thoughtfully we redesign the work.

What decisions should be delegated?

What principles should guide them?

What authority should agents have?

Where must humans remain accountable?

And how do we ensure the system behaves as intended when real-world conditions become complicated?

These questions will become increasingly important as organizations move from isolated AI assistants to networks of specialized agents collaborating across functions and systems.

In that environment, organizational values cannot remain abstract statements in a corporate handbook.

They must become operational.

They must be reflected in the design of jobs, workflows, permissions, constraints, escalation paths, and governance.

And they must be tested against what the system actually does.

For executives, this represents a new responsibility.

We are not simply deciding what AI can do. We are deciding what AI should be allowed to do on our behalf.

That distinction may ultimately prove more important than the technology itself.

Because when software begins making decisions, the organization's values are no longer expressed only through its people.

They are also expressed through its systems.

And whether we design those systems intentionally or not, we remain accountable for the decisions they make.

The future of agentic AI is not just about building more intelligent systems. It is about building systems worthy of the authority we give them.