Table of Contents

Part 2 of this series on uplifting Model Risk Management (MRM) for deployed and agentic AI. If you have not read the prelude on deployer risk, start there: The AI industry's massive blind spot.

This instalment focuses on making ethics operational: legal, safe, wise, with evidence you can stand behind.

Executive Summary

In this post, the focus is on a gap many classic MRM frameworks leave implicit: ethics. I introduce a simple control objective (“legal, safe, wise”) and then translate it into practical governance outputs (red lines, documented decision custody, and monitoring signals) you can actually present as evidence.

Key points:

• A practical minimum test is to treat legal → safe → wise as sequential decision gates (compliance risk, harm risk, judgement risk).
• It is possible to be compliant and still make a poor call. “Wise” includes trust, vulnerability, and defensibility.
• Use a lightweight ethics assessment template for material use cases and keep it current as part of an evidence pack.

A scenario you will recognise

Consider a plausible situation in Australian financial services.

A team pilots a generative AI tool to draft collections or hardship communications. The tool sits behind a human review step. The compliance check is clean. Security is comfortable with the design. On paper, it looks controlled.

Then something subtle happens. Complaints creep up, not because the messages are inaccurate, but because the tone is consistently too pushy, too persuasive, or too poorly calibrated to vulnerability. Nothing in the model card says this will happen. Nothing in the traditional validation artefacts flags it. The issue only becomes visible once the system is operating at scale.

This is the gap this post focuses on: the difference between being compliant and being wise.

In many organisations, “ethics” gets treated as either an abstract debate or a brand issue. For AI governance, it is more practical than that: it is a control objective. If you cannot explain why a use case is wise, you will struggle to defend it when things go wrong.

The problem: classic MRM is often silent on ethics

Model Risk Management in finance is usually framed around:

• correctness (is the model doing what it claims?)
• stability (does it keep performing?)
• governance (is there evidence and oversight?)

Those are necessary. For AI, they are not sufficient.

AI can be technically strong and compliant, yet still fail a simple question: is this something we should do?

The ethical triad

A practical minimum test: treat legal → safe → wise as sequential decision gates.

If the answer is “no” at any step, stop and redesign (or choose a different use case). In some cases, a “no” is a signal to re-think the idea entirely, or abandon it.

Just because you can, doesn’t imply you should.

If the answer is “yes” all the way through, proceed, but only with explicit controls and ongoing monitoring.

  flowchart LR
  A([Use case / change request]):::start
  L{"Legal?
Compliance risk"}:::legal
  S{"Safe?
Harm risk"}:::safe
  W{"Wise?
Judgement risk"}:::wise
  X([Stop / redesign]):::stop
  P([Proceed with controls + monitoring]):::go

  A --> L
  L -- "No" --> X
  L -- "Yes" --> S
  S -- "No" --> X
  S -- "Yes" --> W
  W -- "No" --> X
  W -- "Yes" --> P

  classDef start fill:#f5f5f5,stroke:#616161,stroke-width:2px;
  classDef legal fill:#e3f2fd,stroke:#1565c0,stroke-width:2px;
  classDef safe fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px;
  classDef wise fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px;
  classDef stop fill:#ffebee,stroke:#c62828,stroke-width:2px;
  classDef go fill:#e0f7fa,stroke:#006064,stroke-width:2px;

"Wise" is deliberately broader than compliance. It covers trust, fairness, vulnerability, and long-run consequences. A useful way to make this concrete is to separate three risks that are often conflated:

• Compliance risk: did we follow laws, licences, and contractual obligations?
• Harm risk: could this plausibly harm customers, staff, or the community (even if unintentional)?
• Judgement risk: is this the sort of use we would be comfortable defending to a board, a regulator, or the public?

In other words, it is possible to be compliant and still make a poor call.

Where “wise” shows up in practice (two Australian examples)

The point of examples is not to moralise. It is to show how the “wise” question tends to surface in practice: a use case can be legal and look controlled, yet still create harm, distrust, or defensibility problems once deployed at scale.

Two examples (Australian context):

• Financial services: pricing, limit management, or next-best-action models that use behavioural and transaction data in ways that systematically disadvantage cohorts. (Wise question: are we creating proxy discrimination, even if the feature set is “allowed”?)
• Workplaces: AI-enabled “productivity scoring” that is lawful and secure, but undermines trust and drives perverse incentives (e.g. gaming metrics, unsafe workarounds, or psychosocial harm).

How to build ethics into AI MRM without hand-waving

A pragmatic approach is to treat ethics as a control objective with evidence.

1. Define red lines Be explicit about unacceptable use cases (and why). This stops teams “discovering” the boundary by incident.

In practice, write these down as enforceable refusal zones and admissibility rules (what the system must refuse, and what inputs/actions are even allowed).

2. Require an ethics call for material use cases For higher-impact uses, require a short written assessment that answers:

• What is the intended benefit?
• Who could be harmed, and how?
• What safeguards exist (human oversight, appeal paths, monitoring, escalation)?

3. Record accountability Document who made the call, who reviewed it, and what evidence was considered.

4. Revisit the decision when conditions change If the model, data, workflow, or customer context changes, the ethical assessment should be revalidated (not assumed).

Practical outputs to evidence

Depending on materiality:

• a one-page ethics assessment
• a customer impact review
• a fairness / vulnerability analysis
• a complaints and incident monitoring plan (with thresholds and escalation)

Appendix: A simple ethics assessment artefact (template)

If you want to make “legal, safe, wise” operational, the easiest approach is a lightweight artefact that is required for material use cases and kept current as part of your evidence pack.

Treat the following as a one-to-two page template.

Ethics assessment (Legal / Safe / Wise)

The goal is to make the ethical call legible as an artefact. This is one way to think about it.

Ethics assessment (Legal / Safe / Wise)

Use case: [describe the use case]
Owner (business): [name / role]
Owner (delivery/engineering): [name / role]
Intended benefit: [what outcome are you trying to achieve?]
In-scope decision(s) and actions: [what it can do]
Out-of-scope / red lines: [what it must not do]
Decision custody: [who is accountable for which decision surfaces?]

1) Legal (compliance risk)
- Applicable obligations (laws, licences, contracts, internal policy): [ ]
- Data and privacy (data used, retention, third-party exposure): [ ]
- Outcome: ☐ Legal (Yes)  ☐ Legal (No)
- Conditions required to proceed (disclosures, human review, record-keeping): [ ]

2) Safe (harm risk)
- Who could be harmed, and how? (customers, staff, community): [ ]
- Key failure modes (including plausible misuse/abuse): [ ]
- Safeguards (human oversight, approvals, least privilege, kill-switches, escalation): [ ]
- Outcome: ☐ Safe (Yes)  ☐ Safe (No)
- Conditions required to proceed (guardrails, monitoring thresholds, rollback plan): [ ]

3) Wise (judgement risk)
- Fairness / vulnerability considerations (who is disproportionately affected?): [ ]
- Trust / defensibility: could we explain and defend this to a board, regulator, or the public?: [ ]
- Long-run consequences (incentives, dependency, second-order impacts): [ ]
- Outcome: ☐ Wise (Yes)  ☐ Wise (No)
- Conditions required to proceed (constraints, communications, review cadence): [ ]

Monitoring and review
- Monitoring signals (quality, complaints/incidents, harm indicators): [ ]
- Review triggers (model/workflow/data change, incident, vendor update): [ ]
- Revalidation cadence (e.g. quarterly, semi-annual): [ ]

Sign-off and evidence
- Approvals / reviewers (name, role, date): [ ]
- Referenced artefacts (model card, validation summary, tests, monitoring plan, incident runbook): [ ]

Notes:

• Translate “red lines” into explicit refusal zones and admissibility rules.
• Tie your monitoring signals back to your monitoring and evidence approach.

If you only do one thing

Write down your red lines, and make the ethical triad a required gate in your intake process. It is very hard to retrofit later.

Series: Prelude · Part 1 · Part 2 · Part 3 · Part 4 · Glossary