Theme Name

Trustworthiness of multi-objective agentic AI: a collective-choice formalism

Proposer(s)

Arnaud Taddei, Chair, ITU-T Study Group 17.

Raising this on behalf of an existing ITU-T work item rather than as a new idea (see Related Work). No claim on leadership — the theme needs people this proposer does not have.

Summary

An agentic AI system asked to be secure, private, safe, resilient, accountable, transparent, fair and robust is asked to satisfy objectives that conflict. Every decision it takes silently resolves that conflict somehow, and today nobody declares how, measures how, or can detect that the resolution has changed. This theme proposes that the resolution is formally a collective choice problem — an agent aggregating conflicting objectives is doing what a society does aggregating individual preferences — and that the existing mathematics of collective choice therefore gives us a way to name, measure and constrain agent behaviour. If that holds, trustworthiness stops being an adjective and becomes a quantity.

Scope / Open Questions

The claim to be tested first. Multi-criteria choice optimisation is in mathematical bijection with collective choice theory. This is stated as a working hypothesis, not a result to be assumed: the group should decide whether it accepts it or wants it defended.

What follows if it holds. Arrow's impossibility theorem applies to agentic AI. No aggregation rule satisfies all the reasonable axioms simultaneously, so the bias of a multi-objective agent is not a defect that better engineering removes — it is unavoidable. Neither a person nor a synthetic intelligence escapes it. That reframes the useful questions:

• Which aggregation operator is a given system actually using?
• Is it the operator its operator declared?
• How far has it drifted, and can drift be given a number?
• Which operators are compatible with a given regulation, and which are not?

The apparatus already exists. The algebraic school of choice theory developed at the Institute of Control Sciences of the Russian Academy of Sciences — Aizerman and Malishevski's axiomatic characterisation of choice functions, Mirkin's work on group choice — does not treat choice rules one at a time. It characterises them by axioms (heritage, concordance, outcast, and their combinations) and yields a classification of choice operators. A classification is what makes engineering possible: a finite vocabulary of named operators a system can declare it uses.

What exploration would need to produce.

1. An operator catalogue — vendor-neutral, implementable, with each operator's axioms and known failure modes.
2. Declaration and measurement — how a system declares its operator; how an observer estimates the effective operator from observed behaviour; how drift between declared and effective is defined and measured.
3. A regulatory bridge — a mapping from regulatory requirements onto operator axioms. "Removing an irrelevant alternative must not change the outcome" is a statement about an axiom, not about a vendor, so a regulator can state a requirement once, technology-neutrally, and have it hold across every system that declares its operator. To be worked against at least two frameworks, one of them non-European, so the bridge is not European by construction.

Open questions.

• Can the effective operator be estimated from externally observable behaviour alone, or does it require instrumentation inside the agent? This decides whether the approach is auditable by a third party or only by the operator itself — and therefore whether it is a governance tool or a self-assessment.
• Does the drift measure survive an adversary who knows it is being measured?
• Arrow's unavoidable bias is exploitable: an adversary who knows the operator knows how to manipulate the agenda to force an outcome. Does the same structure serve as a detection signal for defenders?
• What of an agent that builds a counter control plane — attempting to evade, subvert or duplicate its own declared control plane, either autonomously or under external manipulation?
• Which operator families are realistically implementable inside a production agent's decision loop, and at what latency cost?
• Who, inside a regulator, is the reader of deliverable 3 — and has anyone asked one?

Related Work

ITU-T SG17 — this theme's parent. Work item XSTR.magAI-behavior, "Capabilities for Agentic AI security and trust controls to manage the behavior of single or multiple agentic multi-objective AI systems", agreed under Question 16/17 at the SG17 plenary of June 2026.

• SG17-TD328/WP4 — outcome of the discussion, Source: Co-Rapporteur Q16/17. Contacts: Jae Hoon Nah (ETRI, Korea), Oscar Avellaneda (Innovation, Science and Economic Development Canada), Xiongwei Jia (China Unicom).
• SG17-C-0433 — the originating contribution, "Security framework for emergent, misaligned and multi-objective behaviours in advanced and multi-agentic AI systems".

The work item already carries the thesis: multi-objective agentic decisions modelled as collective choice operators; policies expressed as off-the-shelf operators in an agentic AI control plane; drift measured between specified and effective operator; Arrow's unavoidable bias treated as both attack surface and detection signal. It distinguishes three categories of behaviour that are usually conflated — emergent capabilities, misaligned behaviours, and agentic security-relevant behaviours — with empirical grounding in published frontier model system cards from several vendors.

Mathematics. Arrow's impossibility theorem. Aizerman & Malishevski, axiomatic theory of choice functions (Institute of Control Sciences, RAS). Mirkin, group choice. Multi-criteria decision analysis.

Regulatory. EU AI Act; and deliberately at least one non-European instrument, to be selected.

Adjacent ITU-T Focus Groups. FG-AINN (AI-native networks) and the AI Focus Groups were identified in the SG17 editor's notes as requiring liaison on this work item. Probably in particular FG-EAI (evaluation approaches)

Related Themes

• 1 — Accountability and attribution after an agent has acted. Attribution after the fact presupposes a statement, made before the fact, of what the agent was supposed to do. This theme proposes the form that statement could take.

• 3 — Global Trust and Accountability for Agentic AI. Same relationship: this theme offers the formal machinery, not a competing scope.

Neither of these themes is superseded by this one. If anything the dependency runs the other way — this theme is only useful if #1 and #3 find it usable.

Discussion

A proposal on the mechanism, offered for discussion and not for decision.

The obvious move would be to transfer XSTR.magAI-behavior from SG17 to FG-TIDA. There is a real case for it: the SG17 delegations judged the topic very powerful but would prefer to see it as a Technical Report first before moving it back as a normative Recommendation, which is why it was converted from X. to XSTR. Moving to FG-TIDA would give it a much stronger maturity level and that maturity level is precisely what pre-standardisation is for. The framework also explicitly involves regulators as stakeholders, and FG-TIDA is open to anyone with no ITU membership and no fee, while SG17 is not. The audience this work needs is the audience this Focus Group can actually reach.

Against transfer: Canada has committed to support and promote the work item within SG17, and that coalition should not be stranded; a Focus Group deliverable carries no normative weight and this group's mandate ends in 2028; and SG17 has already taken a decision that should not be casually reopened.

The shape that seems to hold is a dual track. XSTR.magAI-behavior stays in Q16/17 as the ITU-T deliverable. FG-TIDA incubates the formal core specifically — operator catalogue, drift measurement, regulatory bridge — at Focus Group speed and in the open, with an explicit commitment that results feed back into the XSTR. The Focus Group does what it is good at; the Study Group keeps the pen.

Who needs to be in the room. Someone who has read Aizerman and Malishevski. Someone who has shipped an agent to production. And at least one regulator. The theme fails if any of the three is missing, and the third is the one we are least likely to have.

Comments and disagreement welcome — particularly on the bijection claim, which is the load-bearing wall.