[Theme] Model-level Trust Foundations

Theme Name

Model-level Trust Foundations

Proposer(s)

Ward Duchamps, Thales
Aayush Gandhi, DatumND
Jack Lord, ODS
Scott Allen

Summary

How an agent behaves is dictated by multiple factors, the AI model is the earliest point of intervention in influencing agentic behavior.

Yet almost all standardization efforts at FG-TIDA today act on the runtime system around the model, or on evaluating the model's outputs after it is built. The way a model is formed, its training data, its reinforcement learning, its retraining and unlearning, sits outside any of the existing work streams.

The intention of this Model-level Trust Foundation theme is to assess if it makes sense to start a standardization discussion on trust at the model-level. This may touch at multiple layers:

• Knowing how the model was generated
• Evaluating and benchmarking the ‘vanilla’ capabilities/behaviors of the model
• Assessing what guardrails can be added by either the inference provider or the agent harness
• Evaluating and benchmarking the capability/behavior implications of those guardrails

This theme should take into account the diversity in models and model architectures. Commercial models provided by frontier labs come with constitutions, embedded guardrails, and transparent harnesses that add system-level controls without the user noticing it, e.g., classifiers, filters, rate limiters. Self-deployed open-weight models have none of these and expose the end-user to true/raw model capabilities.

Scope / Open Questions

Trust in an agentic system depends on many things. The model is one of them, and it is the one that decides how the agent reasons and acts. Correcting behavior at the model is the earliest point of control. A fault removed there never reaches the runtime, the deployment, or the user. Catching the same fault later through monitoring and intervention requires system-level controls.

Model-level controls cannot replace or make system-level controls obsolete. Some behaviors of the model aren't easily alterable and/or can only be fixed in post-training. System-level controls may compensate for these faults without the need to spend additional compute to remove or hide model capabilities.

It is sometimes much more difficult to identify why a model behaves a particular way, versus the easier task of identifying that it does behave that way. The gap is manifold. The pretraining data and regime often aren't clear. Likewise the precise post-training methodologies employed also aren't shared. All of these in combination with the architecture of the model, lead to the ultimate behavior. There may be value in knowing precisely what went into the full training of a model. But importantly, it wouldn't be obvious or possible to predict a priori the emergent behaviors/capabilities the model will have as a result of that training.

Standardizing model-level trust risks either locking in one lab's specific values as the global baseline or producing standards that are too vague to be useful. For instance, if Anthropic sets the baseline, what incentive does any other lab have to follow their baseline, and how to evaluate that baseline? Also, standards may become incredibly vague to ease their adoption, while not being specific enough to address the complex misuse scenarios.

Most current standardization effort goes to attestation and evaluation; that is, checking a finished model's behavior from the outside. Understanding or controlling how behaviours originate is the core question of a large part of the alignment research community. This theme assumes that there is still a gap between research and standardization. It’s in the scope of this theme to assess whether it is desirable and feasible to close this gap.

Why this may be worth standardizing

The behavior originates in the model. If the reasoning of a model can be assured, wrongdoing is prevented at the source rather than contained afterwards.

The guardrails that shape model behavior today are corporate and voluntary. Frontier labs invest real effort in the norms and character of their models; Claude's Constitution and the OpenAI Model Spec are public examples. They are published, but they are built by companies on their own terms, and there is no shared reference for what such guardrails should cover. The two named examples already differ on a basic point: Anthropic's constitution explicitly includes the UN Declaration of Human Rights; OpenAI's Model Spec does not name it. A lab can also revise its own specification at any time, and take the effort to teach the model reflecting that specification. Without a common baseline, a buyer or a regulator has nothing external to check a model's formation against.

Note that the presence or publishing of a constitution-like asset really provides no sense as to how that has been internalized by the model. A lot depends on the post-training approach taken to imbue the model with the essence of that asset. That is, starting with the same model and the same constitution but implementing separate RL training regimes and ending up with very different models.

A shared vocabulary would let a requirement be stated once and checked across providers, instead of each lab describing its choices in its own language. However, this does not guarantee that models eventually behave. Even if providers have the same constitution at model training, and post train with that constitution identically, the net result or the way and degree to which the constitution is imbued into the model will be different because different providers may use different architectures. Nevertheless, there could be value in knowing the constitution that was leveraged and the post training methodology that was used.

Which domains could be standardized

The scope of this theme includes the formation of which model-level controls may require transparency and eventually could be standardized, which may be embedded in pre- and post-training stages of the model lifecycle.

It is important to not only consider frontier AI labs, but also specialized models, and locally trained models.

Initial topics for consideration include:

• Curation requirements for training data, for example respecting human rights and diversity
• Specifications for acceptable behavior of model characters, for example refusing illegal or harmful behavior.
• Removing unacceptable responses, for example refusing chemical, biological, radiological and nuclear (CBRN) information.
• Curated learning procedures for model providers
• Transparency requirements to disclose training and fine-tuning methods, for example which regional norms and ethics were taken into account during fine-tuning

Note: There may be conflicting objectives between standardization initiatives and commercial choices of model providers (private companies). How much of the values, principles, and priorities may become industry standards? This topic needs to be addressed while developing this theme.

Note: Authors acknowledge that some of these topics are actively in research, e.g., how post-training can avoid and/or unlearn certain model capabilities.

Current state

The UN Transparency Protocol may be an interesting comparator here. This is designed for supply chains and sustainability, which involves attestation of the standards adhered to in production of a product.

The General-Purpose AI Code of Practice attempts to set a regional standard for EU that helps industry comply with the AI Act legal obligations on safety, transparency and copyright.

The OWASP CycloneDX Bill of Materials Specification (ECMA-424) is a standard that provides advanced supply chain capabilities for cyber risk reduction.

The System Package Data Exchange® (SPDX®) Specification provides an AI Profile which is designed to provide a standardized way of documenting and sharing information about AI software packages

Several initiatives are under active development to cover attestation and evaluation, such as:

• MLCommons, whose AILuminate benchmark evaluates chat models across twelve hazard categories and assigns risk ratings for developers and consumers;
• Fide AI, which helps labs and organizations in high-trust domains build trustworthy AI systems;
• EvalEval, a coalition on evaluating evaluations;
• CSA AARM, on conformance requirements and benchmarking.

These act mainly on the finished model. The formation of the model is where the discussion proposed here would begin.

Smaller AI research labs are building specialized models that implement almost like a ‘trust-by-design’ principle. Worth mentioning:

• Thinking Machines Lab — thinkingmachines.ai. An AI research and product company building frontier models that are more understood, customizable, and adaptable to a user's own needs and values; it commits to publishing model specs and safety recipes. Relevant because it treats a model's values and safety as things set during training and made public.
• LawZero — lawzero.org. A nonprofit, insulated from commercial pressure, building a new "safe-by-design" form of AI under Yoshua Bengio, on the premise that current systems are capable but opaque and misaligned. Relevant because it illustrates designing trustworthiness into how a model is formed rather than containing faults afterward.
• Safe Superintelligence Inc. (SSI) — ssi.inc. A lab with a single goal, advancing safety and capability "in tandem" as engineering problems, with a business model deliberately insulated from product cycles and short-term pressure. Relevant because it makes safety a formation-time objective, not a runtime add-on. It also illustrates that ‘trust’ rests on each company's own terms with no external baseline.
• Jev by TypeSafe — typesafe.ai. TypeSafe is an AI lab building "machine-native" models for automation. Its first model, Jev, emits typed probabilistic decisions in parallel instead of generating text token by token, which it claims removes hallucination and type errors by construction. Relevant because it builds a trust property into the model's architecture itself.

Related Work

Anthropic, 2026 - Claude’s Constitution

OpenAI, 2026 - OpenAI Model Spec

NIST AI Risk Management Framework (NIST AI 100-1 RMF 1.0), which lists risks to address when building guardrails into a model and names threats such as confabulation (not "hallucination"), harmful bias, and information insecurity.

NIST AI 600-1, the Generative AI Profile (GAI), adds specific threats: eased access to CBRN information, sophisticated disinformation, and "model collapse" through training on synthetic data.

The WMDP Benchmark, proxy measurement for hazardous knowledge in LLMs, used as a benchmark for unlearning. Measuring the model's knowledge, not whether a user is going to maliciously act on knowledge.

S. Casper et al., 2026, "Open Technical Problems in Open-Weight AI Model Risk Management", arXiv:2608.07514v1.

Related Themes

Discussion