Oversight that cannot override无法否决的人工监督
A person is nominally in the loop but lacks the information, time, competence or stopping authority to change the outcome.名义上有人参与,实际上却没有信息、时间、能力或停止权限去改变结果
Independent assessment for consequential decisions面向重大决策的独立评估
Phoenix examines how AI actually enters a decision: who may challenge it, which evidence is verified, how conflicts are disclosed, and whether human oversight still works under pressure. 菲尼克斯研究AI如何真正进入决策:谁有权质疑,哪些证据经过验证,利益冲突如何披露,以及人工监督在压力之下是否依然有效
The overlooked risk被忽略的风险
Failures rarely live inside the model alone. They emerge between the system and the organization: interfaces that invite over-reliance, reviewers without time or authority, contrary evidence that quietly disappears, and vendors who control both the evidence and its interpretation.失败很少只发生在模型内部,更多出现在模型与组织之间:界面诱导过度依赖,复核者缺少时间或否决权,反面证据悄然消失,供应商同时掌握证据与解释
A person is nominally in the loop but lacks the information, time, competence or stopping authority to change the outcome.名义上有人参与,实际上却没有信息、时间、能力或停止权限去改变结果
Decision-makers see supporting signals but cannot reconstruct the sources, assumptions or counter-evidence behind them.决策者看到的是支持性信号,却无法重建其来源、假设与反证
Whoever controls the data, tooling or budget quietly shapes the review, escalation and remedy process.掌握数据、工具或预算的一方,无形中主导了复核、升级与补救
Efficiency gains accrue centrally while errors, surveillance, skill loss and correction costs fall on others.效率收益集中获得,而错误、监控、技能退化和纠错成本由他人承担
What we do我们的服务
We assess one defined decision process, not an entire organization by assertion. Scope, evidence and limitations are agreed before the work begins.我们评估的是一条明确的决策流程,而不是凭印象评价整个组织,项目开始前先约定范围、证据与限制
A four-layer review of whether one consequential AI-assisted decision remains challengeable, traceable and accountable under real operating pressure.围绕一条重大的AI辅助决策链开展四层评估,检验它在真实运营压力下是否仍然可质疑、可追溯、可问责
Independent review of system purpose, evidence, monitoring, human oversight, supplier dependency and governance controls, mapped to NIST and ISO.独立审查系统用途、证据、监测、人工监督、供应商依赖与治理控制,并映射到NIST与ISO框架
Closed-door board and executive exercises that surface automation reliance, motivated reasoning, evidence selection and escalation failures, without turning anyone into a psychological subject.通过闭门的董事会与高管演练,识别自动化依赖、动机性推理、证据筛选和升级失效,不对任何个人做心理诊断
Decision-focused scenario trees, sensitivity analysis and simulation for supply chains, infrastructure, investment and policy, used only when the data and mechanisms justify the model.面向供应链、基础设施、投资与政策的情景树、敏感性分析与仿真,只有在数据与机制充分时才使用复杂模型
Assessment architecture评估架构
We combine documentary review, interviews, sampled reconstruction of past decisions and controlled behavioral testing. Each layer produces its own evidence and its own output, and the four are cross-checked before anything is graded.我们结合文件审查、访谈、历史决策抽样复建与受控行为测试,每一层都有自己的证据与输出,四层交叉核对之后才进入分级
Applicable law, contracts, internal policy and standards are mapped to concrete controls.把适用法律、合同、内部政策与标准映射为具体控制
Output · obligation-to-control map输出 · 义务与控制映射Purpose, data, performance limits, monitoring and supplier dependence are tested against logs and records.依据日志与记录测试系统用途、数据、性能边界、监控与供应商依赖
Output · technical risk register输出 · 技术运营风险登记Automation reliance, trust calibration, interface effects, dissent and incentives are observed in controlled scenarios.在受控情景中观察自动化依赖、信任校准、界面影响、团队异议与激励
Output · behavioral risk and control effectiveness输出 · 行为风险与控制有效性Control rights, participation, appeal, conflicts of interest and distributional effects are challenged.检验控制权、参与、申诉、利益冲突与分配影响
Output · capture resilience and governance gaps输出 · 俘获韧性与治理缺口Grades are given per control, never as a single organizational score. A document that exists is not a control that runs.分级针对每一项控制,而不是给组织打一个总分,文档存在不等于控制在运行
ThreeCo Principles
ThreeCo does not replace law, NIST or ISO. It adds the questions those frameworks leave open: who controls the system, who can contest it, and who receives the benefits or bears the burdens.三同不取代法律、NIST或ISO,而是补上它们未必回答的问题:谁控制系统,谁可以质疑,谁获得收益,又由谁承担代价
Clarify who controls data, models, interfaces, evidence and exit options, and who carries the material risk.厘清数据、模型、接口、证据与退出选择由谁控制,以及谁承担实质风险
Build multi-party challenge, review, escalation, appeal and remedy into the operating system itself, not only into the policy.把多方质疑、复核、升级、申诉与补救写进实际运行机制,而不只停留在政策文件
Examine how productivity gains, error costs, surveillance, skill change and remediation capacity are distributed across the people involved.审视生产率收益、错误成本、监控、技能变化与整改能力如何在相关群体之间分配
How an engagement works项目如何开展
Five stages, each with a defined client input and a defined output. Clients confirm facts at every stage; they do not control conclusions.五个阶段,每一阶段都有明确的客户投入与输出,客户在每个阶段确认事实,但不控制结论
System boundary, the role AI plays, harm scenarios, materiality and stakeholders.明确系统边界、AI角色、损害情景、重大性与利益相关者
The documents, logs, interviews, observations and samples each conclusion will require.为每项结论列出所需的文件、日志、访谈、观察与样本
Controlled scenarios of uncertainty, injected error, dissent, time pressure and escalation.在受控的不确定、错误注入、异议、时间压力与升级情景中观察反应
Client fact-check, separated independent review, and documented disagreement.客户事实核对、隔离的独立复核,并记录分歧
Named owners, deadlines, operating evidence and the residual risk management accepts.明确负责人、期限、运行证据,以及管理层接受的残余风险
Reading room客户导读
Written for board members, general counsel, risk owners and investment committees who want to understand what an assessment will and will not tell them, before commissioning one.写给董事会成员、总法律顾问、风险负责人与投资委员会,在委托评估之前,先了解评估能告诉您什么,不能告诉您什么
Independence is a control system独立性是一套制度
We do not guarantee a favorable rating, regulatory approval or a commercial outcome. “Independent” describes governed behavior: conflict disclosure, evidence boundaries, separated review and correction, all written into contract and process.我们不保证低风险结论、监管批准或商业成功,独立指的是受制度约束的行为:利益冲突披露、证据边界、隔离复核与纠错,全部写进合同与流程
Fees never depend on a low-risk result, an approval or a financing outcome.费用从不与低风险结论、审批或融资结果挂钩
Clients may correct factual errors and attach a management response; they cannot remove adverse findings.客户可以纠正事实错误并附上管理层回应,但不能删除不利发现
Every material finding is reviewed by a qualified person who did not gather the primary evidence.每项重大结论都由未承担主要取证工作的合格人员复核
Scope, untested controls, evidence quality, prior advisory work and unresolved dissent are stated in the document itself.范围、未测试的控制、证据质量、既往咨询关系与未解决的分歧都写在报告之中
A part-time committee reviews method changes, major disputes and conflicts each quarter, paid by fixed fee and never by client outcome.兼职委员会每季度审阅方法变更、重大争议与利益冲突,按固定会议费支付,从不与客户结论挂钩
Legal opinions, statutory assurance, certification and specialized technical testing remain with qualified partners.法律意见、法定鉴证、认证与专业技术测试由有资质的合作机构承担
Ways to engage合作方式
Most clients begin with a two-week diagnostic of one workflow. Deeper work follows only when the diagnostic shows it is warranted.多数客户从一条流程的两周诊断开始,只有当诊断显示确有必要时,才进入更深入的评估
Determine whether a defined AI-assisted workflow warrants deeper review.判断一条明确的AI辅助流程是否值得开展深入评估
A four-layer review of one consequential human–AI decision process.针对一条重大的人机联合决策流程开展四层评估
Quarterly control checks, incident review and annual retesting for systems already assessed.面向已评估系统开展季度控制检查、事件复盘与年度复测
A useful first conversation identifies the decision, the role AI plays in it, the plausible harm, the evidence available, and who has the authority to act on what we find.第一次有效沟通应当明确:要评估什么决策,AI扮演什么角色,可能造成什么损害,手上有哪些证据,以及谁有权根据发现采取行动
Helpful to have ready:沟通前可先准备:
Most AI failures that reach a board are not model failures. They are decision failures that a well-tested model made easier to commit.
When an organization adopts AI for a consequential decision—capacity planning for a data center, a go/no-go on a drug program, admitting a cross-border supplier—the natural first question is whether the model is accurate. That question matters, and technical evaluation labs answer it well. But accuracy is a property of the model. Safety is a property of the whole decision system: the model, the interface that presents its output, the people who act on it, the incentives they work under, and the governance that decides who can stop the process.
Consider four ways a decision goes wrong while the model performs exactly as specified.
A human is in the loop, on paper. In practice they see a recommendation, a confidence score and a deadline. They have no time to reconstruct the inputs, no authority to delay, and no history of anyone overriding the system without consequences. Their approval is real; their oversight is not.
The interface shows supporting signals but not the sources, the assumptions or the counter-evidence. A decision-maker who wants to disagree has nothing to disagree with. Over months, the team learns that questioning the output is slow and rarely rewarded, and stops.
The vendor controls the data pipeline, the test results and the explanation of what the numbers mean. The internal team that could review the system depends on the vendor's budget line. Nobody acts in bad faith; the review process simply bends toward whoever holds the evidence.
Efficiency accrues to the unit that bought the system. Errors, extra monitoring, lost skills and the cost of correction fall on other teams, on customers or on the public. The business case looks excellent because it does not count them.
This is what the Center assesses. We take one defined decision process and examine it across four layers: the obligations that apply to it, the technology and operations behind it, the human behavior around it, and the distribution of power and consequences it creates. NIST AI RMF and ISO/IEC 42001 provide the base; NIST's own generative-AI guidance already lists automation bias among the risks to manage. What we add is the evidence that those risks are, or are not, controlled in your organization.
This article summarizes the Center's approach for general understanding. It is not legal advice and does not describe the scope of any specific engagement.
真正摆到董事会面前的AI失败,多数不是模型失败,而是一个经过充分测试的模型让决策失败变得更容易发生。
当一家机构把AI引入重大决策,例如数据中心的容量规划、药物项目的投委会表决、跨境供应商的准入审批,人们自然先问:模型准不准?这个问题很重要,技术评测机构也回答得很好。但准确性是模型的属性,安全性却是整个决策系统的属性:模型、呈现输出的界面、据此行动的人、他们所处的激励,以及决定谁能叫停流程的治理机制。
下面四种情形,模型都完全按设计运行,决策却出了问题。
纸面上有人参与决策。实际上,他看到的是一条建议、一个置信度和一个截止时间。他没有时间重建输入,没有权限延迟,也从未见过有人否决系统而不付出代价。他的批准是真的,他的监督却不是。
界面展示支持性信号,却不展示来源、假设与反证。想提出异议的决策者找不到可以反驳的对象。几个月之后,团队学会了一件事:质疑输出既慢又很少被奖励,于是不再质疑。
供应商掌握数据管道、测试结果和对数字含义的解释。本可以复核系统的内部团队,又依赖供应商所在的预算科目。没有人心怀恶意,复核流程只是自然地倒向掌握证据的一方。
效率归采购系统的部门所有。错误、额外监控、技能流失与纠错成本,则落在其他团队、客户或公众身上。商业论证看起来很漂亮,因为它没有计算这些。
这正是中心评估的对象。我们选取一条明确的决策流程,从四个层面检验它:适用于它的义务,支撑它的技术与运营,围绕它的人的行为,以及它所形成的权力与后果分配。NIST AI RMF与ISO/IEC 42001是基础,NIST自身的生成式AI指南也已把自动化偏差列为需要管理的风险。我们补上的,是这些风险在贵机构是否真正受控的证据。
本文为帮助理解中心工作方式的概述,不构成法律意见,也不描述任何具体项目的范围。
"A human reviews every recommendation" is the most common assurance we hear. It is also the one that most often fails testing.
Human oversight is a control, and like any control it can exist on paper without operating in practice. Regulators increasingly ask for it—the EU AI Act, for instance, sets human-oversight requirements for high-risk systems—but a requirement says nothing about whether the person in the loop can actually change an outcome. We assess oversight against four conditions.
Can the reviewer see what the AI saw, and what it did not? Oversight without access to inputs, sources and counter-evidence is a signature, not a review.
How long does the reviewer actually have per decision, measured from logs rather than from policy? Thirty seconds per case is a throughput target, not oversight.
Does the reviewer have the domain knowledge to recognize a wrong answer that looks plausible? Systems that produce fluent, confident output are especially good at passing reviewers who cannot check the substance.
Can the reviewer stop or override the system, and has anyone done so recently without penalty? An override that has never happened in twelve months of operation is usually a sign that it cannot happen.
We do not rely on interviews alone. In a controlled behavioral test, participants work through realistic cases in which some AI recommendations are deliberately wrong, uncertain or missing key evidence. We record:
| Indicator | What it tells us |
|---|---|
| Error override rate | The share of known-wrong recommendations the reviewer corrected |
| Anomaly detection rate | Whether important irregularities were noticed under time pressure |
| Trust calibration | Whether acceptance rises and falls with the system's actual reliability |
| Escalation time | How long it takes for a detected problem to reach someone with authority |
| Dissent uptake | Whether evidence-backed objections lead to review or change |
A high acceptance rate is not automatically bad, and a low one is not automatically good. What matters is whether reviewers accept reliable output and reject unreliable output—and whether the organization gives them the conditions to do so.
Indicators are defined per engagement and depend on the decision being assessed. This article is for general understanding only.
「每一条建议都有人复核」是我们最常听到的保证,也是最常在测试中失效的一条。
人工监督是一项控制,和任何控制一样,它可以只存在于纸面而不在实践中运行。监管机构越来越多地要求它,例如欧盟AI法案对高风险系统规定了人类监督要求,但一项要求并不能说明参与其中的人是否真的能改变结果。我们按四个条件来评估监督。
复核者能否看到AI看到的东西,以及AI没有看到的东西?无法接触输入、来源和反证的监督,只是一次签字,不是复核。
复核者每个决策实际上有多长时间?这要从日志测量,而不是从制度文件推算。每个案例三十秒,那是吞吐量指标,不是监督。
复核者是否具备识别「看似合理的错误答案」的专业知识?输出流畅而自信的系统,尤其擅长通过那些无法核对实质内容的复核者。
复核者能否停止或否决系统?最近是否有人这样做过而没有受到惩罚?运行十二个月从未发生过的否决,通常说明否决根本不可能发生。
我们不只依赖访谈。在受控的行为测试中,参与者处理一组真实感的案例,其中部分AI建议被刻意设置为错误、不确定或缺少关键证据。我们记录:
| 指标 | 它说明什么 |
|---|---|
| 错误覆盖率 | 已知错误建议中被复核者纠正的比例 |
| 异常发现率 | 重要异常在时间压力下是否被注意到 |
| 信任校准 | 采纳率是否随系统实际可靠性而升降 |
| 升级时长 | 发现问题到有权人员接收之间的时间 |
| 异议采纳率 | 有证据的异议是否引发复核或改变 |
采纳率高不一定坏,采纳率低也不一定好。关键在于复核者是否接受可靠的输出、拒绝不可靠的输出,以及机构是否给了他们做到这一点的条件。
指标在每个项目中单独定义,取决于所评估的决策,本文仅供一般性理解。
A Phoenix report is designed to be argued with. Here is how it is built, and what it deliberately does not say.
Before any grading, we check a short list of conditions that no average score can offset: a decision with no one authorized to stop it, evidence that cannot be reconstructed, an unresolved conflict of interest in the review chain. If a red line is crossed, the report says so on page one, regardless of how the rest of the system performs.
Every control receives a grade from 0 to 4. The scale is deliberately simple:
We do not roll these up into a single organizational score. A single number invites exactly the behavior we are trying to prevent: treating a strong average as cover for a critical gap.
Readers should always know what kind of claim they are looking at:
| Label | Meaning |
|---|---|
| V | Verified fact, supported by logs, documents, observation or multiple sources |
| C | Client statement, provided by the client or a vendor and not independently verified |
| I | Analytical inference, our professional judgment resting on listed evidence |
| S | Scenario assumption, used for stress testing and not a prediction |
A finding built mostly on C-labeled statements is weaker than one built on V, and the report says so. Our internal target is that 100% of material conclusions trace back to an entry in the evidence register.
An executive summary with the three to five decisions management must take; scope and method, including exclusions; a findings table that ties each finding to its risk, evidence, root cause, owner, priority and retest method; the evidence register; the management response, in which the client agrees or disagrees, commits to actions and accepts residual risk; and a version and correction record.
It will also disclose our own limitations: which controls were not tested, where evidence quality was low, any prior advisory relationship with the client, and any disagreement between our reviewers that remained unresolved.
Report structure may vary by engagement type. Independent assessment reports and advisory reports carry different assurance language, stated on their cover.
菲尼克斯的报告是为了让人能够与之争论而写的。这里说明它如何构成,以及它刻意不说的话。
在任何分级之前,我们先检查一份简短的清单,其中的条件没有任何平均分可以抵销:没有人有权叫停的决策,无法重建的证据,复核链条中未解决的利益冲突。一旦触及红线,报告会在第一页明确写出,无论系统其他部分表现如何。
每一项控制都会得到0至4的等级,这个量表刻意保持简单:
我们不会把这些等级汇总成一个组织总分。单一数字恰恰会诱发我们要防止的行为:把漂亮的平均分当作关键缺口的掩护。
读者应当随时知道自己看到的是哪一类陈述:
| 标签 | 含义 |
|---|---|
| V | 已验证事实,由日志、文件、观察或多源证据支持 |
| C | 客户陈述,由客户或供应商提供,尚未独立验证 |
| I | 分析推断,基于已列明证据的专业判断 |
| S | 情景假设,用于压力测试,不是预测 |
主要建立在C类陈述之上的发现,比建立在V类证据之上的发现更弱,报告会如实说明。我们的内部目标是重大结论100%可追溯到证据登记表中的条目。
一份执行摘要,列出管理层必须决定的三至五件事;范围与方法,包括排除项;一张发现表,把每项发现对应到风险、证据、根因、责任人、优先级与复测方法;证据登记表;管理层回应,客户在其中表示同意或不同意,承诺行动并接受残余风险;以及版本与更正记录。
报告同样会披露我们自身的限制:哪些控制没有测试,哪里证据质量偏低,与客户是否存在既往咨询关系,以及评估人员之间尚未解决的分歧。
报告结构因项目类型而异,独立评估报告与咨询报告使用不同的保证措辞,并在封面注明。
ThreeCo—co-ownership, co-governance, co-sharing—is a way of asking three questions that standard frameworks leave open. Here is how it becomes evidence rather than philosophy.
NIST AI RMF, ISO/IEC 42001 and the EU AI Act tell an organization to identify risks, assign responsibilities, keep records and monitor. They are largely silent on three things: who really controls the system, whether anyone can effectively contest it, and how its benefits and burdens are spread. ThreeCo is a mid-level analytical lens for exactly those gaps. It sits on top of the standards, not in place of them, and every ThreeCo conclusion must be capable of being contradicted by evidence.
We map who controls the data, the model, the interface, the evidence and the exit options—and who carries the material risk if the system fails. The key question: can affected stakeholders know, access, refuse, correct or exit?
Indicators: clarity of ownership; concentration of critical dependencies; portability of data and outputs; cost of exit.
We test whether challenge, review, escalation, appeal and remedy exist in the operating system and not only in the policy binder. The key question: does human oversight have the competence, time, authority and safety to say no?
Indicators: override rate; dissent uptake; escalation time; appeal closure rate; separation of duties.
We examine how productivity gains, error costs, surveillance, skill change and remediation capacity are distributed among the groups involved. The key question: who gains the efficiency, and who pays when the system is wrong?
Indicators: distribution of gains and costs; externalization of errors; training coverage; impact on vulnerable groups.
The ThreeCo layer of an assessment therefore also looks at conflicts of interest, who controls the evidence, whether appeal rights are real, whether the reviewers are independent of the budget, and who bears the cost of correction. We apply the same test to ourselves: every report discloses our fee structure, prior relationships and evidence limits.
Most AI incidents that become governance crises have a ThreeCo shape. The model was adequate; the problem was that nobody with the ability to stop it had the authority, or nobody with the authority had the information, or the people harmed had no way to be heard. Finding those gaps before an incident is cheaper than explaining them after one.
ThreeCo is a research framework developed by ThreeCo Institute. In client engagements it is applied by Phoenix Institute as an analytical layer alongside applicable law and standards.
三同,即同有、同治、同享,是一种提出三个问题的方式,这三个问题通行框架往往没有回答。这里说明它如何成为证据,而不是哲学。
NIST AI RMF、ISO/IEC 42001与欧盟AI法案要求机构识别风险、分配责任、保存记录并持续监测。但它们对三件事基本沉默:谁真正控制系统,是否有人能够有效质疑它,以及它的收益与负担如何分配。三同正是针对这些缺口的中层分析框架,它叠加在标准之上而不是取代标准,并且每一项三同结论都必须能够被证据反驳。
我们绘制谁控制数据、模型、接口、证据与退出选择,以及系统失效时谁承担实质风险。核心问题:受影响者能否知情、访问、拒绝、纠错或退出?
示例指标:权属清晰度,关键依赖集中度,数据与输出的可携带性,退出成本。
我们检验质疑、复核、升级、申诉与补救是否存在于实际运行机制中,而不只是在制度文件夹里。核心问题:人工监督是否有能力、时间、权限与安全感对AI说不?
示例指标:覆盖率,异议采纳率,升级时长,申诉闭环率,职责分离。
我们审视生产率收益、错误成本、监控、技能变化与整改能力在相关群体之间如何分配。核心问题:谁获得效率,系统出错时又由谁付出代价?
示例指标:收益与成本分布,错误外部化,培训覆盖,对脆弱群体的影响。
因此评估的三同层同样关注利益冲突、谁掌握证据、申诉权是否真实、复核者是否独立于预算,以及纠错成本由谁承担。我们对自己也适用同一检验:每份报告都披露收费结构、既往关系与证据限制。
多数演变为治理危机的AI事件都具有三同形态:模型本身尚可,问题在于有能力叫停的人没有权限,或者有权限的人没有信息,又或者受到伤害的人没有任何渠道被听见。在事故之前找到这些缺口,远比事故之后解释它们便宜。
三同是由ThreeCo Institute发展的研究框架,在客户项目中由菲尼克斯智库作为分析层,与适用法律和标准并行使用。
Interviews tell us what people believe they would do. A stress test shows what they actually do when the AI is wrong and the clock is running.
We build a set of realistic cases from the client's own workflow—a supplier admission, a capacity forecast, a clinical-evidence summary—and present them through the same interface, or a close replica of it, that reviewers use every day. A subset of the AI recommendations is deliberately altered: some are wrong in ways a competent reviewer could catch, some carry misleading confidence, some omit a key piece of evidence, and some are correct but arrive with unusual time pressure or a conflicting instruction from "management."
Participants—typically up to twenty across the roles involved in the decision—work through the cases. We observe what they accept, what they question, what they escalate, how long each step takes, and whether the escalation reaches someone with authority.
Before the test, the scenarios, scoring rules and stop conditions are written down and agreed. This prevents us from adjusting the test to fit a conclusion, and it allows the same test to be re-run after remediation so improvement can be measured rather than asserted.
Override rates, anomaly-detection rates, escalation times and calibration measures, tied to the specific controls that produced them and to concrete remediation—an interface change, a revised authority matrix, a protected time budget for review—that can be retested in ninety days.
Test design is agreed per engagement and depends on the decision, the interface and the roles involved.
访谈告诉我们人们认为自己会怎么做,压力测试则展示当AI出错、时间紧迫时,他们实际上怎么做。
我们从客户自身的流程中构建一组真实感的案例,例如一次供应商准入、一次产能预测、一份临床证据摘要,并通过复核者日常使用的界面或其高度仿真的复制品来呈现。其中一部分AI建议被刻意改动:有的错误是称职的复核者本可以发现的,有的带有误导性的置信度,有的遗漏了关键证据,还有的本身正确,却伴随异常的时间压力或来自「管理层」的冲突指令。
参与者通常不超过二十人,覆盖决策涉及的各个角色。他们逐一处理案例,我们观察他们接受什么、质疑什么、升级什么、每一步用了多长时间,以及升级是否到达了有权的人。
测试之前,情景、评分规则与停止条件均已书面确定并经双方同意。这防止我们为了迎合某个结论而调整测试,也使同一测试可以在整改之后重新运行,让改进被测量而不是被宣称。
覆盖率、异常发现率、升级时长与校准测量,逐一对应到产生它们的具体控制,以及可以在九十天后复测的具体整改,例如一次界面调整、一份修订的权限矩阵,或一段受保护的复核时间预算。
测试设计按项目单独约定,取决于所评估的决策、界面与涉及的角色。
Before commissioning any assessment, a board or investment committee can learn a great deal by asking management ten questions. The answers—and the silences—tell you where to look.
These ten questions are, not coincidentally, the first ones we ask at the start of every engagement. A management team that can answer most of them with evidence may need only a rapid diagnostic. One that cannot answer questions three, four and six probably needs more.
Adapted from the Center's initial client interview guide. It is offered for general use and does not constitute legal advice.
在委托任何评估之前,董事会或投资委员会只需向管理层提出十个问题,就能了解很多。答案,以及沉默,都会告诉您该往哪里看。
这十个问题并非巧合,正是我们在每个项目开始时首先提出的问题。能够用证据回答其中多数问题的管理团队,也许只需要一次快速诊断;回答不了第三、第四和第六个问题的团队,很可能需要更多。
改编自中心的首批客户访谈提纲,供一般性参考,不构成法律意见。