A caseworker receives an AI recommendation with a confidence score. The interface offers “accept” and “dismiss,” but the caseworker has no access to the evidence, no time to check the model’s reasoning and no safe route to report a recurring error. The final click belongs to a human. The conditions for meaningful human judgment do not.
Agency as a design requirement means building systems so people can understand, direct, question and change consequential action. The phrase is the Institute’s design synthesis, not a single validated psychological scale. It draws on research about human agency, autonomy and human-AI interaction. The practical test is whether people have usable authority and resources, not whether a product says “human in the loop.”
Agency belongs to an interaction
Bandura’s social cognitive account describes people as agents who can anticipate, reflect, regulate action and influence their environments. That account does not picture people as unconstrained. Personal beliefs, behavior and conditions interact. In an AI setting, the interface and institution become part of those conditions.
A person can feel capable of using a tool yet lack the right to change its output. Another can have formal authority to overrule the tool but little information with which to do so. Self-efficacy, autonomy and institutional authority are related but different. Design should identify which one is missing before promising “empowerment.”
Ryan and Deci’s self-determination theory adds the distinction between acting with endorsement and acting under pressure. A worker may accept a recommendation to avoid a performance penalty, even when an override button is visible. The visible click cannot by itself establish that the decision was freely endorsed or well informed.
A button is not a decision right
Interfaces often display a choice at the end of a process. If the user cannot inspect relevant data, adjust the task, request another explanation or appeal the result, the choice may be thin. A real decision right includes information, time and consequences. It also includes clarity about who remains accountable after the button is pressed.
Amershi and colleagues developed human-AI interaction guidelines covering expectations, errors, feedback and control across the life of an interaction. The guidelines are a design resource grounded in prior research and evaluation, not proof that any product following a checklist produces agency. They prompt questions about what the system shows before, during and after a recommendation.
For the caseworker, a useful system might show what information was used, where the model is uncertain and how to correct a mistaken input. It would record a reason for override without treating every override as failure. These are design hypotheses that need testing in the actual workflow.
High automation and high human control
Ben Shneiderman argues for human-centred AI that can combine strong automation with strong human control and reliable governance. The framing resists a false choice between no automation and complete machine authority. A system can automate repetitive work while leaving people better able to set goals and handle exceptions.
That possibility is not automatic. If automation hides the steps needed to understand an unusual case, the human reviewer may become a ceremonial approver. If the interface floods the reviewer with details, the review may be technically possible but practically unusable. Effective control is a task-specific design and organizational achievement.
The Institute’s synthesis is to ask where control is exercised: before the system runs, while it proposes options, after an error and when the institution revises the system. A single override control at the end covers only one moment. People affected by the output may also need a route to contest the result.
Accuracy is not the whole team outcome
Bansal and colleagues studied AI advice in a setting where a person can accept a recommendation or solve a task themselves. Their analysis and experiments show that the most accurate model alone need not yield the best human-AI team performance. Predictability, verification cost and the human’s own strengths matter. The result is specific to the studied teaming arrangement, not a general rule to choose a less accurate model.
A caseworker needs to know when a recommendation is dependable and when to look more closely. A model whose errors are difficult to recognize can create more work than one whose limits are clear. The team metric should include the final decision, time needed to verify, costs of mistakes and whether people can handle unfamiliar cases.
A procurement process that compares products only by isolated model accuracy misses those features. The Institute’s synthesis is to evaluate the whole decision path. Who notices a wrong recommendation? How much work does correction require? What happens when the person disagrees? These questions are part of performance, not merely user satisfaction.
Overreliance and productive friction
People may follow AI advice too readily, especially when it is fluent or repeatedly correct. Buçinca, Malaya and Gajos studied cognitive forcing functions intended to reduce overreliance in AI-assisted decisions. Their work suggests that some forms of friction can encourage more careful thought in the tested tasks. It does not prove that every extra click improves judgment.
Friction can also burden users, especially people facing time pressure or accessibility barriers. A prompt that asks for an initial judgment before seeing AI advice may be useful for a high-stakes training exercise and impractical for every routine action. The design should target a known failure mode and measure both benefit and cost.
The goal is appropriate reliance. Trusting a tool when it is reliable and checking it when a mistake is plausible are both parts of competence. A system that trains people to distrust everything can waste attention. A system that hides uncertainty can invite passive acceptance. Good design helps users calibrate, not simply “trust more” or “trust less.”
Agency before deployment
People affected by an AI system should have a voice in defining the problem it is meant to solve. A workplace might ask whether a task needs automation, a simpler form or a change in staffing. If consultation begins after the model and success metric are fixed, participants may influence only cosmetic details.
This draws on participatory-design traditions as well as human-AI interaction research. The Institute’s proposal is to record which decisions workers and affected users could actually change. Their presence at a workshop is weaker evidence than a documented change to requirements, data practices or appeal routes.
Participation should include those likely to encounter errors and those who may have limited digital access. A system built only with expert users may work poorly for people at the edges of the service. Recruitment and compensation matter if participation is to be more than a symbolic exercise.
Agency while using the system
The user needs a clear task, legible recommendations and an understanding of what the system has and has not considered. They need to be able to pause, inspect evidence and pursue an alternative path when warranted. The interface should not make correction so costly that acceptance becomes the only practical choice.
Training is necessary but insufficient. A worker who can explain the model’s limitations may still be required to process cases too quickly to check them. The organization must allow time for review and protect people who flag errors. Otherwise literacy becomes knowledge without permission to use it.
The person affected by the decision also has interests. They may need to know that AI contributed, receive an understandable account of the decision and challenge inaccurate information. Their agency cannot be represented solely by the operator’s ability to click a button.
An appeal route should be tested from the affected person’s starting point. Can they find the right contact, provide the needed evidence and understand when to expect a response? Does an alternative channel exist for someone who cannot use the online form? A policy that allows appeal on paper but gives no usable route in practice offers a weaker form of control than its wording suggests. The Institute’s proposed evaluation would ask a small, diverse group to attempt the process and would record where they stop, what information is missing and whether the organization repairs the barrier.
Agency after an error
Errors are inevitable enough that recovery should be designed in advance. A user should know how to correct a record, retract an output and notify others who relied on it. The institution should examine recurring failures and change the system when a pattern appears. An individual override without a learning loop leaves the next person exposed to the same problem.
Documentation can help: what input was used, which model version ran, what recommendation appeared, who reviewed it and what was changed. Recording every detail without limits creates privacy and workload risks, so the log should serve a specific accountability purpose. The relevant question is whether a decision can be investigated and repaired.
Shneiderman’s governance perspective suggests responsibility at multiple levels, from product development to organizational oversight. A person at the interface cannot compensate for a system whose design and incentives make error reporting futile. Agency must be supported through institutional processes.
Skill development in the presence of AI
An assistant can help a novice perform a task now while changing the opportunities to learn that task. If every decision arrives as a recommendation, the novice may get less practice forming an independent judgment. But an AI system could also expose examples, provide feedback and help the novice compare alternatives. The direction depends on design and use.
The older cognitive-offloading literature suggests asking what is retained internally and what moves outside. Agency adds a question about authorship of the task: can the person set a goal, formulate a problem and reject a suggested path? A training programme should test performance later under altered conditions, including when the AI is unavailable or wrong.
These are research and design questions, not a general claim that AI always erodes expertise. A useful study would compare learning with different kinds of assistance and follow participants beyond one session. It would distinguish immediate speed from durable judgment.
How to evaluate agency
A survey asking whether people “feel in control” can be useful but incomplete. Behavioral evidence might include whether users can identify model errors, override them appropriately and explain a final decision. Institutional evidence might include whether error reports lead to change and whether affected people can appeal. Each measure captures a different part of agency.
A stronger evaluation would define the decision rights before testing the interface. It might compare a basic recommendation system with one that shows limits, supports correction and permits a documented alternative. It would record final decision quality, review time, overreliance, user burden and unequal effects across groups.
Success should not be inferred from a high override rate or a low one. Many overrides might show useful scrutiny or a poor model. Few overrides might show reliability or deference under pressure. Interpretation requires sampling cases, examining reasons and talking with people who use or are affected by the system.
Evaluation should also examine a case that the system was not designed to handle. What does the interface do when evidence is missing, a person disagrees with the input record or the recommendation falls outside an ordinary category? A rehearsed path for escalation gives users a choice other than acceptance or improvisation. Researchers can observe whether that path is found and used under realistic time pressure. A policy can promise escalation, but a test of actual cases reveals whether staffing, language and response time support it. The difference between a documented right and an exercised right is central to the claim that agency has been preserved.
Agency should also be tested when a user disagrees with a system. In a case review, the evaluator could record whether the person could locate the source information, understand the recommendation’s uncertainty, request a different path and see a correction take effect. A visible override button alone is weak evidence if pressing it leads to delay, penalty or a demand to justify disagreement without access to the machine’s evidence. Conversely, a team may benefit from automation when it removes routine work and preserves a real point for human judgment. The cited human-AI studies motivate this test but do not supply a universal agency score. The outcome depends on the task, stakes, worker skill and organization that receives an appeal.
What the evidence does not show
The cited studies do not validate one universal “agency score” for AI products. Human-AI guidelines are design resources; they are not guarantees. A cognitive forcing result in one task does not show that all friction is helpful. A team-performance model does not settle every workplace’s ethical or legal responsibilities.
Nor does formal human review prove meaningful human control. Information, time, authority and a route to change the system must be present and usable. Claims that a product “keeps humans in control” should be checked against those conditions in the setting where it is deployed.
The caseworker’s next case
In a better system, the caseworker can see the evidence behind the recommendation, understand its limits and ask for help with an unusual case. They can correct a mistaken input and know that a pattern of errors will be investigated. The person receiving the decision can contest it. Human judgment then has room to operate.
Agency in the AI age is not merely a reassuring interface label. It is a design and governance requirement: people need real means to shape goals, inspect advice, act on disagreement and repair mistakes. The measure of success is what they can actually do when the model’s answer is insufficient.
Research references
- Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44(9), 1175–1184. View source
- Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. View source
- Amershi, S., et al. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Paper 3. View source
- Shneiderman, B. (2020). Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction, 36(6), 495–504. View source
- Bansal, G., Nushi, B., Kamar, E., Horvitz, E., & Weld, D. S. (2021). Is the most accurate AI the best teammate? Optimizing AI for teamwork. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13), 11405–11414. View source
- Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188. View source