When the machine decides
Automated decisions are arriving in credit, insurance, hiring and public services. The question is not whether machines may decide, but whether the people they decide about are still treated as people.
There is a moment, familiar to anyone who has applied for credit, a policy, a job or a grant, when you realise the answer has already been given and nobody in the room gave it. The decision came from somewhere else. You ask why, and you receive a reference number.
That moment is becoming ordinary. Institutions across Southern Africa are moving decisions about people into models: who is approved, who is flagged, who is priced higher, who is called back. Much of this is sensible. A model can be more consistent than a tired person at the end of a long day. It can widen access where human gatekeepers were the bottleneck. It can catch fraud that everyone else would otherwise pay for.
But consistency is not the same as fairness, and speed is not the same as dignity.
What the law already says
South Africa did not wait for the AI debate to arrive. Section 71 of the Protection of Personal Information Act (POPIA) deals directly with automated decision-making. In summary, a person may not be subject to a decision that has legal consequences for them, or affects them to a substantial degree, where that decision is based solely on the automated processing of personal information intended to build a profile of them, such as their creditworthiness, reliability, health, conduct or performance at work.
There are exceptions. Such decisions may be made, for example, in connection with a contract, or where a law or code of conduct provides for them, but only where appropriate measures protect the person's legitimate interests. And the Act is specific about what those measures must include: an opportunity for the person to make representations about the decision, and enough information about the underlying logic of the automated processing to allow them to do so.
Read that slowly. It is, in effect, a design brief. The law asks every machine-led process to offer the people it decides about two things: a voice, and an explanation.
The law asks every machine-led decision to offer two things: a voice, and an explanation.
Dignity is a design property
Dignity in an automated decision is not a sentiment added at the end, in the tone of a rejection letter. It is a property of the system. It is either designed in, or it is absent. Three questions tell you which:
- Can the person tell that a machine made, or shaped, the decision?
- Can they understand why, in words that make sense to them, in a language they use?
- Can they reach a human being with the authority to change the answer?
If the answer to any of these is no, the institution has not automated a decision. It has automated a closed door.
Humans in the loop, not humans as decoration
The usual reassurance is that a human remains "in the loop". Look closely at what that loop often contains: a reviewer who approves case after case without the time, the information or the authority to disagree. That is not oversight. It is a signature on a decision someone else made.
We would argue that a human who cannot realistically overturn the model does not change the nature of the decision, whatever the process diagram says. Meaningful review needs three things: the reviewer sees what the model saw, the reviewer has time to think, and the reviewer is rewarded, not penalised, for disagreeing when disagreement is right.
The weight of history in the data
Models learn from the past. In societies shaped by exclusion, the past is not a neutral teacher. Data carries the outline of who was served and who was not, whose income was recorded and whose was informal, who had a long credit history and who never had the chance to build one.
A model that treats an absence of data as evidence of risk will quietly penalise exactly the people institutions say they want to include. That is not a malicious outcome. It is a design outcome, and so it can be designed against: by testing for it, by choosing what the model may and may not use, and by giving people a route to supply the evidence the data never captured.
Explanation is a service, not a disclosure
Institutions tend to treat explanation as a compliance artefact: a paragraph of logic, technically accurate and practically useless. A better standard is to treat explanation as a service to the person in front of you. What were the main factors? What could change the outcome? What can I do now, and by when will someone respond?
An explanation that answers those questions does more than satisfy section 71. It turns a refusal into a relationship. Many people will accept a hard answer they understand. Very few will accept a soft answer they cannot question.
What we imagine
Imagine every automated decision about a person arriving with a receipt: what was decided, the main reasons in plain language, how to respond, and a named point of human contact. Imagine model owners who can show, on request, how a decision was reached and what was done when it was wrong. Imagine institutions that measure not only how fast they decide, but how often their decisions are successfully challenged, and what they learned.
We have sketched one version of this as a concept. It is not a product. It is a provocation, and a standard we think institutions should be held to.
The test of an automated institution is not how many decisions it can make in a second. It is how it treats the person on the other side of a decision they did not expect.
This essay summarises POPIA section 71 for discussion. It is not legal advice.