A legal expectation
In several jurisdictions, individuals subject to an automated decision may have a right to an explanation of how that decision was reached. This turns interpretability from a nice to have into a compliance obligation for systems that affect people, such as credit or hiring.
What an explanation must offer
- The main factors that drove the decision for this person.
- Enough detail that the person can contest an error or unfair outcome.
- Often a path to a human review rather than a final automated verdict.
Counterfactuals as a tool
A practical form is the counterfactual explanation: it states the smallest change that would have flipped the decision, such as a higher income threshold. It is actionable and reveals the boundary without exposing the entire model.
Engineering implications
- Log the inputs and the model version behind every decision.
- Prefer models or attribution tools that yield faithful per decision reasons.
- Avoid opaque pipelines where no honest explanation can be produced.
Key idea
A right to explanation can legally require systems to give individuals the main factors behind an automated decision and a way to contest it, with counterfactual explanations offering an actionable and faithful form.