The danger of using AI in legal submissions

In Bangladesh as well as in other common law jurisdictions, court decisions are heavily dependent on the pleadings, briefs, arguments, etc. submitted by lawyers. As a result, if the lawyer or self-represented litigant blindly relies on Artificial Intelligence (AI) tools, particularly generative AI, in the preparation of the case, there is a substantial risk that hallucinated citation of information may find its way into the decision without the judge ever typing a prompt.

The Supreme Court (SC) of the US state of Georgia has recently found that the impugned order of the trial court in Payne v State (2026) case contains several legal citations that either did not exist or did not support the arguments being made. Later, it was revealed that the draft of the proposed order was prepared by the prosecutor with the aid of AI tools. As such, the state justices urged trial judges to examine proposed court orders “with the understanding that artificial intelligence software, with all of its potential risks and benefits, may have been used”.

Tap here to add The Daily Star as a trusted source

It is noteworthy that the Court did not criticise the structure or format of the trial court’s order. Instead, it vacated the order because it contained numerous fictitious and misattributed case citations. This contamination calls into question the sanity of the process. It is also worth noting that the Payne case is not the first time that such an incident has occurred in the world. In March 2026, in another case ( In re Domestic Partnership of Torres Campos and Munoz), a California appellate court upheld a trial court order relying upon non-existent cases as the appellant had forfeited his right to challenge the decision.

In the illustrated cases, only the lawyers were punished, as it is a non-delegable duty where attorneys are personally responsible for the accuracy of all parts of their filing, whether or not they use technology to create it. Apart from the above illustrations, there is a long list of hallucinated content in judicial proceedings. Perhaps the widely publicised case was  Mata v. Avianca, where the attorneys presented court filings citing fake legal precedents generated by ChatGPT in 2023 in the Southern District of New York.

The truth is that despite having professional obligations and repeated caution against AI hallucination and numerous sanctions for citing fake cases, legal professionals keep citing non-existent cases and references due to their failure to adequately verify AI-generated output.

However, the question is why this trend is not stopping. One might wonder why the opposing lawyer did not bring such contamination before the appellate court. Usually, competing lawyers identify hallucinated citations and seek redress against that. However, the truth is that despite having professional obligations, and repeated caution against AI hallucination and numerous sanctions for citing fake cases, legal professionals keep citing non-existent cases and references due to their failure to adequately verify AI-generated output.

Human cognitive tendencies, such as heuristics, automation bias, confirmation bias, anchoring, framing effect, cognitive offloading, and so on, also contribute to this ongoing tendency. Additionally, AI hallucinations do not look like ordinary human mistakes. AI outputs look highly plausible, confident, and convincing, which motivates human psychology to trust them. As a result, there are high risks that the opposing lawyer and judge may not suspect each and every citation. Moreover, opposing counsel and judges are not immune to the same cognitive biases as the filing lawyer. The professional duty to verify only triggers when the lawyer or judge knows or suspects an error, whereas AI hallucinations are designed not to trigger suspicion, which makes it much harder to recognise. Nonetheless, there are techniques and specialised tools to reduce the risk of AI hallucinations, although many lawyers employing AI tools do not have sufficient training and expertise in it. That emerges as a great challenge in the legal profession.

Consequently, when a lawyer files a document with non-existent case information, that undoubtedly creates a serious problem in the court proceeding. But when a judge directly imports information from lawyers’ submissions and signs a court order without careful verification, the problem becomes even more serious. Those hallucinated contents become the court’s statement and carry legal authority.

Again, a lawyer’s error can be detected by an opposing party or by the superior court and subsequently remedied by the appellate court. The same cannot be done when a superior or appellate court has no further recourse for appeal. Although there is scope for review and correction of such misinformation, that also makes the process lengthy and costly.

As such, it is clear from the above discussion that the danger of issuing a corrupt verdict does not decrease in the age of AI. Judges may rely on their own conscience and may not even employ a single prompt to write these judgments but still end up with judgments based on non-existent authorities when they heavily rely on the lawyers’ submissions. Hence, it is high time to think critically as to how we can achieve efficiency in the judiciary without weakening fundamental principles of justice.

The writer is a Doctoral Researcher at the University of Galway, Ireland.