Privilege and AI hallucinations

AI use by lawyers has hit the headlines recently, with a number of high profile judgements on this from courts in both the UK and US.  

In the UK, the recent decision of the Upper Tribunal (Immigration and Asylum Chamber) in UK v Secretary of State for the Home Department [2026] is a key judgment in a line of cases addressing the use and pitfalls of AI in legal drafting. 

While the court was supportive of the proper use of AI tools, it also emphasised (in line with prior cases) that rigorous checks as to the veracity of their output are essential. If such checks are not carried out, firms and individual lawyers relying on such tools in preparing court documents run the risk of public censure and regulatory consequences.  

Of note, the court also held that uploading documents or confidential information into an “open source” AI tool, operates as a waiver of privilege.

What happened?

The judgment addressed two cases both of which involved incidents relating to the use of AI which resulted in false and inaccurate case citations being put before the Tribunal. 

1. The first case involved a regulated immigration adviser who “unknowingly” used fictitious case law obtained from Google’s ‘AI Mode’ in drafting an application for permission to appeal to the Upper Tribunal. The adviser had also put draft client emails and Home Office decision letters into ChatGPT. The adviser had self-reported to the Immigration Advice Authority and the Solicitors Regulation Authority (“SRA”), with the court indicating that, if he had not done so, they would have.
    
2. The second case concerned a solicitor who the court suspected of having used ChatGPT in drafting grounds for judicial review which had included erroneous references to cases that did not in fact exist. The court referred the solicitor to the SRA.

What did the court decide?

The judgment is clear that, properly used, AI tools represent “a step forward in legal practice”, with clear benefits for targeted research and large-scale disclosure exercises. However, both cases highlight that the use of AI does not change core professional obligations such as maintaining confidence in documents, supervision and the need for careful verification.

Privilege

The court found that uploading confidential documents into open-source AI tools, such as ChatGPT, is “to place this information on the internet in the public domain, and thus to breach client confidentiality and waive legal privilege”. The court differentiated the position with uploading such confidential documents into “closed source” AI tools where the confidentiality in the documents is protected. ‘Open source’ AI tools retain, store and use user inputs for model training meaning that inputting data and documents into the tool risks it losing confidence. ‘Closed source’ AI tools, by way of comparison, operate in a closed system whereby they do not use user’s prompts, data or files to train the model.

This decision is not surprising (and is consistent with comments on this same topic recently made by the Chancellor of the High Court, Sir Colin Birss in a speech to the City of London Law Society). However, they are still a useful clarification and reminder to practitioners of the risks posed to legal privilege of using ‘open source’ AI tools.

Verification and supervision

The judgment also more broadly followed the Divisional Court’s decision in R (Ayinde) v London Borough of Haringey; Al-Haroun v Qatar National Bank QPSC [2025] (“Ayinde”) where a barrister was referred to the Bar Standards Board after submitting to the court AI-generated arguments containing fictitious citations. In Ayinde, the Divisional Court characterised AI as presenting “risks as well as opportunities” and emphasised the need for its use to be subject to appropriate oversight within “well-established professional and ethical standards”. That balanced approach has since been reflected in guidance issued by the Law Society, which similarly recognises both the “boundless opportunities” and “new risks” associated with generative AI.

In both Ayinde and the recent decision of the Upper Tribunal, the court has reiterated that the obligation was on the lawyer submitting court documents to verify that cases put before the court exist, and that AI hallucinations just add another layer to the exiting and well-established principles of governance and supervision in the legal profession.