This post explores a little experiment using OpenAI’s GPT3, Google Collab, Python and this interesting hack to summarize CJEU cases. And it wasn’t a total failure! I originally posted this on LI in February and forgot to post it here. It’s also got new and improved updates. If you want to just look at the newest shiniest version of the code, you can go here: https://codeberg.org/privacat/GPT3CaseParser/src/branch/main/caseparserv3.py)
The Problem: I Am Lazy, and CJEU Cases are Dense
Everyone is buzzing over Large Language Models (LLMs), and rightly so. (Well, not everyone – my boss Daragh O Brien argues that ChatGPT may well lead to the Enshittening of Knowledge).
In my opinion, one major benefit LLMs could provide us as over-taxed and exhausted humans is the ability to summarize cases, statutes, contracts, etc., both for attorneys and lay people. There are a few services that seem to tease this as an offering (Spellbook.legal and CaseText’s Co-Counsel) and I know others like Michael Bommarito and Jillian Bommarito, CPA, CIPP/US/E have ideas on how to make this a reality. Still, there’s really nothing I can play around with yet (if anyone wants to let me test their tools, I will be forever grateful and provide targeted feedback, hmu).
After ChatGPT was launched late last year, I immediately began arguing with myself over whether I should try coming up with my own legal chatbot of sorts — to read and summarize the dense cases put out by the CJEU, US Supreme Court, EDPB, and various regulators, as well as the seemingly endless supply of data processing addendums, privacy notices, and contracts that I have to read daily.
While I am programming-curious — I can articulate what I want a machine to do, and sometimes can understand and manipulate already written code — I cannot, no matter how much I wish it to be so, write this stuff myself. Thus, the idea of doing the heavy lifting to code and train a ML model sounded about as realistic and likely as trying to do brain surgery on myself.
Enter GPT3 and Google Collab!
On Saturday I started to look around a bit, and logged back in to OpenAI to play with the GPT3 playground. Editors’s Note: I have since upgraded to a paid account, and now use GPT4, which is actually much better.
Like ChatGPT, GPT3/4 has a fixed corpus it knows about, going back to around 2021, but you can get it to summarize new information as well. However, GPT3 has a rather annoying upper bounds limit of 4,000 tokens in the Playground. A ‘token’ is roughly 4 characters of text, and this is distributed across both the input text and the result. While that’s not a big deal for a short prompt question (‘Will you Destroy all Humans, GPT3?’) it makes it nearly impossible to use as a tool for doing what I want to do — summarizing dry, overly-wordy legal things.
Right. So, I can’t code, but I can do research. I started digging around and thought to myself ‘Surely, some programmer-type has come up with a nifty solution to this problem’. And sure enough, someone has — here’s the author (Sung Kim)’s write up: https://blog.devgenius.io/how-to-get-around-openai-gpt-3-token-limits-b11583691b32
Now, our purposes differ slightly — he was trying to summarize meeting minutes, while I was hoping for an assist with cases — but we both functionally have the same problem — really large blocks of text that we need to get the gist of. Kim figured out that with a little bit of Python-fu, one could cleverly break up a large document into small enough chunks, summarize those chunks and then get GPT3 (ver. 3.5) to return a full summary of a document as a whole — a way to work around the token limit.
He cleverly put this in Google Collab, which actually makes it easier for non-programming types to understand how the actual code was working (and, because this is Python, easily figure out where Python inevitably broke).[1] The code didn’t work out of the box, and I did have to make a few minor modifications (and later, with Husbot’s help, a few more substantial modifications after that when we moved it to Python). I’ll discuss that in another post. I also changed the question prompt at the beginning to be a bit more descriptive, as I wasn’t getting good details on the factual nature of the case.
Our new question prompt reads:
Summarize this Case, including factual details, and any interesting quotable language
All told, the Collab project, after a few tweaks, ran like a champ!
So it was now time to test. And for this, I wanted to test on two cases that I could easily read and confirm after my own review. The first case was X-FAB Dresden GmbH & Co. KG v FC (9 February 2023) and the second was Leistritz AG v. LH (22 June 2022). Both of these concern the termination of DPOs, and balancing national employment law against Article 38(3) GDPR.
And the Results?
… the results are, actually better than I thought they’d be. While I’ve added paragraph breaks, here’s the summary for X-Fab:
This case involves a conflict between a Member State’s national legislation and the second sentence of Article 38 (3) of the General Data Protection Regulation (GDPR).
The issue is whether the national legislation, which provides that a controller or processor may dismiss a Data Protection Officer (DPO) who is a member of staff solely where there is just cause, is compatible with the GDPR.
The Bundesarbeitsgericht (Federal Labour Court, Germany) referred four questions to the Court of Justice for a preliminary ruling, which focus on the interpretation of the GDPR and whether or not the dismissal of a DPO by the controller, who is their employer, is subject to certain conditions set out in the BDSG.
The Court of Justice concluded that the second sentence of Article 38 (3) of the GDPR must be interpreted as not precluding national legislation which provides that a controller or a processor may dismiss a DPO who is a member of staff of that controller or processor solely where there is just cause, even if the dismissal is not related to the performance of that officer’s tasks. Additionally, the court determined that a conflict of interests, as provided for in Article 38 (6) of the GDPR, may exist where a DPO is entrusted with other tasks or duties, which would result in him or her determining the objectives and methods of processing personal data on the part of the controller or its processor.
The court concluded that it is for the national court to determine, case by case, on the basis of an assessment of all the relevant circumstances, in particular the organisational structure of the controller or its processor and in the light of all the applicable rules, including any policies of the controller or its processor, whether a conflict of interests exists.
And here’s the summary for Leistritz AG:
The Court (First Chamber) ruled that the second sentence of Article 38 (3) of the General Data Protection Regulation (GDPR) must be interpreted as not precluding national legislation which provides that a controller or a processor may terminate the employment contract of a data protection officer, who is a member of their staff, only with just cause, even if the contractual termination is not related to the performance of that officer’s tasks, as long as such legislation does not undermine the achievement of the objectives of the GDPR.
The Court clarified that the European Union is to support and complement the activities of the Member States in the field of the protection of workers in cases where their employment contract is terminated, and that it is by means of directives that the Parliament and the Council may lay down minimum requirements in that regard.
However, each Member State is free to lay down more protective specific provisions on the termination of a data protection officer’s employment contract, in so far as those provisions are compatible with EU law.
Now, GPT3 missed some things — including more of the substantive argument as it relates to Article 38 in the Leistritz case. That means, I’ll need to continue tweaking things, such as specific words in my prompts.
GPT3 also doesn’t really summarize the factual details of the parties at all (as a Stockholm-Syndrome survivor of Legal Research and Writing, this made me twitch violently). It also misses a few juicy sentences by the Court that I would have included, as well as some refinements on context. But overall, it gets the holding mostly correct, highlights the legal issues at play, and is clear. That’s 80% of the way to a better understanding of the case. I’ll take it.
Final Thoughts
Personally, I don’t think lawyers and law-adjacent types like DPOs have much to fear from these tools, nor do I share Daragh’s pessimism and worry. In other words, I don’t think that LLMs will lead to the enshittening of knowledge, any more than use of any new technology has done in the past. That’s not to say that LLMs and advances in machine learning won’t be abused by some, just as some people abused calculators when they replaced slide-rules, and Wikipedia when it replaced print encyclopedias.
Still, the pearl clutching and agonizing, while good for eyeballs and page views, isn’t likely to bear out. It didn’t really bear out when the steam engine came into existence, or when we first obtained telephones. We haven’t as a collective become massively stupider because TV and the Internet exist (I’d argue that in general, education and knowledge have actually improved since, as information is now widely available to people in parts of the world that had very little access before). It is, however now much easier to spot every town’s share of idiots, posers, and charlatans.
Right now, LLMs and other ML tools for art and video like DALL-E and Synthesia, are still very early days. People who rely on them blindly and disengage their brains will discover very quickly that this is a foolish choice. And so will their clients. ChatGPT and the like aren’t perfect substitutes for human knowledge, and they won’t help you win in a legal case (notwithstanding what the jokers at DoNotPay claim).
But they don’t have to be. Ultimately, what I want is a tool that assists me to use my brain more effectively. Something that, for example, might kick off the initial brainstorm of a blog post, whitepaper, or email. Or a tool that summarizes a lengthy case and identifies key points to consider. I have no plans to stop thinking critically, or stop engaging with the material. But just like the calculator, cell phone, or even writing, I’m excited about the development of tools that can help me use my precious gray matter less on the basics, boilerplate, & bullshit, and more on providing useful, generative, insightful, and nuanced contributions to my clients and the world.
——
[1] In 20 or so years of dabbling with Python, I have yet to run anything posted on the internet successfully based on the code provided. Due to Python being Python (and things being written in different versions of Python), there’s always some dependency hell, undeclared variable, or some random code wonkery that breaks things.
[2] If you want to check out my cleaned up, working version, you can go here: https://codeberg.org/privacat/GPT3CaseParser. You will need to provide your own OpenAI API key. CaseParserv3.py can be found here. I have substantially re-done the lot of it (with GPT4’s help). It now runs _much_ more quickly, and provides a more detailed, easier to follow output. Here’s a summary of the Breyer case, for example:
Response:
Facts of the Case:
- Mr Breyer brought an action before the German administrative courts for an order restraining the Federal Republic of Germany from storing the IP address of his host system, after consultation with the public websites accessible to the German Federal institutions’ online media services.
- It was determined that the information retained in the logfiles after those sites have been accessed include the name of the web page or file to which access was sought, the terms entered in the search fields, the time of access, the quantity of data transferred, an indication of whether access was successful, and the IP address of the computer from which access was sought.
- Additional data available to the user’s internet service provider, allowing the identification of the user through the IP address, was taken into consideration.
Legal Issues Presented to the Court:
- Does Article 7(f) of Directive 95/46 prelude provision in national law under which a service provider may collect and use a user’s personal data without their consent only to the extent necessary in order to facilitate, and charge for, the specific use of the telemedium by the user concerned, and under which the purpose of ensuring the general operability of the telemedium cannot justify use of the data beyond the end of the particular use of the telemedium?
- Is a dynamic IP address registered by an online media services provider when a person accesses a website, which the service provider makes accessible to the public, personal data within the meaning of Article 2(a) of Directive 95/46 in relation to that provider, where the latter has the legal means which enable it to identify the data subject with additional data which the internet service provider has about that person?
Findings & Conclusions:
- Article 2(a) of Directive 95/46 must be interpreted as meaning that a dynamic IP address registered by an online media services provider when a person accesses a website that the provider makes accessible to the public constitutes personal data within the meaning of that provision in relation to the provider, where the latter has the legal means which enable it to identify the data subject with additional data which the internet service provider has about that person.
- Article 7(f) of Directive 95/46 must be interpreted as precluding legislation of a Member State under which an online media services provider may collect and use a user’s personal data without their consent only to the extent necessary in order to facilitate, and charge for, the specific use of those services by the user concerned, and under which the purpose of ensuring the general operability of those services cannot justify use of the data beyond the end of the particular use of them.
And here’s a similar response from GPT4 (although, IMHO, not as detailed) when I manually dumped the case in based on size limits and some additional prompt tweaks.
- ‘Facts of the Case’:
The case involves an online media services provider in the Federal Republic of Germany, which registers dynamic IP addresses of users accessing its website. The provider lacks the additional data necessary to identify users directly. The referring court’s questions are based on the premise that the dynamic IP addresses and the date and time of website access do not allow the service provider to identify the user without the additional data held by the user’s internet service provider. - ‘Legal Issues Presented to the Court’: The Court was asked to address two main legal issues:
- Whether a dynamic IP address registered by an online media services provider constitutes personal data within the meaning of Article 2(a) of Directive 95/46.
- Whether Article 7(f) of Directive 95/46 precludes national legislation under which an online media services provider may collect and use a user’s personal data without their consent only to the extent necessary to facilitate and charge for the specific use of those services, and whether the purpose of ensuring general operability of those services can justify the use of the data beyond the end of the particular use.
- ‘Findings & Conclusions’: The Court concluded that:
- A dynamic IP address registered by an online media services provider constitutes personal data within the meaning of Article 2(a) of Directive 95/46, in relation to that provider, where the provider has the legal means to identify the data subject with additional data held by the user’s internet service provider.
- Article 7(f) of Directive 95/46 precludes national legislation that allows an online media services provider to collect and use personal data relating to a user of those services, without their consent, only to the extent necessary to facilitate and charge for the specific use of those services by that user, even though the objective of ensuring the general operability of those services may justify the use of those data after consultation of those websites.