Write on

• Citation: Vol. 49 No. 3
• Publication year: 2023

WRITE ON Vol. 49 No. 3 Pg. 13 Vermont Bar Journal October 2023

Change in the Legal Writing Classroom: The NextGen Bar Exam and Generative AI

by Catherine Fregosi, Esq.

Legal education has been remarkably static for close to 150 years. In the late nineteenth century, law school Dean Christopher Langdell introduced the casebook and Socratic method at Harvard.^[1] At the same time, Langdell solidified the law school curriculum, requiring first-year students to take Civil Procedure, Contracts, Criminal Law, Property, and Torts.^[2] This 1L schedule likely sounds familiar: not only did other schools pick up Langdell's curricular and pedagogical innovations, many schools have stuck with the same basic schedule and teaching methods since the early twentieth century.^[3] The casebook and Socratic methods have been finessed, and law schools have added classes in statutory interpretation, legal research, and legal writing, but in many ways little has changed in the delivery of legal education since Langdell's deanship.^[4]

Change may^[5] now be coming from two directions, both of which could lead to shifts in law school curriculum generally and legal writing instruction specifically. First, the National Conference of Bar Examiners (NCBE) will roll out the new NextGen bar exam beginning in July 2026.^[6] Jurisdictions will adopt the NextGen bar exam on their own schedules, but current Uniform Bar Exam (UBE) jurisdictions (including Vermont) are expected to transition to the NextGen exam over the next few years. This is a big deal for legal writing and, perhaps even more so, for legal research.

The NextGen exam introduces direct skills testing in seven areas, including the core skills of legal writing and legal re-search.^[7] The new exam also clarifies the list of documents a test taker may need to analyze or produce during the exam. According to NCBE, the NextGen exam is "[designed to balance the skills and knowledge needed in litigation and transactional legal practice."^[8] To that end, the NextGen exam won't exclusively focus on memos and briefs, but will also do things like give students draft contract provisions and ask them to identify language that should be changed while suggesting ways to change the language consistent with governing rules, relevant facts, and a client's expectations.^[9] In short, the NextGen exam will evaluate writing and research skills to a degree not seen in earlier bar exams. This is a good thing, given that surveys have showed a desire for law school graduates to have practice-ready skills, including writing and research skills.^[10] The NextGen's skills orientation should match not only changes law schools have already made,^[11] but also the skills employers and others say they want from new graduates.

Enter generative AI.

Generative AI is a type of artificial intelligence that can create new content, such as text, images, music, or video, based on existing data. It uses machine learning techniques to learn from the patterns and structure of the input data and then generate new data that has similar characteristics. For example, generative AI can produce realistic images of faces, animals, or landscapes that do not exist in real life, or write stories, poems, or essays based on a given topic or prompt. Several generative AI programs reliably produce natural language responses to questions. For example, Microsoft Bing in Creative Mode, which uses a version of GPT-4, produced the three sentences at the beginning of this paragraph when asked to define generative AI. Most of the focus on generative AI in recent months has been on Open AI's ChatGPT interface.^[12] That model is a chatbot built on a generative pretrained transformer (GPT), a type of neural network that processes natural language and generates text based on a given prompt. These programs are trained on a large volume of digital text, and over time the program learns the features of text such that it can generate responsive natural language that mimics human writing or speech.

Generative AI such as GPT also improves over time. For example, in June 2023, I gave ChatGPT the following prompt: "Is a plow a point source under the Clean Water Act?" Legal research and writing classes at Vermont Law &Graduate School assign a similar question to first-semester students, and I wanted a general idea of what a student would see if they ran this simple legal question through generative AI.

ChatGPT produced three paragraphs in response to the prompt. The first paragraph began with the purpose of the Clean Water Act and finished with the definition of a point source. The text did not say where either point of information came from, so the reader would have no way of knowing if the Clean Water Act is statutory, or the point source definition is regulatory or from caselaw. The reader would also have no way of checking the accuracy of ChatGPT's answer without independent legal research.

The second paragraph of ChatGPT's response addressed the prompt directly. That paragraph began with a direct answer to the question posed, followed by a sentence giving the reasoning for the conclusion. The paragraph ended with a third and final sentence suggesting an alternative analysis might be appropriate in limited factual circumstances. The ChatGPT response then concluded by telling the reader that the answer to the prompt depended on many factors and the analysis given in ChatGPT's answer may not be reliable. The paragraph finished this way: "If you have specific concerns or questions about a particular situation, it's advisable to consult with legal or environmental professionals who can provide guidance based on the specific circumstances and relevant regulations in your jurisdiction."

Overall, the text of ChatGPT's June answer was clear but conclusory. The introductory paragraph framed the issue by providing the relevant purpose and definitions, but without legal references. The second paragraph responded to the prompt directly, but there were flaws here too. The ChatGPT response did not anticipate related questions or points, nor did it fully explain the conclusions it reached; the analysis was fairly superficial and further information would need to be elicited through additional questions.

I gave ChatGPT the same prompt in September 2023. In that response, it kept the legal disclaimer language, and it added more analysis and anticipation of related questions. There was marked improvement over the June response.

First, the September response is four paragraphs instead of three. While a longer response is not necessarily a better response, in this case the added text was helpful. Instead of beginning with the purpose of the Clean Water Act, the September answer begins with a conclusion sentence responding directly to the prompt. Legal writing students are taught to begin with a conclusion sentence, and ChatGPT used that style too. Like the June response, the September response leaves out all legal references. It is therefore still no help with specific sources for legal research, and independent legal research would be necessary to check ChatGPT's accuracy. But the September response otherwise improves on the content of the June response. ChatGPT's second and third paragraphs include more detailed analysis. The detail included anticipates follow up questions, providing terms that might be used for a follow up search and a more precise description of the reasons for ChatGPT's conclusion and possible alternative analysis. Like the June response, the September response concludes by noting the limitations of the given analysis and suggesting the reader seek further...