Using an LLM for Humanities research

dabbling in the dark arts

I’ve been a bit of a late adopter of LLMs, but considering that I’ve built my career on tech and knowledge work, this summer I decided to at least familiarize myself with what LLMs can do. I’m aware of the ethical concerns surrounding LLMs (does training an LLM on stolen texts constitute a copyright violation? It feels like it should), and environmental issues. But I’m also an independent researcher who doesn’t have university funding to hire 1-5 research assistants.

I’ve used Claude Sonnet for things like practicing my Spanish and summarizing key points of articles. I found its arguments limited and forced. But in the past two weeks, I’ve been working with Claude Fable and it’s leaps and bounds better than Sonnet. In an effort to “work with the garage door up,” I want to show everyone what I’ve been working on.

1795 etching depicting the author using an LLM to summon monsters and pollute the air while others look on in horror. From the British Museum via Wikimedia Commons.

What it’s worth using Sonnet for:

Thematic search of a corpus

Primary texts

As a researcher, I have a several corpi on hand. One is created by my friend Michael Austin. He downloaded the full PDFs of early LDS periodicals (like Women’s Exponent, Juvenile Instructor, and many others). He ran Adobe Acrobat’s OCR on them to make them text-searchable. He also vibe-coded a database with Chat-GPT to search them all for keywords. he’s currently expanding the corpus to include public domain works of LDS literature and local newspapers. This is an excellent database on its own. However, for my research, keywords weren’t helping a lot.

I’m interested in literature from LDS periodicals that retells Book of Mormon stories, and short stories about the pre-existence or the afterlife. Searching “Nephi” gives some stories about Nephi, but it also includes places that mention people named Nephi, or that mention the town of Nephi. Similarly, for speculative literature, “ghost” turns up a lot of sermons on the Holy Ghost. Fable was able to tell the difference between sermons and fiction and helped me find several stories that are what I actually want to read.

I know, searching is half the fun! But I would rather search among 40 stories for the three I want to write about than search thousands.

Secondary Texts

My other data horde collection of texts is of literary criticism. I use Readwise Reader (more about this in a different post) to collect and tag PDFs and websites that I want to use later in my research. It also keeps track of my highlights. Claude Fable was able to access my Readwise Reader documents. I asked it to summarize my articles on Mormon literary criticism and aesthetics. I had previously asked Sonnet to do this, so this provided for an interesting contrast. You can read an extract of this conversation with Fable here. In particular, I hadn’t had time to read the articles I’d collected on Mormon aesthetics. Fable’s analysis provided arrows to articles that I could start my research with, should I return to this topic:

The tradition [of Mormon aesthetics] exists but refuses to consolidate — and your own documents say so, almost in unison. Bradshaw opens by noting scripture provides no aesthetic and artists avoid the topic for fear one would be imposed on them. Hicks argues against the whole project: continuing revelation means doctrine is “built in a river” where “some things that seem to be settled give way as the stream deepens,” so a settled Mormon aesthetics would be a false “orthodoxy of the beautiful.” And Morris’s meta-survey observes that nearly every attempt “lands in the same place... a call to arms for more work to be done.”

[…]

Not “Mormonism lacks an aesthetic theology” but “Mormonism has an anti-systematic aesthetic tradition, and the anti-systematicity is itself doctrinally motivated.” Hicks’s river might be the most Mormon aesthetic statement in your whole library.

Sources: Handley, “A Poetics of the Restoration”, Hicks, “Three Puzzles for a Mormon Aesthetics”, Bradshaw, “Reflections on the Nature of Mormon Art”, Morris, “Attempts at Mormon Aesthetics”, Rees, “The imagination’s new beginning”

That said, when I asked Fable to identify key texts in Mormon literary criticism, I could tell that I had biased it towards my research interests, based on the kind of articles I’ve been saving for my own interests. I asked it to use several different methods of identifying the “importance” of an article, but I think it leaned on the number of citations an article had more than other measures (so it was biased towards older and more “prestigious” texts). Still, this is a great starting point if I ever want to create an anthology of Mormon literary criticism.

Identifying several key texts in a field

Summarizing texts I already have is one thing. How does Fable do at giving me a reading list for fields have no expertise in?

I asked Fable to identify articles I could read in four hours to catch me up on the latest 15 years of research on first-year writing. It gave me this list, which was limited to texts it could find online. The list was an excellent way to get my headspace back into the buzzwords for rhetoric and composition, and I was able to compose answers to interview questions that helped me remember my previous training and sound like I knew what I was talking about. But I knew that this list was limited. My favorite article from the list, Downs & Wardle’s, “Teaching about Writing, Righting Misconceptions,” cited some articles that were focused on issues I think are extremely important (on knowledge transfer and how to give feedback that students will listen to).

Meanwhile, my friend who also teaches first-year writing pedagogy generously sent me her class’s reading list. The articles from it that I was most interested in were the same articles I had started to find from the Downs and Wardle article. However, there was a TON of stuff that Fable had missed because it couldn’t search in academic journals behind a paywall (and also, it probably stopped after finding “enough” stuff for me to read).

I also asked Fable to identify 1-2 books or articles in key areas of English literature to help catch me up on thought in those areas (should I ever end up teaching Shakespeare, for example). The list looks interesting, but I doubt that it has identified the most useful texts—however, they are good enough that I would probably learn a lot by reading them.

What Fable is bad at

There are things that Fable is very bad at. For example, when I asked it to download 12 PDFs from archive.org, it took longer, and took more usage credits, than I did just doing it myself. I also asked it to write a first chapter of the book it wanted to write contrasting Mennonite, Catholic, and Mormon literary criticism. The result was interesting, but underwhelming. I felt excited to read it because I want to read more on the topic of Mormon literature as a kind of religious literature! But there’s a kind of emptiness, because even if I taught this LLM to be a better writer, it’s not like it’s going to go on to teach high school English or present at MLA.

We can use LLMs as a tool to speed up research at a large scale, similarly to how we use library databases and full-text searches. When it comes to learning about a new field, any kind of anthology or systematic winnowing of texts is helpful to reduce overwhelm, but humans are still experts with valuable knowledge, and asking them for help is one way that we keep our intellectual communities alive.