Skip to main content

11  Zero-shot Classification with LLMs

One of the newest and most powerful tools in the computational social science toolbox are Large Language Models (LLMs) such as ChatGPT and Claude. You’ve probably used them at some point with their chat interface, and maybe also through something like Claude Code. In this chapter we will see another side of LLMs: they can be used as powerful analysis tools. Since LLMs have a good understanding of natural language, asking them whether a text is negative can give a much more valid answer than using a dictionary-based sentiment analysis.

In this chapter you will learn how to use LLMs to do classification tasks such as sentiment analysis, and the importance of good prompts and validation. We will also discuss the ethical, climate and privacy considerations around LLMs and discuss the tradeoffs of proprietary vs local models.

TipLLMs in this interactive textbook

As a small technical note: we cannot actually call LLMs from the browser. So, for all examples in this chapter we have ‘mocked’ the LLM: when it looks like you call the LLM, it actually produces a canned response. Practically, this means that you cannot change the prompts in the interactive version of the book. Of course, you are still encouraged to play with the code itself to change the analysis or sampling or even to swap the model, as long as you use one of the models used in this chapter.

You are encouraged to play with LLMs in your local R installation as well. This will require either getting an API key for one of the proprietary models such as Claude or GPT, or installing an open model on your own computer, see the instructions in Section 11.2.2.

To see the script we used to pre-record the LLM answers, you can check the record_llm.R script on the book’s GitHub page as well as the resulting csv file.

Chapter objectives

In this chapter, you will learn how to use the ellmer package to call LLMs from R. You will also practice with existing tidyverse functions to analyse the results.

  • What is the general place of LLMs in the (computational) social science toolkit? (Section 11.1)
  • Using hosted LLMs via their API and local models via Ollama (Section 11.2)
  • Validating LLM results against a gold standard (Section 11.3)
  • Understanding how to improve and test instructions (prompts) (Section 11.4)
  • Using structured output (Section 11.5)

As always, the chapter ends with a multiple choice quiz organized per section that you can use to test your knowledge and understanding. You can also expand the table below to see which functions are covered in this chapter.

Function What it does Section
chat_ollama(), chat_openai() etc Create a new chat context for a given model Section 11.1, Section 11.2
chat$chat() Ask a single question to a LLM Section 11.1
interpolate() Build a prompt, e.g. based on instruction and text to classify Section 11.1, Section 11.2
parallel_chat_text() Automatically classify multiple examples with text output Section 11.1, Section 11.2, Section 11.3
type_object(), type_enum() etc Define the desired output structure Section 11.5
chat$chat_structured() Ask a single question to an LLM with structured output Section 11.5
parallel_chat_structured() Use structured output for multiple examples Section 11.5
unnest() Tidyverse function to turn a nested data frame into separate rows and columns Section 11.5

Data for this chapter

For this chapter, we will use the Dutch sentiment data described in Van Atteveldt et al. (2021), who published their dataset on GitHub:

As you can see, this has the sentiment and text (translated to English) for 284 headlines. They are all from (Dutch) economic news during the 2015 eurocrisis, and are coded by the three authors based on whether the headline describes the economy as doing well or poorly.

  • How would you approach a sentiment analysis of this data?
  • Do you think a dictionary should work well for this?
  • What question would you ask ChatGPT or Claude to do a sentiment analysis?
  • Would it work better on the Dutch original, or on the English translation?

11.1 LLMs in the social sciences

Many social science questions require analysing the content of messages or other cultural artefacts. This can range from analysing newspaper coverage or social media posts about political or societal issues, to emails or messages exchanged in an organization or even the profile pictures and chat messages in dating apps.

This is traditionally called content analysis, defined by Krippendorff (2018) as “making replicable and valid inferences from data to their context”. As this definition implies, this always entails an amount of inference or interpretation, since we are generally not interested in the usage of particular words (or image pixels), but rather in what these words can tell us about the sender, the receiver, or the broader society or culture in which they are produced.

Although manual content analysis is still needed for validation, automatic text and image classification is very useful for analysing larger data sets. We have seen dictionary approaches in the previous chapters, but for many research questions this turns out to not lead to valid inferences: there is no set of words that accurately captures anger, let alone the moral framing of an argument (cf. Van Atteveldt et al. (2021)). For this reason many scholars turned to machine learning, but due to the requirement for large training sets this is also not always feasible, even with advanced “deep learning” approaches (e.g. Laurer et al. (2024)).

LLMs have changed this equation quite dramatically: their understanding of natural language, including the ability to be prompted in natural language, means that they are very good at ‘zero-shot classification’: you can ask an LLM to classify a specific text using categories that you can describe in the instructions, and the LLM can often perform very well without needing any training data (Törnberg 2025).

As an example, the code below uses a locally running LLM to classify a sample headline:

As you can see, the model correctly identifies the headline as being negative. Looking at the code, you see we first create chat, which represents a link to the chosen model (Gemma). Next, we can call chat$chat to run single questions. You can also see that the chat object remembers the chat history, so the second question is answered in the context of the first.

To do a more systematic sentiment analysis, we can make some more structured instructions and apply them automatically to a sample of headlines from the example data:

The LLM we used did pretty well: it correctly classified 8 out of 10 example sentences, without any training data and with a minimal instruction in the prompt. This model (gemma4:e4b) is a new and fairly small model that runs well on my own computer, so it is heartening to see that even this model already does fairly well (on these 10 sentences, at least!).

Note that we created a new chat object, since the old chat would have remembered the question we asked above. The parallel_chat_text ensures that all headlines are prompted in a separate context, so the second headline is not influenced by the first.

Tipellmer syntax

The code above introduced three new functions from the ellmer package. Let’s take a second to look at these in detail.

  • First, we create a chat object using chat_ollama. This object encodes various choices such as what provider to use, but also which model and optionally other parameters like temperature. For example, we could also use gpt <- chat_openai(model = "gpt-4.1-nano", params=params(temperature=0)) to use OpenAI’s nano model with a temperature of zero (which makes the model more deterministic, but possibly less creative).
  • Next, interpolate was used to create the prompts to be sent to the model. In this case, it creates a single message containing both the instructions and the headline, similar to asking it as a single question in the chat interface. The interpolate function is very similar to the str_glue function we saw in Section 9.1.1: it fills the instruction object and headline column values into the template text, producing 10 prompts with different headline texts. If interpolate is used within mutate as it is here, it can directly access both columns of the data frame and global objects like instruction. Do note that because task instructions often need to use actual curly braces, the makers of interpolate decided to use double braces for its variables: {instruction}.
  • Finally, the parallel_chat_text function sends the prompts created with the interpolate function to the LLM specified in the chat= argument. The LLM then responds with a single message, (hopefully) following the instruction to use one of the valid responses (in this case, positive, negative, or neutral). This process is repeated for each row in the data frame, so each sentence is classified completely independently.

Note that we use set.seed(1) to set the random seed used in the sampling, to ensure that the same sentences are ‘sampled’ if we re-run (and to be totally honest, I picked seed 1 because the models show some interesting variation on these sentences). The random seed would normally not fix the responses, as the LLM itself is non-deterministic, even with temperature set to zero. However, since we used canned responses, they will always be the same if you run this in the book. If you try it on your own computer the answers can vary every run. Note that you are free to change the random seed and explore performance on other subsets, as we pre-recorded the answers for all headlines in the dataset.

It should be noted that the LLM is essentially a ‘black box’. We don’t really know what goes on inside, we can only observe the output it produces given our input. This underscores the need for validating the performance of your specific instructions and model on your specific task.

Moreover, the same input does not always give the same output: queries can be made more repeatable by setting a lower model temperature, but even at temperature zero the models are not fully deterministic (cf. Vallejo Vera and Driggers (2025)).

It can often be useful to ask for the ‘reasoning’ of the model as well, that is, ask the model to explain why it made a specific choice (see Section 11.5). Very often, this can be useful in spotting mistakes or fine-tuning the instructions. Ultimately, however, this is just more words produced by the same black-box LLM: we don’t actually know if the reason it states is actually its reason for making a choice (and Turpin et al. (2023) conclude that in some circumstances LLMs can give “unfaithful explanations” about their decisions, to the point of post-hoc rationalizing wrong decisions).

Concerns and considerations

When using LLMs for text classification, there are a number of things you should be aware of.

First, LLMs can make mistakes. Even with the newest models and the best prompts, they can get things wrong. It’s even worse if the answer is biased, that is, that the error is systematic rather than random. Since LLMs are trained on existing data, there is always a risk that inequities in the training material can be reinforced. For example, Vallejo Vera and Driggers (2025) found that LLMs are biased when coding political text: the same statement on immigration was consistently coded more negatively when attributed to a right-wing party than to a left-wing party, even with explicit instructions to ignore the party label.
For this reason, it remains important to validate the results of the classification, and especially to check that performance is similar across categories of interest (e.g. gender, ethnicity, or party).

Second, there are issues of privacy and reproducibility, especially when using proprietary models accessed via an API. If you use an API for classifying e.g. open survey responses, these responses will be sent to e.g. OpenAI or Anthropic’s server. Depending on many details, this might be against the terms of use of the data, the consent form signed by the participants, and/or privacy regulations. A second potential problem is reproducibility: the model hosted by an API can change without notice and without retaining access to earlier versions, so there is no guarantee we can call the same model 5 years from now. Both issues are mitigated by using local models, either on your own computer or on a server hosted within your organization. This adds costs and complexity, however, and this is limited to open-weights models: models like llama or qwen that have been published and can be downloaded locally.

Third, the issue of sustainability is often raised. Data centers are huge consumers of power and water supplies, and by using LLMs you contribute to the pollution they cause. It should be noted, however, that single queries, especially to a relatively small model, do not directly cause that much pollution or water usage. Although it’s hard to determine energy use of proprietary APIs, Jegham et al. (2025) estimate that a single query to a small model can even be less energy intensive than a normal Google search, but the more complex reasoning models can consume significantly more than that. This also points to the importance of not always using the most powerful model for every task: if a task can be solved with a small, non-reasoning model it would waste resources to use a bigger model for that task. Similarly, if a question can be answered on a sample of 100 texts, it would be much more polluting to analyse 10,000 texts instead.

11.2 Choosing and connecting a model

Before we can call LLMs from R, we need to connect to a specific server and model.

Using hosted LLMs

The easiest way to get started is by using a hosted proprietary model such as OpenAI’s GPT models or Anthropic’s Claude. The first thing you would need to do is go to their respective websites, create an account, and create an API key. This will require a payment method, but even just 10 euros or dollars can go a long way. Of course, whether you are able and willing to do this is completely your decision.

The specific instructions depend on the platform, but it’s generally just a couple of clicks (and a credit card). See for example the instructions from OpenAI and the Anthropic instructions.

This API key identifies you to the service, and allows R to use the API in your name. For this reason, the API key should be treated as sensitive information: anyone who has the key can use your credits, and so your money, to query the API! So, never place the API key in an R script or report. Instead, you should place this key in your environment file, which is loaded by R at startup. In RStudio, you can run the command file.edit("~/.Renviron"), and it should open this file directly. There, add the API key on a single line in the format KEY=VALUE, so something like:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-...

After that, restart R (or run readRenviron("~/.Renviron")). You can check that the key is loaded with str_sub(Sys.getenv("OPENAI_API_KEY"), end=10) - this should print the first 10 characters of your key in the console output.

After this, you can use models from that provider in your code, for example to use Claude Haiku for the same sentiment task:

As you can see, this uses claude-haiku-4-5, which is the smallest of the Claude family of models. It also does fairly well, getting 7 out of 10 answers correct, which is as likely to be a sampling variation as a real difference from the results above.

Note that ellmer will automatically search for the ANTHROPIC_API_KEY for this model, so if that key is in your environment it should work out of the box on your own computer. For using OpenAI, change that line to e.g. chat <- chat_openai(model = "gpt-4.1-nano") (and make sure that OPENAI_API_KEY is provided). How well does GPT nano classify these 10 headlines?

Note that the way this is set up it’s very easy to change LLM providers and even switch from a proprietary to a local model: just change the chat_ function and use the new chat object, and the rest of the code will run as before. This makes it easy to compare performance across different models and providers, which can be an important part of validation.

After running this, you can use token_usage() on your own computer to check how many tokens you used, and what the dollar cost was for this. In my case, using Haiku for classifying all 284 headlines in both original Dutch and English cost a grand total of 5 cents, while OpenAI cost was rounded to zero cents.

Using local models with Ollama

As an alternative to using a hosted model, it is perfectly possible (and sometimes preferable) to run a model on your own computer, especially if you have a powerful computer with a good graphics card.

First, you should download and install Ollama. For Windows and Mac, you can choose ‘manual download’ and run the downloaded installer as normal (and Linux users and general nerds like me can of course run the terminal command). After that, go to http://localhost:11434/ in your browser, it should say “Ollama is running”.

The next step is to actually download a model. Their library lists all available models, where each model shows a “size” statistic - if this is (comfortably) smaller than your GPU memory size it should work relatively quickly. For example, we could pick llama3.2:3b, listed as 2GB in size, which should fit most computers (and very easily fits my GPU’s 24GB memory). Note that even if you don’t have a dedicated GPU, or the model is too large to fit, it will still run, just (much) slower.

To install this model, you can open a terminal and run ollama pull llama3.2:3b. You can also download the model from R with pull from the ollamar package.

library(ollamar)
pull("llama3.2:3b")

You can then check which models exist with

Now that Ollama is running and we have a model, we can use it in the same way as earlier (using the prompt defined above):

Results here show an interesting development: It’s substantively correct in 6 out of 10 headlines, but in three cases it replied with a capitalized response (‘Neutral’ and ‘Negative’), and in one case added a period at the end of its response (‘negative.’).
Apparently it follows the instructions on what to emit less scrupulously than the other models did. In this case it can be fixed post-hoc rather easily using str_to_lower and str_remove_all (see Chapter 9) but in other cases the model might just start chatting about the task, rather than solving it for you.

11.3 Validation: can we trust the labels?

The code above showed performance on a very small subset of the data. How well do the four models we tested so far do on the full data?

(Note that this runs quickly here because we are using pre-computed responses, but it can easily take multiple minutes if you run this on your own computer)

As you can see, the values for each model are now added to each sentence, so we can compare performance. Note that we computed prompts as a separate column here since it’s shared between all models, with the added benefit that we can now also inspect it to see if the prompt reads as intended.

Analysing the results

An easy way to do this is to use pivot_longer (see Section 7.1.2) to turn this into a long format data frame, so we can see performance per model:

We start by pivoting the results so each model is in a separate row and each row contains a single llm_result, and we can then remove the llm_ prefix from the model name. Next, we create a clean version of the llm result by running str_to_lower and then removing all characters other than lowercase a-z (see Chapter 9 for the str_* functions).

We can then report the overall accuracy by taking the mean of the comparison between the gold standard value (sentiment) and the llm results: since TRUE and FALSE are equivalent to 1 and 0, the mean of a comparison is the proportion of cases in which they were the same, i.e. the accuracy. Since we group by model, this reports accuracy of each model separately, and it computes scores for both the raw and the cleaned answers.

As you can see, performance varies quite a bit between models. Looking at the cleaned accuracy, haiku does best on this specific test, with nano not far behind. Interestingly, when you compare the raw and cleaned accuracy, you can see that especially llama and nano had a lot of trouble following the output instructions: llama produced a response other than the three categories in 108 of 284 headlines (all capitalization and/or full stops) but 72 of these were actually correct after cleaning.

The Törnberg (2025) article we cited above is provocatively titled “Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages”. Do they?

Let’s compare our numbers to the table of results from Van Atteveldt et al. (2021).

  • Do our LLMs outperform dictionaries and classical machine learning approaches?
  • How do our LLMs compare to crowd coding and/or trained manual coding?
  • Our current models are relatively cheap models, with a fairly naive prompt. Where do you think performance might go to if we use a bigger model and/or a better prompt?
  • Can we generalize these answers to the more general task of classifying political texts? Why (not)?

Confusion matrices

To understand the kind of errors made by a classifier, let’s see which categories the gemma model gets wrong:

(see Section 7.1 for pivot_wider, and Section 5.3 for group_by |> summarize)

This produces what is known as a confusion matrix, which is a frequently used tool for getting a better understanding of classifier performance. The rows contain the actual (gold) answers, and the columns the classification produced by the model. The diagonal cells are correct (gold equals classification), and the off-diagonal cells show how often each possible mistake is made.

In this case, it shows a quite interesting picture: by far the most frequent mistakes are assigning neutral sentences to negative (n=31) or positive (34). So, apparently the model is too reluctant to use the neutral category.

We noticed above that gemma is likely to overuse positive and negative at the expense of neutral.

  • How could we improve this?
  • Can you do the same analysis for haiku? What is the outcome?
  • Would the improvement for gemma also help haiku? What is the way forward?

11.4 Writing a good prompt

We saw above that Gemma was underutilizing the neutral category compared to the gold standard. One possible reason for that is that the instructions we gave the LLM are worded differently from the codebook used by the human annotators when creating the gold standard. Like a codebook in human coding, getting the instructions (or ‘prompt’) right is a crucial driver of performance for classification with LLMs.

So, let’s see if we can improve the performance by using instructions that are based directly on the instruction given to the human coders (see Van Atteveldt et al. (2021) p. 125):

The code above uses a structured approach to creating the prompt. The codebook is defined using a (named) vector of codes, which makes it easy to change, add, or remove codes. An instruction for each code is then created with str_glue, which are collapsed into a single text using str_c.

Next, the overall task definition, context, and answering instructions are given. After that, str_glue is used to make the overall instructions, combining the codebook with the other definitions.

As you can see, these instructions make it clear what the overall task is, and that the coder should disregard any substantive background knowledge. It also gives a more detailed explanation of the categories, emphasizing that it should be about the economy and that positive can either be that things are good or that they are improving. It also gives a broader definition of neutral cases. So, let’s see if this codebook actually improves performance for Gemma?

Exercise: Evaluating the new instructions

Part I: run Gemma with the new instructions

Classify all headlines again with Gemma, but now using the new instruction we created earlier. You can base yourself on the code from Section 11.3, but you only need to run a single model.

Complete the mutate function below, but note that you have to use the same prompt template as before with the new instruction as answers are pre-computed in this interactive exercise. instruction is available as an object in this code so you don’t need to copy/paste it.

TipHint:
  1. Use interpolate to create the prompt using the same template as before: "{{instruction}}\n\nThe headline: {{headline}}".
  2. Use parallel_chat_text with the chat object and the prompts.
Important🎓 Solution:

This is the same code as in Section 11.3, but with only one model. The template is also unchanged: because the new instructions are stored in a variable with the same name, interpolate automatically uses them. This is a nice benefit of building the prompt from separate parts: you can change the instructions without touching the code that runs the classification.

Note that the template must match exactly, since we can only replay the answers that were recorded for these exact prompts.

chat_gemma <- chat_ollama(model = "gemma4:e4b") results_codebook <- headlines |> mutate( prompts = interpolate("{{instruction}}\n\nThe headline: {{headline}}"), llm_gemma = parallel_chat_text(chat_gemma, prompts) ) head(results_codebook) 
chat_gemma <- chat_ollama(model = "gemma4:e4b")

results_codebook <- headlines |>
  mutate(
    prompts = interpolate("{{instruction}}\n\nThe headline: {{headline}}"),
    llm_gemma = parallel_chat_text(chat_gemma, prompts)
  )
head(results_codebook)

Part II: evaluate the result

Due to how exercises work in this interactive textbook, we will work from the results of the previous exercise, regardless of whether you managed to solve it or not.

So, you can start from the results_codebook dataframe, which has a sentiment column and llm_gemma column per headline.

Compute the accuracy for this model by computing the proportion of headlines for which Gemma gave the correct answer. Next, show the confusion matrix, i.e. a data frame with the gold standard sentiment in the rows, Gemma’s sentiment in the columns, and frequencies in the cells.

TipHint:
  • For accuracy, you can summarize the mean of whether the two columns are the same (since there is only one model, no need to group_by here)
  • For the confusion matrix, count the frequency of each sentiment and llm_gemma pair, and then pivot_wider to create the table.
Important🎓 Solution:
# Compute and print accuracy accuracy <- results_codebook |> summarize(accuracy=mean(sentiment == llm_gemma)) cat("Overall accuracy:\n") accuracy cat("\nConfusion matrix:\n") results_codebook |> group_by(sentiment, llm_gemma) |> summarize(value=n()) |> ungroup() |> pivot_wider(names_from=llm_gemma) 
# Compute and print accuracy
accuracy <- results_codebook |>
  summarize(accuracy=mean(sentiment == llm_gemma))
cat("Overall accuracy:\n")
accuracy

cat("\nConfusion matrix:\n")
results_codebook |>
  group_by(sentiment, llm_gemma) |>
  summarize(value=n()) |>
  ungroup() |>
  pivot_wider(names_from=llm_gemma)
  • Compare the accuracy of Gemma with the new instructions to the accuracy we found in Section 11.3. Did the codebook help?
  • Compare the confusion matrix to the one we made for Gemma before. Where does the improvement come from? Did we fix the problem with the neutral category, and did that come at a cost for the other categories?
  • The results_codebook actually also contains the results of the other models, e.g. in llm_nano. Can you look at the accuracy of the other models as well? Does the new codebook help them too?
  • Can you make a confusion matrix for the nano model? What is your diagnosis?
  • What do these results tell you about whether there is a single ‘best’ prompt?
  • On your own computer, with a real LLM (see Section 11.2.1 and Section 11.2.2), try to improve the instructions further. For example, you could add decision rules for difficult cases, or give an example for each category. Does it help? Does it help for all models?
TipWhat makes good instructions?

Drafting the right instructions is probably the most important part of using LLMs for text classification (cf. Atreja et al. 2025).

These instructions can be seen as similar to the codebook used in manual annotation, so we can rely on the decades of experience in drafting manual annotation codebooks as a good starting point for writing LLM instructions. Like with codebooks, the two most important criteria for LLM instructions are clarity and validity: the instructions need to be clear and unambiguous; and the instructions need to match the theoretical concepts that are being measured.

In general, good codebooks often contain a good explanation of the overall task, as well as clear definitions of each category. Often, decision rules can be given for specific hard or borderline cases. Finally, it’s possible to give one or more examples for each code, or to provide examples at the end.

It should be noted that there are no ‘perfect’ instructions: which instructions work best can vary over time or between models. For example, many prompts used to start with a ‘persona sentence’ like “You are a helpful annotation assistant”, but Zheng et al. (2024) find little evidence that this actually helps models. There is also a tradeoff between short instructions like this, or longer, more detailed instructions that may also just cause especially smaller models to forget or ignore parts of the instruction (cf. Liu et al. 2024; Jaroslawicz et al. 2025). It also used to be recommended to explicitly ask for reasoning in the instruction (something like: “give your reasons first, and then provide the answer as a single word at the end”), but this is probably no longer beneficial for current models (cf. Sprague et al. 2025).

Note that giving examples at the end turns the task from a “zero-shot classification” into a “few-shot classification”. This can improve performance and can make the expected output format clear. It is important to give the input and output in the exact format that will be used in the actual coding, and that the examples themselves are not present in the test set.

One important note is to avoid using information from your gold standard validation in your instructions: If you check the results on the gold standard, and you notice the LLM makes a mistake, it’s tempting to include a specific rule against this (type of) mistake. However, if you then use the same gold standard to evaluate, you are overestimating the performance of the model since it’s more likely to do well on the sentence(s) you specifically checked compared to unseen sentences. This is even stronger for few-shot examples: if the model has seen a specific example already, it’s a very low bar to then classify it correctly in the evaluation. For that reason, best practice is to have a separate set of sentences as examples and for developing the instructions, so the gold standard remains unseen. For example, this can be achieved by splitting the data into two sets: a development set and a test set.

Finally, longer instructions are also more expensive (in API costs or in local compute time). This can be alleviated by caching: modern LLM providers can ‘remember’ the state after processing the instructions, so it can be re-used by later calls (provided the instructions come first and are longer than a minimum length). The exact workings of this are different per provider, so if you need to annotate a large amount of text and/or have long instructions, it can be worth checking how this works for the provider you are using.

11.5 Structured Output

In the examples so far, we used the LLM to generate a verbal reply, and then parsed that output using stringr functions.

Another possibility is to specify a structured output format for the model to use. This prevents the problem with having to clean the answers and models potentially giving invalid responses, but it becomes especially useful if a reply consists of multiple distinct values. For example, the code below asks for the sentiment as well as for the reason for the sentiment for an example sentence:

As you can see, haiku correctly reasons that this headline is not actually about the economy, so even if it is positive in general it should be coded as neutral.

Looking at the code, we first define the desired output format as containing a reason and a value. We then use a shortened version of the instructions that omits the answer part, since the model already knows the expected form of the answer. Note that the LLM ‘sees’ the field names and descriptions from the type definition, so this should be seen as part of the instructions given to the LLM.

To apply this to multiple headlines, we can use parallel_chat_structured:

As you can see, this works mostly the same as the earlier parallel_chat_text call, with the main difference that we inserted an unnest call after the parallel_chat_structured. The reason for that is that the reply column is no longer a single value, but instead is now a dataframe itself. The unnest function then creates separate columns for the reason and value.

It is also possible to ask for multiple answers for each prompt. For example, we could ask the LLM to identify which specific economic indicators are mentioned and how they are evaluated:

Looking at the results, we see some interesting details. The first sentence contained no indicators, since it’s only about minister Dijsselbloem wanting a solution (and because we specified keep_empty=TRUE, the row is kept with the measurement variables set to NA). The sentence about ‘Deutsche Bank’ has two indicators (and hence two rows): the (mega) fines and the spending (cuts). These sentences also nicely show the difference between direction and evaluation: the fines go 'up', which is 'bad'.

Looking at the code, the main difference is in the structured output type. We first define a single economic_indicator as consisting of three variables (indicator, direction, evaluation) which are described in quite a bit of detail. We then define type_indicator_list as a type_array with this economic_indicator as items. Array means a list or sequence of items here, so this is saying the output should be a list of indicators.

Note that using an intermediate object (economic_indicator) here is a pure stylistic choice: nesting it within the type_array produces the same result. It’s also a choice to put so much of the codebook/instructions in the type definition rather than in the instruction: both end up in the model, albeit in slightly different places. In any case, avoid repeating things in the instruction that are already present in the structured output type definition.

Evaluating the results of the last chunk is left as a challenge for the reader. Some possibly interesting questions:

  • Which headline produced the most indicators? Does that seem right?
  • Produce a table of direction vs. evaluation. Are the two correlated? Do most examples or opposing values make sense?
  • Is there a relation between the gold standard sentiment and either direction or evaluation?
  • Looking in more detail, what perspective is used for the evaluation? Who gets to decide whether lifting financial restrictions is good or bad?
  • On your own computer, try running a smaller model such as gpt-4.1-nano. Do the results differ? Why would that be? How sensitive is this to the phrasing of the descriptions and field names?

If you want, you can use the chunk below to play with these results to try to answer the questions above (except for the last):

As a final note: remember that the schema only determines the form of the answer, not the content. So, writing a good prompt and validating the results remains crucial for any valid application.

Quiz questions

Section 11.1 quiz: LLMs and the ellmer package

Q1: Asking a follow-up question

You run the following code: 

chat <- chat_ollama(model = "gemma4:e4b")
chat$chat("Is this headline positive or negative? Headline: 'The economy just crashed'")
chat$chat("Are you sure?")

What does the model receive when you ask the second question?

  • The first question, the model’s answer to it, and “Are you sure?”
  • Only “Are you sure?”, so the model doesn’t know what it should be sure about.
  • The first question and “Are you sure?”, but not the model’s earlier answer.
  • “Are you sure?” in a new, empty conversation, as each call to chat$chat() starts a new chat.
Q2: Double braces

In the chapter, the prompts are created with 

interpolate("{{instruction}}\n\nThe headline: {{headline}}")

Why does interpolate() use double curly braces, unlike str_glue(), which uses single ones?

  • Because instructions often contain literal curly braces (e.g. an example of JSON output), which should not be filled in as variables.
  • Because double braces are needed to fill in a whole column, while single braces only fill in a single value.
  • Because the LLM needs to see which parts of the prompt are variables.
  • Because double braces refer to columns of the data frame, and single braces to objects like instruction.
Q3: Coding open survey answers

You want to classify open survey answers that contain personal information. The consent form promises respondents that their answers will not leave the university’s servers. Which approach fits this promise?

  • Use an open-weights model running on a university server, e.g. via Ollama.
  • Use a hosted model such as Claude, but set the temperature to zero.
  • Use the smallest hosted model available, such as gpt-4.1-nano.
  • Use a hosted model, but store the API key in your .Renviron file rather than in your script.
Q4: Classifying many headlines

You want to classify all 284 headlines. Instead of calling chat$chat() for each headline, the chapter uses: 

headlines |>
  mutate(llm_sentiment = parallel_chat_text(
    chat = chat,
    prompts = interpolate("{{instruction}}\n\nThe headline: {{headline}}")))

What does parallel_chat_text() do differently from calling chat$chat() 284 times?

  • It sends each prompt as a separate conversation and returns one answer per headline that you can store as a column.
  • It combines all 284 headlines into a single message, so the model can classify them all at once.
  • It automatically calls chat$chat() 284 times so you don’t have to repeat it.
  • It sends each headline to several models at the same time, so you can compare their answers.

Section 11.2 quiz: choosing and connecting a model

Q1: Sharing your script

A student shares the following analysis script with their research group: 

library(ellmer)
chat <- chat_anthropic(model = "claude-haiku-4-5", api_key = "sk-ant-api03-...")

What is the problem, and how should it be fixed?

  • Anyone who has the script can use the key at the student’s expense; the key should go in ~/.Renviron as ANTHROPIC_API_KEY instead.
  • The api_key argument doesn’t exist, so the code will give an error.
  • Claude can’t be used for research, so the student should switch to chat_openai().
  • There’s no real problem, but it’s tidier to put the key in a separate variable at the top of the script.
Q2: Switching to a local model

The following code classifies 10 headlines with Claude Haiku: 

chat <- chat_anthropic(model = "claude-haiku-4-5")
headlines |>
  slice_sample(n=10) |>
  mutate(llm_sentiment = parallel_chat_text(
    chat=chat,
    prompts=interpolate("{{instruction}}\n\nThe headline: {{headline}}")))

What do you need to change to run the same analysis with llama3.2:3b via Ollama?

  • Only the first line: chat <- chat_ollama(model = “llama3.2:3b”).
  • The first line, and parallel_chat_text() should become parallel_chat_ollama().
  • The first line, and you need to add an OLLAMA_API_KEY to your .Renviron file.
  • The first line, and the prompt template, as local models need a different prompt format.

Section 11.3 quiz: validating the labels

Q1: Raw and cleaned answers

For four headlines, a model gave the following answers:

sentiment llm_result llm_result_cleaned
neutral Neutral neutral
negative negative. negative
positive positive positive
negative positive positive

What does summarize(accuracy = mean(sentiment == llm_result)) return?

  • 0.25
  • 0.75
  • 1
  • An error, as you can’t take the mean of TRUE and FALSE values.
Q2: Reading a confusion matrix

This is the confusion matrix of a model on 100 headlines (a simplified version of what we found for Gemma). The rows are the gold standard, the columns the model’s answers:

sentiment negative neutral positive
negative 30 3 1
neutral 11 17 12
positive 1 2 23

Which conclusion does this table support?

  • The model often codes neutral headlines as negative or positive.
  • The model codes too many headlines as neutral.
  • Most errors are positive headlines coded as negative, and vice versa.
  • The model gets more than 90% of the headlines right.

Section 11.4 quiz: writing a good prompt

Q1: Fixing the mistakes you found

You compare an LLM’s answers to your gold standard and notice it gets headlines about interest rates wrong. You add a specific rule about interest rates to the instructions based on this example, run the model again on the same gold standard, and accuracy goes up from 0.70 to 0.74. What is the problem?

  • The rule was based on the same headlines you evaluate on, so 0.74 probably overestimates how well it works on new headlines.
  • There is no problem: adding decision rules for difficult cases is exactly what a good codebook does.
  • Accuracy can’t be compared between two different prompts.
  • An improvement of 0.04 is too small to matter, so the rule should be removed again.
Q2: One prompt, several models

In the chapter, the instructions based on the human codebook improved accuracy for Gemma (from 0.70 to 0.76), but slightly lowered it for Haiku and nano. What is the best lesson to draw from this?

  • Which instructions work best can differ between models, so you need to validate each combination of model and instructions.
  • The codebook instructions are worse than the simple ones, so you should always use short instructions.
  • Gemma is a better model than Haiku and nano for this task.
  • The differences are so small that the choice of instructions doesn’t matter.

Section 11.5 quiz: structured output

Q1: Text or structure?

In the first part of the chapter, we asked the LLM to answer with a single word and then cleaned the answers with str_to_lower() and str_remove_all(). What is different when you use structured output instead?

  • You specify the form of the answer, the model has to answer in that form, and you get the answer back as R values.
  • The model gives more correct answers, because it has to think more carefully about the structure.
  • ellmer automatically cleans the text answers afterwards, like we did with str_to_lower() and str_remove_all().
  • The model always gives the same answer to the same question, which makes the results reproducible.
Q2: Why unnest?

The chapter asks for a reason and a value for each headline: 

headlines |>
  mutate(prompt = interpolate("{{instruction_noanswer}}\n\nThe headline: {{headline}}"),
         reply = parallel_chat_structured(chat, prompt, type_reasoned)) |>
  unnest(reply)

Why is the unnest(reply) step needed?

  • Because each value in reply contains several fields (reason and value), and unnest() turns these into separate columns.
  • Because the model returns a JSON text, which unnest() parses into R values.
  • Because unnest() creates a separate row for the reason and for the value of each headline.
  • Because unnest() removes the answers that are not one of the allowed categories.
Q3: Counting rows

You use the type_indicator_list type from the chapter on three headlines. The model finds no indicators in the first headline, one in the second, and two in the third. How many rows does the result have after unnest(reply, keep_empty=TRUE)?

  • 4
  • 3, because the headline without indicators is dropped.
  • 3, one row per headline.
  • 2, one row for each headline that has at least one indicator.
Q4: What structured output does and doesn’t do

Which of these statements about structured output is false?

  • Because the answer has to follow the type, you no longer need to validate the results.
  • The field names and descriptions in the type are seen by the model, so they act as part of the instructions.
  • With type_enum(), you no longer need to clean answers like “Neutral” or “negative.”.
  • With type_array(), the model can return a different number of items for each text.