The equation

At its core, multiple linear regression is about finding the best-fitting linear relationship between multiple predictor variables (independent variables) and a response variable (dependent variable). The equation is similar to that of simple linear regression, but with more predictor variables:

Where:

• is your dependent variable—the thing you’re trying to predict or explain. (Think: your happiness score.)
• is the intercept. It’s the value of when all the predictor variables are zero. (If you drank no coffee, got zero sleep, and had no emails, how happy would you be? Probably not very.)
• are the coefficients for each predictor variable . They tell you how much each predictor contributes to YY. (For example, does coffee make you happier than sleep? These coefficients will tell you!)
• is the error term. It accounts for the randomness in life that the model can’t explain. (Maybe your cat did something unpredictable, like actually not knocking over your mug.)

The goal of multiple linear regression is to find the values of that minimize the sum of squared errors—basically, the difference between what the model predicts and what actually happens . In other words, we’re trying to make our predictions as accurate as possible.

Generating some data

First, let’s again load some packages and set up our theme.

# Loading packages
library(ggplot2)    # For plotting
library(ggsci)      # Going with a simpsons theme
library(gghalves)   # Using half plots
library(dplyr)      # Data manipulation
library(tidyr)      # Data manipulation

# Setting default theme
theme_set(
  theme_minimal() +
    theme(
      panel.grid.minor = element_blank(),
      plot.title = element_text(hjust = 0.5, face = "bold", size = rel(1.4)),
      axis.title = element_text(face = "bold", size = rel(1)),
      strip.background = element_rect(fill = "grey80", color = NA)
      )
)

To illustrate multiple linear regression, let’s generate some data. For this example, let’s stick with the happiness theme we have going. We’ll create a dataset with three predictor variables representing daily coffee intake, sleep, and emails, and one response variable representing daily happiness.

set.seed(321)

# Generating our predictor variables
coffee <- rnorm(10, mean = 2, sd = 1)   # Coffee intake (cups per day)
sleep  <- rnorm(10, mean = 7, sd = 1)   # Sleep (hours per night)
emails <- rnorm(10, mean = 10, sd = 3)  # Emails (received per day)

# Generating our response variable
happiness <- 5 + 0.2 * coffee + 0.7 * sleep - 0.2 * emails + rnorm(10, mean = 0, sd = 1)

# Creating data set
happiness_data <- data.frame(
  happiness = ceiling(happiness),
  coffee    = floor(coffee),
  sleep     = ceiling(sleep),
  emails    = ceiling(emails)
)

head(happiness_data)
##   happiness coffee sleep emails
## 1         7      3     8     13
## 2        10      1     9     10
## 3         9      1     8     13
## 4        12      1    10      7
## 5         8      1     6      8
## 6         8      2     7     11

Before we dive into regression, let’s take a moment to visualize our data. We’ll use a combination of boxplots and jittered points to show the distribution of each variable

happiness_long <- happiness_data |>
  pivot_longer(
    cols = everything(),
    names_to = "variable",
    values_to = "value") |>
  mutate(variable = factor(
    variable, levels = c("coffee", "sleep", "emails", "happiness")))

happiness_plot <- happiness_long |>
  ggplot(aes(x = variable, y = value, fill = variable, colour = variable)) +
  gghalves::geom_half_point(
    transformation = ggbeeswarm::position_quasirandom(width = 0.1),
    side = "r") +
  gghalves::geom_half_boxplot(side = "l", colour = "black") +
  scale_fill_simpsons() +
  scale_colour_simpsons() +
  scale_y_continuous(breaks = seq(2, 12, by = 2)) +
  labs(x = "Variables", y = "Value") +
  guides(fill = "none", colour = "none")

happiness_plot

What Do We See?

From the plot, we can start to get a sense of how our variables are distributed:

• Coffee intake hovers around 2-3 cups per day.
• Sleep is mostly in the 7-8 hour range.
• Emails are all over the place, because inboxes are chaos.
• Happiness scores are a bit more spread out, reflecting the combined influence of coffee, sleep, and emails.

This visualization sets the stage for our regression analysis. Next up, we’ll dive into the math and find out exactly how much each of these factors contributes to happiness.

Step 1: Computing means

First, calculate the means of happiness, coffee, sleep, and emails.

Step 2: Computing deviations

Next, we calculate the deviations for each variable from their mean.

Observation
1	7 - 9.2 = -2.2	3 - 1.6 = 1.4	8 - 8 = 0	13 - 9.4 = 3.6
2	10 - 9.2 = 0.8	1 - 1.6 = -0.6	9 - 8 = 1	10 - 9.4 = 0.6
3	9 - 9.2 = -0.2	1 - 1.6 = -0.6	8 - 8 = 0	13 - 9.4 = 3.6
4	12 - 9.2 = 2.8	1 - 1.6 = -0.6	10 - 8 = 2	7 - 9.4 = -2.4
5	8 - 9.2 = -1.2	1 - 1.6 = -0.6	6 - 8 = -2	8 - 9.4 = -1.4
6	8 - 9.2 = -1.2	2 - 1.6 = 0.4	7 - 8 = -1	11 - 9.4 = 1.6
7	8 - 9.2 = -1.2	2 - 1.6 = 0.4	8 - 8 = 0	4 - 9.4 = -5.4
8	11 - 9.2 = 1.8	2 - 1.6 = 0.4	8 - 8 = 0	12 - 9.4 = 2.6
9	10 - 9.2 = 0.8	2 - 1.6 = 0.4	8 - 8 = 0	7 - 9.4 = -2.4
10	9 - 9.2 = -0.2	1 - 1.6 = -0.6	8 - 8 = 0	9 - 9.4 = -0.4

Step 3: Computing cross deviations

Now that we know our means and deviations we are able to solve for our set of cross-deviations.

Step 4: Computing squared deviations

Finally, we can calculate the squared deviations for each variable.

Step 5: Computing our coefficients

Now that we have all the necessary values, we can calculate our coefficients.

Step 6: Writing our regression equation

Finally, we can write our regression equation:

Phew, that was a lot of work!!

But what does this now tells us??

• For every additional cup of coffee you drink, your happiness score decreases by 0.74.
• For every additional hour of sleep you get, your happiness score increases by 0.92.
• For every additional email you receive, your happiness score decreases by 0.005.

Let’s verify this by running the lm() function in R

fit <- lm(happiness ~ coffee + sleep + emails, data = happiness_data)

broom::tidy(fit)
## # A tibble: 4 × 5
##   term        estimate std.error statistic p.value
##   <chr>          <dbl>     <dbl>     <dbl>   <dbl>
## 1 (Intercept)  3.03        3.75     0.808   0.450 
## 2 coffee      -0.741       0.608   -1.22    0.269 
## 3 sleep        0.925       0.398    2.32    0.0591
## 4 emails      -0.00459     0.143   -0.0321  0.975

The coefficients are very close to what we calculated by hand.

Step 1: Setting up our matrices

To begin, let’s define the key components of our regression model:

• as the response variable
• as the matrix of predictor variables
• as the vector of coefficients we want to estimate
• as the error term

The regression model can be expressed in matrix form as:

Our goal is to solve for , the vector of coefficients. The least squares solution for is given by:

Let’s define the matrix (including our intercept) and matrix

Step 2: Transposing

The next step is to compute the transpose of denoted as . Transposing a matrix involves flipping its rows and columns. For our matrix, this results in:

Step 3: Computing matrix products

With calculated, we now compute two key matrix products:

• : This product of the transposed predictor matrix and original predictor matrix.
• : This product of the transposed predictor matrix and the response variable matrix.

Matrix multiplication involves multiplying corresponding elements of each row and column. For example, the first element of the resulting matrix is the sum of the products of the first row of the first matrix and the first column of the second matrix . To illustrate, the first two elements of would be:

Continuing this process for all elements of and results in:

Step 4: Computing the inverse

The next step is to compute the inverse of , otherwise denoted as . The inverse of a matrix is a special matrix that, when multiplied by the original matrix, yields the identity matrix. In other words:

where is the identity matrix. The inverse of a matrix can be computed using various methods, with Gaussian elimination being one of the most common. Gaussian elimination transforms the matrix into row echelon form through a series of row operations, ultimately reducing it to the identity matrix.

• Start by augmenting with an identity matrix such that :

• Perform row operations to reduce the left side of the matrix to the identity matrix, resulting in the inverse of on the right side.
• The resulting matrix on the right side will be the inverse of .

For my own mental sanity I will not perform this operation by hand, but rather use the solve() function in R to calculate the inverse of .

XT_X <- matrix(c(
  10, 16, 80, 94,
  16, 30, 127, 154,
  80, 127, 650, 749,
  94, 154, 749, 962
), nrow = 4, ncol = 4, byrow = TRUE)

solve(XT_X) |> round(digits = 3)
##        [,1]   [,2]   [,3]   [,4]
## [1,]  9.162 -0.456 -0.885 -0.133
## [2,] -0.456  0.240  0.021 -0.010
## [3,] -0.885  0.021  0.103  0.003
## [4,] -0.133 -0.010  0.003  0.013

We can then write as:

Step 5: Computing

With calculated, we can now compute using the formula:

Multiplying by gives us:

Step 6: Writing our regression equation

Finally, we can write our regression equation:

But wait, this isn’t what we got by doing it by hand, nor does it match the lm() output.

This is because the solve() function in R is able to calculate the inverse of a matrix without any rounding errors. This is not the case when doing it by hand. If I removed |> round(digits = 3) from the above we get a much more precise estimate of

solve(XT_X)
##            [,1]        [,2]         [,3]         [,4]
## [1,]  9.1623030 -0.45619804 -0.885469661 -0.132832858
## [2,] -0.4561980  0.24041444  0.020970344 -0.010237002
## [3,] -0.8854697  0.02097034  0.102990445  0.002978037
## [4,] -0.1328329 -0.01023700  0.002978037  0.013339124

and then ultimately an exact match to the lm() output

XT_y <- matrix(c(92, 143, 746, 859), nrow = 4, ncol = 1, byrow = TRUE)

solve(XT_X) %*% XT_y

            [,1]
[1,]  3.03176573
[2,] -0.74066261
[3,]  0.92455640
[4,] -0.00459114

So finally, we could say our regression equation is:

Just goes to show how much rounding makes a difference!

Generating some data

First let’s load some packages and set our theme

# Loading packages
library(ggplot2)

# Setting default theme
theme_set(
  theme_minimal() +
    theme(
      plot.title = element_text(hjust = 0.5, face = "bold", size = rel(1.4)),
      axis.title = element_text(size = rel(1))
      )
)

To make this more concrete, let’s create a small dataset in R. We’ll simulate some data for study hours and exam scores, and then use linear regression to model the relationship between them.

set.seed(123)

# Hours of study per week (our x variable)
study_hours <- rnorm(10, mean = 7, sd = 2)

# Exam scores (our y variable)
exam_scores <- 25 + 5 * study_hours + rnorm(10, mean = 0, sd = 2)

test_data <- data.frame(
  study_hours = ceiling(study_hours),
  exam_scores = ceiling(exam_scores))

head(test_data)

  study_hours exam_scores
1           6          57
2           7          59
3          11          77
4           8          61
5           8          61
6          11          81

If we plot this data we can see the relationship between study hours and exam scores:

exam_plot <- test_data |>
ggplot(aes(x = study_hours, y = exam_scores)) +
  geom_point(size = 4) +
  labs(
    title = "Exam scores vs. study hours",
    x = "Study hours",
    y = "Exam scores"
  )

exam_plot

From the plot, you can see a clear trend: the more hours a student spends studying, the higher their exam score tends to be. This is the relationship we want to model using linear regression.

Fitting a model in R

In R, fitting a linear regression model is as simple as using the lm() function. We provide the formula for the model and the dataset, and R does the rest. Let’s fit a linear model to our data:

fit <- lm(exam_scores ~ study_hours, data = test_data)

broom::tidy(fit)

# A tibble: 2 × 5
  term        estimate std.error statistic    p.value
  <chr>          <dbl>     <dbl>     <dbl>      <dbl>
1 (Intercept)    21.3      3.93       5.42 0.000627  
2 study_hours     5.23     0.496     10.5  0.00000569

The lm() function fits a linear model where exam_scores is the dependent variable and study_hours is the independent variable . The output gives us the estimated coefficients for the intercept and slope .

• The intercept is 21.33, meaning a student who studies 0 hours is predicted to score approximately 21% on the exam.
• The slope is 5.23, indicating that for every additional hour of study, a student’s predicted exam score increases by 5.23%.

Visually this looks like

exam_plot <- exam_plot +
  geom_smooth(
    method = "lm",
    colour = ggokabeito::palette_okabe_ito(order = 5),
    se = FALSE) +
  # Adding equation of the line
    annotate(
      "text", x = 6, y = 80,
    label = paste0(
      "Y = ", round(coef(fit)[1], 2), " + ", round(coef(fit)[2], 2), "X"),
    color = ggokabeito::palette_okabe_ito(order = 5),
    size = rel(5), fontface = "bold"
  )

exam_plot

But how are these values calculated

Step 1: Calculate the means

The first step is to calculate the means of and . These are the averages of the independent and dependent variables, respectively.

Step 2: Calculate the deviations from the mean

Next, we calculate the deviations from the mean for each data point and their products. These are the building blocks for the slope (

Student
1	6	57	7.82	2.89
2	7	59	1.82	0.49
3	11	77	50.82	10.89
4	8	61	-0.18	0.09
5	8	61	-0.18	0.09
6	11	81	64.02	10.89
7	8	66	1.32	0.09
8	5	44	47.52	7.29
9	6	55	11.22	2.89
10	7	55	4.62	0.49
	77	616	188.8	36.1

Step 3: Calculate the slope

Using the formula for we now get:

Step 4: Calculating the intercept

Using the formula for we now get:

Step 5: Writing our regression equation

Now that we have values for we can write the regression equation:

which if you remember from before is the same as our lm() model in R.

broom::tidy(fit)

# A tibble: 2 × 5
  term        estimate std.error statistic    p.value
  <chr>          <dbl>     <dbl>     <dbl>      <dbl>
1 (Intercept)    21.3      3.93       5.42 0.000627  
2 study_hours     5.23     0.496     10.5  0.00000569

Calculating abundance

bird_abundance <- species %>%
  select(ParkName, CategoryName) %>%
  group_by(ParkName, CategoryName) %>%
  summarise(Abundance = n()) %>%
  rename(Park_name = ParkName, Species = CategoryName)  %>%
  mutate(Park_name = as.factor(Park_name)) %>%
  filter(Species == "Bird")

bird_abundance %>%
  head()

# A tibble: 6 × 3
# Groups:   Park_name [6]
  Park_name                     Species Abundance
  <fct>                         <chr>       <int>
1 Acadia National Park          Bird          364
2 Bryce Canyon National Park    Bird          218
3 Cuyahoga Valley National Park Bird          246
4 Glacier National Park         Bird          277
5 Grand Canyon National Park    Bird          456
6 Grand Teton National Park     Bird          266

PGN Conversion

To facilitate further analysis, we will convert the raw move strings into the PGN format, using a custom convert_to_pgn() function that I wrote.

#|
# Convert moves into PGN format ------------------------------------------------
convert_to_pgn <- function(moves, game_id) {

  # Split the moves string into a list of individual moves
  move_list <- strsplit(moves, " ")[[1]]

  # Initialize an empty string for the PGN format
  pgn <- ""

  # Loop through the moves two at a time (each move is a pair: white and black)
  for (i in seq(1, length(move_list), by = 2)) {

    move_number <- (i + 1) / 2 # Move number calculation

    # Add both white's and black's moves, if available
    if (i < length(move_list)) {
      pgn <- paste0(pgn, move_number, ". ", move_list[i], " ", move_list[i+1], " ")
    } else {  # In case the game ends on white's move (no black move)
      pgn <- paste0(pgn, move_number, ". ", move_list[i])
    }
  }

  # Returning pgn string
  return(pgn)
}

# Converting moves into pgn format
chess_games <- data %>%
  select(game_id, moves) %>%
  mutate(
    game_id = seq(1:nrow(data)),
    moves = mapply(convert_to_pgn, moves, game_id))

# Displaying converted dataset
chess_games %>%
  rename(ID = game_id, Moves = moves) %>%
  head(n = 2)
## # A tibble: 2 × 2
##      ID Moves                                                                   
##   <int> <chr>                                                                   
## 1     1 "1. d4 d5 2. c4 c6 3. cxd5 e6 4. dxe6 fxe6 5. Nf3 Bb4+ 6. Nc3 Ba5 7. Bf…
## 2     2 "1. d4 Nc6 2. e4 e5 3. f4 f6 4. dxe5 fxe5 5. fxe5 Nxe5 6. Qd4 Nc6 7. Qe…

Extracting game history

Even with parallel processing the below code takes extremely long to run when rendering in quarto (4+ hrs). In a normal R session it takes about 20 minutes. I am not quite sure as to why the time difference is so extreme. However, to make my life a little easier and so I’m not waiting for hours when I need to re-render this document I will be reading in the data that this code outputs from an excel file.

However, for those interested I have left the parallel processing code below

#|
# Function to extract game history
process_moves <- function(p) {
  chss <- Chess$new()          # Initialize a new chess object
  chss$load_pgn(p)             # Load the PGN notation into the chess object
  chss$history_detail()        # Extract detailed history of the game
}

plan(multisession, workers = parallel::detectCores() - 1)

# Parallel processing: Convert PGN format to game history
chess_games <- chess_games %>%
  mutate(data = future_map(moves, process_moves)) %>%
  select(-moves) %>%
  unnest(cols = c(data))

#|
# White pieces
white_pieces <- c(
  "a1 Rook", "b1 Knight", "c1 Bishop", "White Queen",
  "White King", "f1 Bishop", "g1 Knight", "h1 Rook")

chess_games <- chess_games %>%
  filter(piece %in% white_pieces) %>%
  sample_n(25000)

#|
chess_games <- readxl::read_xlsx(path = here::here("data/chess_history.xlsx"))

chess_games %>%
  select(game_id, piece, from, to) %>%
  head(n = 4)
## # A tibble: 4 × 4
##   game_id piece       from  to   
##     <dbl> <chr>       <chr> <chr>
## 1    7298 b1 Knight   b1    d2   
## 2   16254 g1 Knight   e5    c4   
## 3   19492 h1 Rook     h1    e1   
## 4   10827 White Queen b6    e3

Creating a chess board

First, we will create a chessboard using ggplot. We can do this using the .chessboard() function, again from the rchess package. This function generates the grid and coordinates for each square which can piped into ggplot.

#|
# Creating a chess board -------------------------------------------------------
board <- rchess:::.chessboarddata() %>%
  tibble() %>%
  select(cell, col, row, x, y, cc)

board %>%
  head(n = 4)

# A tibble: 4 × 6
  cell  col     row     x     y cc   
  <chr> <chr> <int> <int> <int> <chr>
1 a1    a         1     1     1 b    
2 b1    b         1     2     1 w    
3 c1    c         1     3     1 b    
4 d1    d         1     4     1 w    

Adding board movements

Next, we will join the chessboard data with the game data to calculate the origin and destination of each piece movement. This allows us to visualize the trajectories of pieces as they move across the board.

# Join board data with game data to get move origin and destination
chess_games <- chess_games %>%
  left_join(board %>% rename(from = cell, x.from = x, y.from = y), by = "from") %>%
  left_join(board %>%
              rename(to = cell, x.to = x, y.to = y) %>%
              select(-cc, -col, -row), # Exclude unnecessary columns
            by = "to") %>%
  mutate(x_gt_y = abs(x.to - x.from) > abs(y.to - y.from), # Check if x movement is greater than y
         xy_sign = sign((x.to - x.from)*(y.to - y.from)) == 1, # Check if X and Y move in the same direction
         x_gt_y_equal_xy_sign = x_gt_y == xy_sign) # Check if both conditions hold

# Previewing
chess_games %>%
  select(piece, from, to, x.from, y.from, x.to, y.to) %>%
  head(n = 4)

# A tibble: 4 × 7
  piece       from  to    x.from y.from  x.to  y.to
  <chr>       <chr> <chr>  <int>  <int> <int> <int>
1 b1 Knight   b1    d2         2      1     4     2
2 g1 Knight   e5    c4         5      5     3     4
3 h1 Rook     h1    e1         8      1     5     1
4 White Queen b6    e3         2      6     5     3

Plotting piece movements

Now we can visualize their paths on a chessboard.

p <- chess_games %>%
  ggplot() +

  # Adding board data
  geom_tile(data = board, aes(x, y, fill = cc)) +

  # Adding piece movement
  geom_curve(
    data = chess_games %>% filter(x_gt_y_equal_xy_sign),
    aes(x = x.from, y = y.from, xend = x.to, yend = y.to),
    position = position_jitter(width = 0.2, height = 0.2),
    curvature = 0.50, angle = -45, alpha = 0.02, color = "white",
    linewidth = 1.02) +
  geom_curve(
    data = chess_games %>% filter(!x_gt_y_equal_xy_sign),
    aes(x = x.from, y = y.from, xend = x.to, yend = y.to),
    position = position_jitter(width = 0.2, height = 0.2),
    curvature = -0.50, angle = 45, alpha = 0.02, color = "white",
    linewidth = 1.02) +

  # Labs
  labs(
    title = title_text,
    caption = caption_text,
    x = NULL,
    y = NULL) +

  # Scales
  scale_fill_manual(values = chess_tiles) +
  coord_equal() +

  # Facet
  facet_wrap(~ factor(piece, levels = white_pieces), ncol = 4) +

  # Theme
  theme(
    plot.title = element_text(
      size = 50,
      family = "title",
      face = "bold",
      colour = title_col,
      lineheight = 1.1,
      hjust = 0.5,
      margin = margin(t = 5, b = 5)),
    plot.caption = element_markdown(
      size = rel(1.25),
      family = "caption",
      colour = caption_col,
      lineheight = 1.1,
      hjust = 0.5,
      margin = margin(t = 5, b = 5))
    )

p

Overview

Agent-based models are computer simulations used to study the interactions between individuals, entities, environments, and time. These stochastic models are built from the bottom up, meaning individual agents (oftentimes representing people) are assigned specific attributes. The agents are programmed to behave and interact with other agents and their environment in predetermined ways, producing emergent effects that differ from the effects observed at the individual level.

Agent-based models differ from the traditional regression-based methods in several key ways. For starters, regression-based methods focus on dependent and independent variables. Researchers often ask themselves What variables do I need to consider in my model?. In contrast, agent-based models shift the focus to dynamic processes, prompting the question What processes do I need to consider?.

Regression-Based Methods

In regression-based methods, the primary goal is to identify and quantify the relationships between variables. For example, a researcher might want to understand the relationship between smoking and lung cancer. Using regression analysis, the researcher would collect data on smoking (independent variable) and lung cancer incidence (dependent variable), then apply statistical techniques to determine the strength and significance of the association.

• Research Question: How does smoking affect the risk of lung cancer?
• Independent Variable: Number of cigarettes smoked per day.
• Dependent Variable: Incidence of lung cancer.
• Outcome: A statistical measure (e.g., odds ratio) that quantifies the increased risk of lung cancer associated with smoking.

Agent Based Methods

Agent based models, on the other hand, shift the focus to dynamic processes. These models focus on simulating the interactions between individual agents and their environment. Instead of looking solely at variable relationships, agent based models explore how processes and behaviors lead to emergent phenomena at the system level. This approach is particularly useful for studying complex systems where individual actions and interactions create outcomes that are not easily predictable from the behavior of individual agents alone.

• Research Question: How does an infectious disease spread through a population?
• Processes to Consider: Individual movement patterns, infection transmission rates, social interactions, and public health interventions.
• Simulation: Each agent represents a person who can move, interact, become infected, and recover or succumb to the disease. The model simulates how these processes unfold over time, providing insights into potential outbreak scenarios and intervention impacts.

By focusing on dynamic processes, ABMs allow researchers to capture the complexity of real-world systems. They provide a way to explore “what-if” scenarios and understand the potential effects of various interventions.

Key Features

1. Agents: The core of NetLogo models are agents, which can be of different types:

   • Turtles: The primary agents that move around the world.
   • Patches: The grid cells that make up the world; they don’t move but can change state.
   • Links: Connections between turtles that can represent relationships or flows.
   • Observer: A special agent that monitors and controls the simulation.

2. Environment: NetLogo’s world is typically a two-dimensional grid of patches where turtles move and interact. The environment can be customized to simulate various real-world scenarios, from ecosystems to traffic systems.

3. Programming Language: NetLogo uses a Logo-based programming language that for building both simple and complex models.