Unsupervised Methods for Identifying Pass Coverage Among Defensive Backs with NFL Player Tracking Data

06/26/2019
by   Rishav Dutta, et al.
0

Analysis of player tracking data for American football is in its infancy, since the National Football League (NFL) released its Next Gen Stats tracking data publicly for the first time in December 2018. While tracking datasets in other sports often contain detailed annotations of on-field events, annotations in the NFL's tracking data are limited. Methods for creating these annotations typically require extensive human labeling, which is difficult and expensive. We begin tackling this class of problems by creating annotations for pass coverage types by defensive backs using unsupervised learning techniques, which require no manual labeling or human oversight. We define a set of features from the NFL's tracking data that help distinguish between "zone" and "man" coverage. We use Gaussian mixture modeling and hierarchical clustering to create clusters corresponding to each group, and we assign the appropriate type of coverage to each cluster through qualitative analysis of the plays in each cluster. We find that the mixture model's "soft" cluster assignments allow for more flexibility when identifying coverage types. Our work makes possible several potential avenues of future NFL research, and we provide a basic exploration of these in this paper.

READ FULL TEXT VIEW PDF

Authors

page 3

page 15

page 17

page 18

page 20

page 21

page 22

page 23

11/18/2020

Skewed Distributions or Transformations? Modelling Skewness for a Cluster Analysis

Because of its mathematical tractability, the Gaussian mixture model hol...
10/26/2010

A GMBCG Galaxy Cluster Catalog of 55,424 Rich Clusters from SDSS DR7

We present a large catalog of optically selected galaxy clusters from th...
09/11/2020

An unsupervised deep learning framework via integrated optimization of representation learning and GMM-based modeling

While supervised deep learning has achieved great success in a range of ...
12/18/2019

Boltzmann Exploration Expectation-Maximisation

We present a general method for fitting finite mixture models (FMM). Lea...
12/27/2020

Generalized Categorisation of Digital Pathology Whole Image Slides using Unsupervised Learning

This project aims to break down large pathology images into small tiles ...
04/08/2022

Optical tracking in team sports

Sports analysis has gained paramount importance for coaches, scouts, and...
08/07/2019

Route Identification in the National Football League

Tracking data in the NFL is a sequence of spatial-temporal measurements ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In 2019, the National Football Leauge (NFL) released player and ball tracking data from the first six weeks of the 2017 season for its inaugural Big Data Bowl. This tracking data, maintained by NFL Next Gen Stats, marks the locations and trajectories (speed, angle) of all 22 players on the field and the ball at a rate of 10 Hz. Importantly, the NFL is the first major professional sports league to release such detailed and abundant tracking data to the public, ushering in an era of innovative research into the sport of American football.

Prior to the release of this data, publicly available statistics for defensive players was limited to simple counting measures like tackles, sacks, interceptions, etc. Aside from what has been recently made available via the NFL’s Next Gen Stats website, little work has been done publicly to analyze defensive players. Burke (2018) provides a method for analyzing pass rushers using NFL tracking data. Private companies like Pro Football Focus likely provide innovative metrics to NFL teams, but this information is not available publicly.

One difficulty in working with tracking data across all sports is event and strategy annotation. For example, the NBA provides event annotations (e.g. passes, shots, turnovers) in its optical tracking data, but more detailed information (e.g. defensive schemes, picks, or set plays) must be identified by analysts. An example of this is the work of Miller and Bornn (2017), who identify and annotate set plays during basketball possessions. Similarly, the NFL provides some basic annotations (e.g. ball snapped, handoff, first contact, etc), but more detailed annotations must be identified by analysts. An example of this the work of Chu et al. (2019), who identify and annotate wide receiver route types in football. Both Miller and Bornn (2017) and Chu et al. (2019) use unsupervised learning techniques such as mixture modeling to provide these annotations.

We begin to tackle this problem for the NFL tracking data in this paper. Although there are many pieces of information that would be useful to annotate in the NFL tracking data, we choose to provide labels for the type of pass coverage used by players, for several reasons. First, this information is not available publicly on a play-by-play level. NFL teams may collect this information on their own, or turn to third-party companies like Pro Football Focus to provide this information. Second, passing has become increasingly important in the NFL in recent years, with teams passing the ball (vs. running the ball) more than they ever have before. Despite this, there has been limited public research into the play of defensive backs and the efficacy of different coverage schemes. Third and most importantly, this problem is challenging. Unlike the work of Miller and Bornn (2017) or Chu et al. (2019), who provide annotations for offensive player movement, we address an analgous problem for defensive players. They key difference here is that offensive players control their movements on the playing surface, while defensive players act in reaction to the offensive players. This means that techniques involving clustering trajectories are not appropriate for identifying defensive coverage schemes. Instead, we focus on generating a rich set of features describing the movements of defensive players in relation to their teammates and to their counterparts on the offensive side (e.g. wide receivers). We use this extensive set of features to identify groups of plays with similar coverage, then manually assign labels like “man coverage” or “zone coverage” (described in Section 2) to identified groups.

The rest of this paper is organized as follows: We provide a background on passing in football and its associated defensive coverage schemes in Section 2. We describe our methods for feature generation and clustering in Section 3. We present results, comparing the different clustering approaches and providing analysis of the number of groups of coverage types in Section 4, and we discuss future directions for this work in Section 5.

2 Background

A defensive back in football is a player who lines up in the defensive backfield (typically at least a yard past the line of scrimmage, and to the outside of the field). Their primary objective is to prevent the offense from completing any passes by covering wide receivers (offensive players who typically line up on the outside). There are two different specialized positions within the defensive back: cornerback (CB) and safeties (split into “strong safeties”, SS, and “free safeties”, FS). Generally, CBs are more adept at providing close coverage on wide receivers and defending passes, and less adept at making tackles in the open field. The position requires speed and agility, and the ability to track a receiver in man coverage or occupy a space in zone coverage. Safeties usually start the play 10-15 yards beyond the line of scrimmage and can be thought of as the last line of defense. Their roles often depend on how the personnel used by the offensive team: sometimes, they provide additional help in defending long passes; other times, they provide man coverage on additional offensive receivers that are on the field. They are also responsible for reading the play, quickly determining if it is a run or pass play, and reacting accordingly. The typical alignment of players at each of these positions is shown in Figure 1.

Figure 1: Defensive Backs in Football

2.1 Man Coverage

In man coverage, a DB is assigned to defend a specific offensive player (typically a wide receiver). Throughout the play, the DB follows that offensive player until the ball is thrown in an attempt to prevent that offensive player from making himself open and ultimately catching the ball. In man coverage, the DB is focused on the movement of the offensive player, often with his head turned toward the offensive player, not even looking at the ball. As such, the motion of the DB may tend to mirror that of the offensive player that the DB is covering. In the context of our problem, generating features that reflect this type of motion will be important, but challenging. This requires identifying the offensive player to which the DB is assigned to cover and building a set of features that capture this type of movement and distinguish it from that of other coverage types.

Figure 2: Different Zone Defenses

2.2 Zone Coverage

In zone coverage, each of the DBs is assigned a zone on the field to defend. These zones are constructed so that both the linebackers (who typically line up closer to the line of scrimmage and in the center of the field) and the DBs work together to provide a complete coverage of the possible passing areas in order to prevent the completion of a pass. Generally, when a player is in zone coverage, their eyes are on the quarterback and their motion is generally reactive to the pass of the ball rather than focusing on a specific offensive player. This isn’t always the case, however. For example, if a DB in zone coverage sees that there is only one offensive player near his zone, his responsibility shifts, so that he must now cover that single player in a way that resembles man coverage. In other words, the locations and trajectories of teammates, opponents, and the ball can cause the DB to adjust his assignment in the middle of the play. This “hybrid” style of defensive coverage can be difficult to measure through feature generating, and can make the automatic identification of zone coverage challenging.

Most of the time, teams employ some combination of man coverage and zone coverage on a single play. For example, some teams would play a variation of Cover 2 (example shown in Figure 2), where the CBs are in man coverage and the safeties are in zone coverage.

2.3 Related Work

Since the NFL tracking data was released early 2019, there is not much work done involving providing annotations for tracking data in football. As mentioned above, the work of Chu et al. (2019) is a notable exception here. However, similar work has been done in other sports. Most specifically, work with tracking data has been done in other “invasion” style sports, such as soccer and basketball.

In work done in soccer, Bialkowski et al. (2014) have done work in using player tracking data in order to identify a team’s playing style. They are able to identify a team just from the player’s position data on the field by gathering features regarding reducing entropy of role-specific occupancy maps. In doing so, the authors assign some formation-based roles to individual players, and then create occupancy maps based on these roles that are assigned.

In some work done in basketball, Patrick Lucey and Matthews (2014) use the spatio-temporal changes in the team formation in order to determine what features allow for a player to create an “open” shot. Similar to Bialkowski et al. (2014), the authors create role-based features by initially assigning each player a role. In this case, they assign each of the five players on the court one of the traditional five basketball positions. After doing this, they can track the motion of the roles rather than individual players in order to create a permutation-free set of features. Additionally, Miller and Bornn (2017) annotate set plays in the NBA, as discussed above.

More work has been done in invasion sports regarding player tracking in determining formations of teams. Gudmundsson and Horton (2016) provides an excellent overview of this topic, as well as a detailed overview of the different ways that tracking data has been used in sports over the last decade. We encourage interested readers to visit this paper for more information on this topic.

Most of the work that has been done in this field has been to determine how teams as a whole attempt approach the game. In this paper, we attempt to determine how certain players on a team are behaving, in a sport where not much tracking work has been done.

3 Methods

In this section, we define a set of features describing the motion of each DB, and then apply several clustering methods to these features. In this way, we attempt to encapsulate the motion of the DBs and make a distinction between different types of coverage in an entirely unsupervised way.

3.1 Data

We use data from the NFL’s inaugural Big Data Bowl, which begin in December 2018 and ended in March 2019. This dataset consists of game data from the first six weeks of the 2017 NFL season. Each play from each game uses the league’s player and ball tracking technology to record the locations and trajectories of all 22 players on the field (and the ball) throughout the duration of the play, at a rate of 10 Hz (10 frames per second). For each player, each frame contains their and coordinates on the field with and for each frame. Furthermore, for each player each frame consists of their speed , displacement from last position, and direction of motion on the field described by the angle ), . A subset of frames are also labeled with text indicating the on-field event that happened at that frame (e.g. ball is snapped, first contact, etc). For the purposes of this paper, the relevant events are mostly ball_snap and pass_forward. We use this data to generate features characterizing the movement of the DB, such that a clustering of these features will result in a meaningful interpretation of their coverage type.

3.2 Feature Generation

The time of motion that is analyzed is exactly the time between the ball being snapped (ball_snap) and the ball being thrown (pass_forward

). We choose to do this because the assumption is after the ball is thrown players are no longer in coverage, but rather attempting to defend the targeted receiver or pursue the player who catches the ball. The movements of these players during this portion of the play should be uniform regardless of the coverage scheme, and thus would not provide any value in the clustering model. For each play, during the time period described above, and for each DB, we generate a feature vector that consists of the features described in Table

1.

Feature Name Feature Description Feature Equation
VAR_X Variance in the x coordinate
VAR_Y Variance in the y coordinate
SPEED_VAR Variance in the speed
OFF_VAR Variance in the distance from the nearest offensive player at every frame ,

DEF_VAR Variance in the distance from the nearest defensive player at every frame ,

OFF_MEAN Mean distance from the nearest offensive player at every frame ,

DEF_MEAN Mean distance from the nearest defensive player at every frame ,

OFF_DIR_VAR Variance in the difference in degrees of the direction of motion between the player and the nearest offensive player

OFF_DIR_MEAN Mean difference in degrees of the direction of motion between the player and the nearest offensive player

0_DIS_OFF Distance from the nearest offensive player right after the ball snap

2_DIS_OFF Distance from the nearest offensive player right before the ball is thrown

4_DIS_OFF Distance from the nearest offensive player exactly halfway between the frame of the ball_snap and the frame of the ball thrown

RAT-MEAN Mean ratio of the distance to the nearest offensive player and the distance from the nearest offensive player to the nearest defensive player
RAT-MID Ratio of the distance to the nearest offensive player and the distance from the nearest offensive player to the nearest defensive player halfway between snap and throw ,
RAT-PASS Ratio of the distance to the nearest offensive player and the distance from the nearest offensive player to the nearest defensive player at pass ,
RAT-SNAP Ratio of the distance to the nearest offensive player and the distance from the nearest offensive player to the nearest defensive player at snap ,
RAT-VAR Variance of the distance to the nearest offensive player and the distance from the nearest offensive player to the nearest defensive player ,
Table 1: Features describing the movement of defensive backs in pass coverage.

From the features we generate, we would expect the five ratio (RAT-X) features to be most helpful in determining man vs. zone coverage, because in man coverage we would expect the DB to follow the offensive player he is covering very closely, whereas in zone coverage this is not necessarily true. Since there is generally a “hard” assignment for each defender in man coverage (i.e. each defender has a specific player to cover), the ratio of the distance from the DB to the closest offensive player, to the distance from DB’s closest teammate to the same offensive player should be fairly small. That is, we expect the DB in man coverage to have a very small distance to his assignment, and any of the other remaining defenders to have a comparably larger distance to that player. We compute this value throughout the course of the play, and summarize those values with five quantities: the mean and variance of this value, in order to summarize the changes in this value throughout the play; the value of this quantity at the snap; at the time the ball is thrown; and at the mid-point between these two timepoints.

Furthermore, another feature we expect to differentiate coverage types is the OFF_DIR_VAR and OFF_DIR_MEAN. This is because, similar to the above logic, a player in man coverage would be following their assignment around the field and would be much more “reactive” to the motion of the offensive player. Hence we would expect the direction of motion to be almost the same as their assignment throughout the course of the play.111A lagged version of this may also prove valuable, as it may account for delays in reaction time. Implementing this is an exercise left to future work. For a player in zone coverage, they are generally watching the quarterback rather than watching a receiver, and thus their motion might be more static, so that the difference in the direction of motion to the nearest offensive player would both be different and would have more variance.

Figure 3: Pairs Plot of Features

Figure 3 shows a pairs plot of the features described in Table 1. While modality in this plot is limited, we see some group structure in variables like OFF_DIR_MEAN and the ratio variables described above.

3.3 Clustering

Clustering is the process of partioning observations in a dataset into groups without respect to some response variable

(Hartigan, 1975). Often, this process is referred to as “unsupervised learning,” since clustering models can be used to assign labels for discrete groups without training data. Clustering algorithms come in many forms, and can be characterized in at least two ways.

First, we can characterize clustering algorithms by their method for labeling observations, as “hard” or “soft.” Hard clustering algorithms partition the data into distinct groups, with hard assignments, and no uncertainty in those assignments. Soft clustering algorithms allow for uncertainty in the cluster assignments by, for example, assigning probabilities of membership in each cluster.

Second, we can characterize clustering algorithms by their method for separating observations into groups, with “distance-based” and “density-based” being the two most common approaches. Distance-based clustering algorithms compute the distance between each pair of observations in the data, and apply algorithms to those distances to partition groups into clusters. Examples of distance-based clustering algorithms include -means clustering and hierarchical clustering (Hartigan, 1975)

. Density based clustering algorithms are more statistical in nature, using the properties of probability density functions to define clusters. For example, Density-based spatial clustering of applications with noise (DBSCAN) uses non-parametric density estimation and mode hunting within that density estimate to identify clusters of observations

(Ester et al., 1996)

. Similarly, mixture modeling fits a mixture of (typically parametric) probability distributions to the data, so that the properties of the resulting clusters come naturally from the mixture

(McLachlan and Peel, 2000). One advantage to density-based methods is that the resulting clusters are typically more interpretable, since we can characterize each cluster by the features of the density we fit to the data (e.g. each group’s mean and variance in mixture modeling).

While many clustering algorithms exist, we focus on two approaches that are representative of the classes of clustering algorithms we define above: mixture modeling, which is density-based and yields soft cluster assignments; and hierarchical clustering, which is distance-based and yields hard cluster assignments.

3.3.1 Mixture Models

Mixture modeling, or model-based clustering, seeks to fit a mixture of probability density functions to a dataset, where each density is representative of a single group or cluster. A mixture model can be written in the form:

where indexes over the groups (pieces of the mixture model), represents the density for group with parameters , and is the overall mixture distribution.

Commonly, Gaussian mixtures (with representing a Gaussian probability density function) are used because of their ease of implementation and theoretical properties. That said, any parametric distribution (or mixture of parametric distributions) can be used (Banfield and Raftery, 1993). The choice of is left to the user, and is commonly determined by searching over a range of possible values and determining the best with an evaluative measure like the Bayesian Information Criterion (BIC), though many methods for doing this exist. We direct interested readers to McNicholas (2016) for a complete overview of model-based clustering and the related literature.

In this paper, we take to be Gaussian densities, and we test several different values of , with results provided in Section 4. For the remainder of this paper, we refer to our Gaussian mixture modeling approach as GMM. One additional benefit of GMM is that we can obtain cluster membership probabilities from the mixture model for any observation, including data from a holdout set of data. This will prove useful when evaluating the clustering methods later.

3.3.2 Hierarchical Clustering

Agglomerative hierarchical clustering or hierarchical clustering (HC) is a distance-based clustering algorithm that partitions observations into hard clusters (Ward, 1963). HC is flexible in several ways. First, the algorithm requires only the dissimilarities between each observation, and those dissimilarities can take many forms. Commonly, Euclidean distance (“as the Nazgul flies”) is used, but other approaches (e.g. Manhattan distance, Mahalanobis distance, etc) are suitable. Second, the choice of linkage type, or the method by which observations are linked and formed into clusters, is flexible and up to the user. Common examples include single linkage, complete linkage, average linkage, minimax linkage (Bien and Tibshirani, 2011), Ward’s method, etc. Each approach comes with a different set of typical characteristics, and there is no single approach that fits all problems. Third, the choice of the number of clusters us flexible, up to the user, and does not influence the algorithm. Specifically, HC produces a hierarchical tree of clusterings, where each level of the tree combines two new clusters of points, so that the same resulting tree can be used to identify the two-cluster solution, the three-cluster solution, the four-cluster solution, and so on (Hartigan, 1975).

The algorithm works as follows: all observations begin in their own cluster. Then, the algorithm iterates through two steps until all points are linked and the clustering tree is fully formed: (1) Compute the dissimilarities between all observations/clusters (using the chosen distance/dissimilarity measure), and (2) link together the two closest observations/clusters (using the chosen linkage type). Algorithms for HC typically require run-time (Hartigan, 1975).

In this paper, we use HC with simple Euclidean distance (since all of our features are numeric) and Ward’s method for linkage. Ward’s method is a better fit for our data than a method like single linkage, for example, since single linkage is not appropriate when the number of true clusters is small, as it is in our case. Other linkages, such as minimax linkage and complete linkage, may also be appropriate; we did not find any noticeable changes in our results when trying alternative linkage methods (aside from single linkage).

3.4 Evaluating Clustering Results

Partition 2 - same cluster Partition 2 - different cluster
Partition 1 - same cluster A B
Partition 1 - different cluster C D
Table 2: Notation for the cross-tabulation of observation pairs with two different partitions (rows: partition 1, columns: partition 2).

Many approaches exist for evaluating clustering results. Given that we use common clustering algorithms like GMM and HC, a standard evaluation metric will suffice. For this, we choose the adjusted Rand index (ARI), which measures the similarity between two sets of clustering results, adjusting for chance agreement

(Hubert and Arabie, 1985). Following Steinley (2004), we use the notation for observation pairs with two different partitions displayed in Table 2, with the ARI calculated as following:

(1)

where from Table 2. Under completely random partitions , and an is possible indicating that the pairwise comparisons are worse than random. The maximum value of 1 indicates that the two partitions being compared are identical to each other.

We choose the value of with the highest ARI in a predictive setting to determine the “best” number of clusters using the following procedure:

  1. Identify all cornerbacks on all passing plays in the NFL tracking data

  2. Randomly split the data into a two pieces: a training set (80% of the data) and a testing set (20%)

  3. Obtain the cluster labels from the testing set

  4. Fit a GMM with clusters on both the training and test sets, for

  5. For each :

    1. Apply the training set GMM to the test set, and obtain the predicted cluster membership probabilities and corresponding cluster labels to this set

    2. Compute the ARI between the predicted labels and the original cluster labels from the test set (from step 3)

  6. Repeat this process 100 times, and compute the average ARI for each

4 Results

Importantly, there is no “ground truth” data to which we can compare our clustering results. However, we can compare the clusterings to each other, assess the number of clusters, and characterize the clusters. We do this in Section 4.3. Then, we use our GMM to demonstrate how the predicted coverage type probabilities can change throughout the course of a play (Section 4.4). Finally, we provide an analysis of coverage types by player, team, and game situation (Section 4.3).

For all results in this section, we apply the clustering models to only cornerbacks, since the features distinguishing the type of coverage by safeties should differ from those of cornerbacks. That said, the same process could be applied independently to the safety position. This exercise is left to future work.

4.1 Determining the Appropriate Number of Clusters

First, we assess the appropriate number of clusters. We hypothesize that two clusters, representing man and zone coverage, will yield the best fit to the data. To assess this, we use the procedure detailed in Section 3.4.

Figure 4: The ARI scores for the number of clusters from our features

Figure 4 shows the results of this procedure. The two-cluster solution for GMM yields by far the best results in this predictive setting, with a very high ARI of over 0.9. This indicates that the GMMs fit on random train/test splits are in very strong agreement when the training set GMM is used to predict the cluster labels from the out-of-sample test set. For the remainder of this paper, we focus on the two-cluster solution.

4.2 Determining Zone vs. Man Coverage

While we cannot say definitively if a player is intending to play man or zone coverage, we can make some determination of the type of coverage visually. We examined 100 animations of plays and manually determined whether each DB on each play was playing man, zone, or unknown coverage.222In an appeal to our own authority, we note that the co-author who undertook this task has experience playing defensive back in high school football. Using this information in conjunction with the cluster membership probabilities, it is fairly easy to tell which cluster corresponds to man coverage and which corresponds to zone coverage. We provide illustrations of this in Section 4.4.

4.3 Comparing Results from Clustering Models

Our two clustering approaches, HC and GMM, show some agreement in their cluster labels, but are not close to being in full agreement. The four fold plot in Figure 5 demonstrates this.

Figure 5: Four Fold Plot of the two clustering models used.

Four fold plots provide a visualization of a 2-by-2 contingency table, and allow for a visual inspection of the independence or association between the two underlying dichotomous variables

(R Core Team, 2018)

. This particular plot shows the association of the two-cluster solutions for GMM and HC. We see GMM labels over 9000 coverages as “man,” and over 6000 coverages as “zone,” where a single coverages refers to a single DB’s coverage type on a single play. HC, on the other hand, yields a much more unbalanced clustering, with over 14,500 coverages labeled “man,” and only about 2000 coverages labeled zone. Moreover, the plot provides visual representation of a chi-square test for independence of the two clusterings. Specifically, since the 95% confidence bands on adjacent folds do not overlap, we can reject the null hypothesis of independence. This is to be expected, since we fit the two clusterings on the same data and using the same features, so their labels should be associated.

Another takeaway from this plot is that HC seems to struggle to identify zone coverage. The GMM predicted zone 42% of the time and man 58% of the time. The HC model predicted zone 12% of the time and man 88% of the time. Furthermore, there were 5760 coverages where the highest cluster membership probability from GMM indicated zone coverage, but the hard cluster assignment from HC indicated man coverage. While GMM is by no means ground truth, the class imbalance in the HC results is not favorable, leading us to conclude that the GMM model is a better fit. The three plays detailed in the next section provide an illustration of the results of the clustering process, and help demonstrate why the GMM model is favorable in this context.

4.4 Characterizing Pass Coverage Throughout Plays

In modern defensive schemes, teams may try to disguise their pass coverage at the start of the play, in order to confuse the opposing quarterback. Other schemes can involve a “read and react” approach to pass coverage, where specific coverage assignments depend on what happens at the start of the play.

Interestingly, the mixture model does a good job of recognizing these scenarios. Coverage type probabilities often change substantially from the time of the snap to when the play has had a few seconds of “burn-in” time. Some examples are given below. In each of these examples, the offense is in red, and the defense is in blue. The arrows point to the players we focus on.

Figure 6: Play 3765 from Game ID 2017091000, with offense shown in red, and defense shown in blue. The play starts with the formation in the first frame and the ball is thrown in the last frame. The middle frame is exactly in between. In this case the top arrow points to player with jersey 24 (Defense-24) and the bottom to player with jersey 20 (Defense-20).

In Figure 6, we can see the probabilities assigned to man and zone coverage at each frame throughout the play by the GMM. By the end of the play, the GMM predicts that Defense-24 (top arrow) and Defense-20 (bottom arrow) to have been in zone coverage. We can see that they actually start out lined up as if they were going to play “press” coverage (a type of man coverage where the DB physically prevents the receiver from beginning his route at the start of the play), leading the GMM to predict both players to be playing man coverage with high probability. As the play develops, Defense-20 and Defense-24 do not follow the men they initially lined up against and instead occupy their assigned zones. We see this change in the second and third frames, when they clearly are in zone coverage (as they are occupying a space rather than following another offensive player). The model reflects this with high probability. In this play, the hierarchical clustering also labels each player’s coverage as zone coverage.

Figure 7: Play 174 from Game ID 2017091004, , with offense shown in red, and defense shown in blue. The play starts with the formation in the first frame and the ball is thrown in the last frame. The middle frame is exactly in between. In this case the top arrow points to player with jersey 24 (Defense-24) and the bottom arrow points to player with jersey 23 (Defense-23).

In Figure 7, the offense is lined up with three receivers, and the defense does not line up in a way that indicates man coverage with high probability: each of the players that we are tracking are lined up slightly farther away from the player they would be assigned to cover, possibly indicating that they are playing zone coverage. The GMM reflects this, predicting zone for Defense-23 and man for Defense-24, but only modest probabilities of 0.8 and 0.7, respectively. As the play develops, however, we see that both DBs follow a specific offensive player until the ball is thrown, indicating that they were actually playing man coverage the whole time. This change in coverage type probability from GMM is reflected in the middle frame as the probability of man spikes for both players, and finishes in the final frame with high probabilities of man coverage for both players.

Figure 8: Play 1193 from Game ID 2017091713, with offense shown in red, and defense shown in blue. The play starts with the formation in the first frame and the ball is thrown in the last frame. The middle frame is exactly in between. In this case the top arrow points to player with jersey 34 (Defense-34), the middle arrow to player with jersey 21 (Defense-21), the bottom arrow points to player with jersey 23 (Defense-23) (in the first frame). In the last frame 21 and 34 swap arrows.

In Figure 8, the offensive is lined up in a “shotgun” formation with four wide receivers, two on each side of the offensive line. Defense-21 and Defense 23 are playing relatively far from their closest receivers, while Defense-34 is relative close to his. The GMM predicts this starting formation to be man from each of the players, but with some uncertainty in the probabilities. As the play continues, Defense-21 and Defense-34 actually swap positions. In frame 1, Defense-34 is defending Offense-18, but as the play develops, we see that Defense-34 is in man coverage against Offense-88, who runs out from the offensive backfield. This unusual starting point and pattern of motion by Offense-88 leads to a reduced probability that Defense-34 is in man coverage according to the GMM, but it is clear Defense-34 is following Offense-88. Defense-21 doesn’t follow Offense-17 and instead occupies a zone in the middle frame, where he does have a similar pattern of motion to Offense-17. Finally, Defense-23 doesn’t actually move much from his initial position at the start of the play, but there is an offensive player who comes towards him. Interestingly, the model gives about equal probability that Defense-23 is zone (54%) coverage and man (46%) coverage. His lack of motion is more common with zone coverage, but his proximity to the receiver in that area indicates man coverage. Thus, the model gives a fairly uncertain prediction about his coverage type.

4.5 Analysis of Coverage Types by Pattern of Motion, Player, Team, and Situation

Figures 9 and 10

show the patterns of motion of cornerbacks classified as playing man or zone coverage, respectively. Here, we see no apparent relationship between the patterns of motion and the probability of man or zone coverage. This makes sense in context, since the patterns of motion of CBs are typically reactionary to what the opposing receiver is doing. This provided partial evidence that a trajectory clustering approach like that of

Chu et al. (2019) or Miller and Bornn (2017) would not be appropriate for identify coverage types of defensive backs.

Figure 9: Probability motion is classified as Man
Figure 10: Probability motion is classified as Zone

Figure 11 shows the proportion of cornerback coverages in man (blue) and zone (orange) by quarter. While some small differences exist in the first four quarters, they are mostly negligible, meaning we see no apparent trend in the type of coverage throughout the game. Interestingly, we see an increase in man coverage in overtime. However, this result is not statistically significant, since there were only 40 overtime plays in our dataset from the first six weeks of the 2017 season. This brief analysis does not control for factors like the score differential or opposing offensive formation, which may influence coverage type. We leave that exercise to future work.

Figure 11: Man:Zone Percentage by Quarter

Figure 12 shows no apparent relationship between the down and the coverage type of defensive backs.

Figure 12: Man:Zone Percentage by Down

Figures 13 and 14 show the top-10 teams by proportion of zone and man coverage, respectively. Tampa Bay, Chicago, and Green Bay have the highest rate man coverage play during this short span of six weeks, while Washington, Buffalo, and the New York Giants have the highest rate of zone coverage. Overall, man coverage is used more often than zone coverage by all 32 NFL teams in this provided sample of data.

Figure 13: Top 10 teams with the highest Zone Coverage percentage
Figure 14: Top 10 teams with the highest Man Coverage percentage

Finally, Figure 15 and 16 show the top-10 cornerbacks by their proportion of man and zone coverage, respectively (minimum 50 coverages). Bryce Callahan from the Chicago Bears led the NFL in percentage of man coverages during the first six weeks of the 2017 season with almost 80%, followed by Damarious Randall, Kevin King, Phillip Gaines, and others. Joshua Shaw, who played for the Bengals in 2017, led the league in percentage of zone coverages, with about 60%, followed by Janoris Jenkins, Bobby McCaine, Marcus Peters, and others.

Figure 15: Top 10 players with the highest Man Coverage percentage
Figure 16: Top 10 players with the highest Zone Coverage percentage

5 Discussion and Future Work

We present an unsupervised approach for identifying and annotating the type of pass coverage of defensive backs during passing plays in the NFL, requiring no ground truth labels and minimal human oversight. We design a rich set of features that help distinguish between types of coverage and can be updated at each point during a play.

We test two clustering models, Gaussian mixture modeling and hierarchical clustering, in addressing this problem, and find that GMM’s flexibility, interpretability, and soft cluster assignments are more suitable for this problem.

We use an out-of-sample prediction approach and the well-studied adjusted Rand index to determine the appropriate number of clusters. Using this approach, we find that the two-cluster solution yields the best ARIs, which conveniently matches our expectations given what we know about the dichotomy of coverage types in football to “man” and “zone” coverage. Furthermore, through manual review of animated example plays from each cluster, we assign “man” and “zone” labels to the groups identified by the two-cluster GMM and HC solutions.

The GMM’s probabilistic cluster assignments are advantageous in this context, as we demonstrate through an examination of three plays. In particular, since modern defensive schemes involve disguising coverage types at the start of plays and reacting to what the offense is doing, we examine the GMM cluster membership probabilities at different points throughout the course of a play, showing how these probabilities change in reaction to the patterns of motion of cornerbacks and their proximity to opposing receivers.

We provide a brief analysis of coverage types by different game situations, though we acknowledge that there is more work to be done here. However, that encapsulates the primary purpose of this research: to provide additional annotations to NFL player and ball tracking data that will allow future researchers and NFL teams to explore interesting and innovative research problems. We look forward to seeing such new research.

In future work, we apply this work to safeties and, when appropriate, linebackers. We hope to examine using the coverage type of one player to help inform the coverage type prediction of another player. For example, knowing that the left cornerback is playing in man coverage could indicate that the right cornerback is likely also going to play man coverage. Additionally, incorporating team-specific features may help, since certain defensive coordinators are though to prefer certain defensive schemes.

Finally, in future work, we hope to provide additional annotations of other on-field events (actions, coverage schemes, etc). The mixture modeling approach we use here should serve as a foundation for this future work, although a new set of features will need to be designed for each specific annotation problem.

References