Multimodal Shot Prediction Based on Spatial-Temporal Interaction between Players in Soccer Videos

Abstract

Sports data analysis has significantly advanced and become an indispensable technology for planning strategy and enhancing competitiveness. In soccer, shot prediction has been realized on the basis of historical match situations, and its results contribute to the evaluation of plays and team tactics. However, traditional event prediction methods required tracking data acquired with expensive instrumentation and event stream data annotated by experts, and the benefits were limited to only some professional athletes. To tackle this problem, we propose a novel shot prediction method using soccer videos. Our method constructs a graph considering player relationships with audio and visual features as graph nodes. Specifically, by introducing players’ importance into the graph edge based on their field positions and team information, our method enables the utilization of knowledge that reflects the detailed match situation. Next, we extract latent features considering spatial–temporal interactions from the graph and predict event occurrences with uncertainty based on the probabilistic deep learning method. In comparison with several baseline methods and ablation studies using professional soccer match data, our method was confirmed to be effective as it demonstrated the highest average precision of 0.948, surpassing other methods.

1. Introduction

Sports analytics has become the most crucial and extensive method for tactical planning and enhancing athlete performances [1,2,3,4]. This increased importance of sports analytics can largely be attributed to significant advancements in measurement technology, which have enabled the collection of a diverse range of data [5]. In professional soccer, Electronic Performance Tracking Systems (EPTS), such as optical-based tracking systems and wearable devices, facilitate the gathering of player and ball position data, known as tracking data [6,7]. This system also monitors physical performance, including heart rate, speed, and distance covered during a match [8,9,10]. In addition to insights derived from EPTS, event stream data with details of actions such as pass, dribble, and shot are also employed. It covers who played, when, and where and is used to evaluate how each action impacted the match result more comprehensively than traditional stats [11,12]. However, acquiring such high-quality data often requires expensive measurement equipment and manual annotation by experts [13,14]. As a result, many professional teams with limited budgets and amateur athletes find it challenging to benefit from analytical techniques that rely on these data. Therefore, there is a growing demand for alternative analysis methods less reliant on costly equipment and human intervention.

Among the many evaluation methods proposed for soccer, event prediction-based performance evaluation [11,12,15,16,17,18,19] is one of the most remarkable methods in recent years. Many of these methods predict specific match events that will occur next or subsequently from tracking data and event stream data by applying machine learning techniques. In addition, it is possible to evaluate individual or series of events based on their predicted probabilities. For instance, the study in [12] performs the prediction of major attacking events (pass, dribble, cross, and shot) by using a recurrent neural network (RNN) or transformer [20]. This study also quantitatively evaluates each team’s possession with the likelihood of a shot or cross at the occurrence time of an individual event. However, event prediction methods described above use tracking data and event stream data, which require costly equipment and expert intervention. Here, the use of video data that are widely available in sports can be considered as one of the alternative approaches. With the rapid development of computer vision [21] and the release of large video datasets such as SoccerNet [22], there has been widespread research conducted on video analysis techniques (multi-player tracking [23,24,25,26], event detection/classification [27,28,29,30], video summarization [31,32], etc.) in the field of soccer analysis. A few proposals have also focused on predicting who receives the pass and the occurrence of fouls from videos and pose information of players [33,34]. Based on a wealth of visual information in match videos, the above methods provide valuable insights without using expensive instruments. Additionally, since the direct use of video reduces reliance on manual annotation, it is expected to become a more accessible and efficient solution for tactical planning and improving player performance at various competition levels. However, it is crucial to predict the occurrence of major events such as passes and shots rather than who receives a pass or the foul occurrence for player evaluation, and thus technology is required to predict these events based on soccer videos.

To tackle the aforementioned issue, we propose an event prediction method in soccer videos focusing on the shot, one of the primary events. In the proposed method, we employ unedited soccer videos, which are known as scouting videos, captured from a bird’s-eye view with a single camera. In contrast to the broadcast soccer video, the scouting video can adequately capture lengthy spatial–temporal relationships between most players since it does not focus on the ball carrier and does not include scene highlights, as shown in Figure 1. However, the extremely small image size of the ball in unedited soccer videos makes it difficult to focus on players performing actions with the ball. Hence, by constructing the relationships between players in each video frame as a graph with audio-visual information as nodes, the proposed method enables shot prediction from those videos. To introduce the importance of players based on field position and team information into the graph edge, we take advantage of knowledge that reflects detailed match situations. Then, our method extracts latent features from graphs by incorporating graph convolutional networks (GCNs) [35] and a graph convolutional recurrent network (GCRN) [36]. The graph convolutional recurrent neural network (GCRNN), which combines the above two deep learning methods, enables the extraction of latent features that consider both spatial and temporal relationships of the graph. Finally, the proposed method predicts shots with Bayesian neural networks (BNNs) [37]. The BNN is a method that can make predictions with uncertainty by introducing probability distributions for the weight parameters of the neural network. Thus, the BNN enables more reliable predictions than conventional methods based on deterministic machine learning. Note that we have previously presented the preliminary results in previous studies [38,39,40,41], where we demonstrated the effectiveness of incorporating visual information and players’ distances into graphs to achieve shot prediction from soccer videos.

In summary, the main contributions of our method are as follows:

• For the shot prediction from soccer videos, we use a graph that introduces soccer-specific knowledge (players’ importance based on the field positions and team information) with audio-visual information as graph nodes.
• Predicting and obtaining uncertainty with BNNs enables us to provide more robust predictions.

The remainder of the paper is organized as follows. In Section 2, we describe the shot prediction method based on multimodal features in soccer videos with GCRNN and BNNs. Experimental results and the discussion are presented in Section 3 to verify the effectiveness of the proposed method. Finally, we conclude our work in Section 4.

2. Proposed Multimodal Shot Prediction in Soccer Videos

We explain the details of the proposed multimodal shot prediction method in soccer videos, as shown in Figure 2. In the proposed method, we use a graph with audio-visual features obtained from the input soccer videos as nodes. To enhance the feature representation capability, we construct a complete bipartite graph that introduces players’ importance based on their field positions and teams into the graph edge. This is described in more detail in Section 2.1. After constructing the graph, the proposed method extracts latent features through GCRNN. In this way, our method can make predictions with latent features that incorporate relationships between connected nodes and between graphs of different time steps, which correspond to spatial–temporal relationships. This is discussed further in Section 2.2. As a final step, we achieve the probability predictions of shot occurrence while accounting for uncertainty with BNNs, which is explained in Section 2.3.

2.1. Graph Construction

To effectively incorporate the match situation obtained from the soccer video, we employ the relationship between players in each frame as a graph. In the proposed method, we obtain audio segments and two types of images with full-frame size and each player region, respectively, at each time step from the soccer video. Although the audio segment does not include commentary, it contains the cheers of spectators and other auditory responses, which are considered to indicate the importance of the play and the tension involved. Especially, significant or sudden increases in cheers suggest a high likelihood of important plays, such as scoring opportunities or plays leading to them, making it effective for shot prediction. As for the object detector, we employ Mask R-CNN [42], which also performs instance segmentation simultaneously. The detected humans are used for classifying the team, referee, or other class that each detected person belongs to [43]. After converting the audio sequence into a spectrogram image, we extract features from the respective type of image. As feature extractors in the proposed method, we use ResNet-101 [44] pre-trained with ImageNet [45]. Then, to enhance the representation capability of the node features, the proposed method utilizes the features that concatenated all types of extracted audio-visual features as graph nodes ${\mathit{X}}_t\in {\mathbb{R}}^{N\times 3D}$ calculated through the following equation:

$$\begin{array}{c}\hfill {\mathit{X}}_t=\left[\left[{\mathit{x}}_{1,t}^{\mathrm{player}};{\mathit{x}}_t^{\mathrm{frame}};{\mathit{x}}_t^{\mathrm{audio}}\right],\left[{\mathit{x}}_{2,t}^{\mathrm{player}};{\mathit{x}}_t^{\mathrm{frame}};{\mathit{x}}_t^{\mathrm{audio}}\right],\cdots ,\left[{\mathit{x}}_{N,t}^{\mathrm{player}};{\mathit{x}}_t^{\mathrm{frame}};{\mathit{x}}_t^{\mathrm{audio}}\right]\right],\end{array}$$

(1)

where ${\mathit{x}}_{n,t}^{\mathrm{player}}\in {\mathbb{R}}^D$ ($n=1,\dots ,N;N$ representing the total number of detected players) is the visual feature of the $n$-th player’s region at time step t. ${\mathit{x}}_t^{\mathrm{frame}}\in {\mathbb{R}}^D$ and ${\mathit{x}}_t^{\mathrm{audio}}\in {\mathbb{R}}^D$ are audio and visual features of the full-frame image and the audio segment, respectively. Notably, D denotes the dimension of features. In this way, with the use of audio-visual information as graph nodes, the proposed method can construct a graph that has high feature representation capability.

Next, we describe the detailed settings of the graph edge in the proposed method. In soccer as a dynamic team sport, players’ positions on the pitch and their relative distances from other players dramatically change over time. All of these factors significantly contribute to the instantaneous understanding of the players’ environment and their decision-making [46,47]. Thus, we use the graph edge based on players’ field positions and the team information, as well as their distances from other players. The weights of the graph edges at time step t are represented by the adjacency matrix ${\mathit{A}}_t\in {\mathbb{R}}^{N\times N}$ as follows:

$$\begin{array}{cc}\hfill {\mathit{A}}_t^{\left(ij\right)}& =\left\{\begin{array}{cc}0& \left(\mathrm{team}\right(i)=\mathrm{team}(j\left)\right)\hfill \\ {\displaystyle \frac{a_t^{\left(ij\right)}}{{\sum }_{ij}{a}_t^{\left(ij\right)}}}& \left(\mathrm{team}\right(i)\ne \mathrm{team}(j\left)\right)\hfill \end{array}\right.,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill a_t^{\left(ij\right)}& =exp\left(-\frac{\left|\right|{\mathit{p}}_t^{\left(i\right)}-{\mathit{p}}_t^{\left(j\right)}\left|\right|}{\mathrm{xG}\left({\mathit{p}}_t^{\left(i\right)}\right)\times w_a}\right),\hfill \end{array}$$

(3)

where ${\mathit{p}}_t^{(·)}$ denotes the 2D position on the field of each player, and $w_a$ is the edge weight coefficient. The function $\mathrm{xG}(·)$ is the expected goals (xG) at the attacking player’s position. In the proposed method, we employ the xG shown in Figure 3. It is established based on data provided by American Soccer Analysis (https://www.americansocceranalysis.com/, accessed on 4 April 2024) and is assigned a value based on the mean xG of shots in that zone in Major League Soccer matches. Furthermore, the weights are connected to players of only different teams, that is, in a complete bipartite graph. In soccer, the closer the distance between players, the more important it is to consider their relationships. Since the relationships between distant players may introduce noise, the proposed method attenuates the influence of connections between distant nodes, thereby focusing primarily on relationships between nearby nodes with the exponential function $\exp (·)$. By incorporating xG, we enable predictions that focus on offensive players with a high likelihood of taking a shot and the defensive players around them. From the above, we can construct the weighted edge that considers the players’ importance and the relationship between players. By constructing such a graph, we can utilize the player interaction that closely reflects the complex match situation and affects the decision-making.

2.2. Graph Convolutional Recurrent Neural Network

To learn the spatial–temporal interaction of graphs and calculate latent features, we adopt GCRNN. In previous studies [48,49,50], several architectures of GCRNN have been proposed. The basic GCRNN architecture first updates the graph node features by applying a GCN to the graph data. The updated graph features are used to acquire hidden states by RNNs or its derivative methods such as gated recurrent unit (GRU) [51] and long short-term memory (LSTM) [52]. By inputting such hidden states to the GCN at the next time step, the node features of the graph can be updated to take into account not only the spatial relationships of the graph but also the temporal-dependent relationships. In the GCRNN model used in the proposed method, as input to the constructed graph in Section 2.1, we calculate the latent feature ${\mathit{F}}_t\in {\mathbb{R}}^{N\times D}$ through the following equation:

$$\begin{array}{cc}\hfill {\mathit{F}}_t& =\mathrm{ReLU}\left({\widehat{\mathit{A}}}_t[\mathrm{ReLU}\left({\widehat{\mathit{A}}}_t{\mathit{X}}_t{\mathit{W}}_f^{\left(0\right)}\right);{\mathit{H}}_{t-1}]{\mathit{W}}_f^{\left(1\right)}\right),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\widehat{\mathit{A}}}_t& ={\mathit{I}}_N+{\mathit{D}}^{-\frac{1}{2}}{\mathit{A}}_t{\mathit{D}}^{-\frac{1}{2}}.\hfill \end{array}$$

(5)

Here, ${\widehat{\mathit{A}}}_t$ represents the normalized adjacency matrix with added self-connections. ${\mathit{I}}_N$ and $\mathit{D}$ are the identity and degree matrices, respectively. ${\mathit{W}}_f^{\left(0\right)}\in {\mathbb{R}}^{3D\times D}$ and ${\mathit{W}}_f^{\left(1\right)}\in {\mathbb{R}}^{2D\times D}$ are trainable weight matrices. $\mathrm{ReLU}(·)=max(0,·)$ is a widely used activation function in neural networks. The proposed method uses two stacked GCN layers to calculate latent feature ${\mathit{F}}_t$. In the constructed complete bipartite graph, the node features are propagated two hops away in the graph, which means that the graph is updated by considering all players through the players of the opposite team. Furthermore, the final layer of GCNs is enhanced with the hidden state ${\mathit{H}}_{t-1}\in {\mathbb{R}}^{N\times D}$ of the GRU-based GCRN at the previous time step. ${\mathit{H}}_{t-1}$ is calculated from the following equation:

$$\begin{array}{cc}\hfill {\mathit{Z}}_t& =\beta ({\mathit{W}}_{xz}{\ast }_{\mathcal{G}}{\mathit{G}}_t+{\mathit{W}}_{hz}{\ast }_{\mathcal{G}}{\mathit{H}}_{t-1}),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathit{R}}_t& =\beta ({\mathit{W}}_{xr}{\ast }_{\mathcal{G}}{\mathit{G}}_t+{\mathit{W}}_{hr}{\ast }_{\mathcal{G}}{\mathit{H}}_{t-1}),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\tilde{\mathit{H}}}_t& =tanh({\mathit{W}}_{xh}{\ast }_{\mathcal{G}}{\mathit{G}}_t+{\mathit{W}}_{hh}{\ast }_{\mathcal{G}}({\mathit{R}}_t\odot {\mathit{H}}_{t-1})),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\mathit{H}}_t& =(1-{\mathit{Z}}_t)\odot {\tilde{\mathit{H}}}_t+{\mathit{Z}}_t\odot {\mathit{H}}_{t-1}.\hfill \end{array}$$

(9)

To determine the hidden state ${\mathit{H}}_t$ at each time step, GCRN uses the update gate ${\mathit{Z}}_t\in {\mathbb{R}}^{N\times D}$, the reset gate ${\mathit{R}}_t\in {\mathbb{R}}^{N\times D}$, and the new memory content ${\tilde{\mathit{H}}}_t\in {\mathbb{R}}^{N\times D}$. The update gate ${\mathit{Z}}_t$ helps determine how much past information to retain and pass to the next time step. The reset gate ${\mathit{R}}_t$ determines how much past information to forget. The new memory content ${\tilde{\mathit{H}}}_t$ is composed of the input ${\mathit{G}}_t$ and the past information retained by the reset gate ${\mathit{R}}_t$. Each gate of the GCRN receives the graph ${\mathit{G}}_t=([{\mathit{X}}_t;{\mathit{F}}_t],{\mathit{A}}_t)$ with features that combine the initial graph node and the nodes obtained through GCNs. The operator ${\ast }_{\mathcal{G}}$ is the graph convolution operator, and ⊙ denotes the Hadamard product. $\beta (·)$ and $tanh(·)$ are the sigmoid function and the hyperbolic tangent function, respectively. ${\mathit{W}}_{x·}\in {\mathbb{R}}^{D\times 4D}$ and ${\mathit{W}}_{h·}\in {\mathbb{R}}^{D\times D}$ are trainable weight matrices for computing update gates, reset gates, and candidate hidden states at time step t. This process allows the hidden state to be obtained while considering both graph-specific and latent features. In this way, the proposed method calculates latent features while capturing the spatial–temporal interactions.

2.3. Uncertainty-Based Event Prediction

In this subsection, we describe the uncertainty-based prediction and the loss function for the optimization of the proposed model.

2.3.1. Construction with Bayesian Neural Networks

The proposed model finally predicts the shot occurrence with BNNs. The Bayesian approach estimates the posterior distribution of the weights by integrating prior knowledge about the weights and information learned from the data. In this way, BNNs can calculate both model and data-dependent uncertainty about inference results and provide confidence in model decisions. In traditional evaluation methods [12,16] in soccer, inference results calculated from deterministic machine learning models are used, yet users have to be cautious about its use due to output with unknown confidence levels. Especially, since soccer matches do not always follow the same patterns as training data, uncertainty-based analysis and evaluation are needed for performance improvement. Therefore, this study tackles such a problem by not using evaluations from inference results, but predicting based on BNNs. In the proposed method, given the latent feature ${\tilde{\mathit{F}}}_t$ of unknown data, the probability of a shot occurrence is calculated according to a predictive probability distribution [53]. This predictive probability distribution $p\left({\tilde{\mathit{y}}}_t\right|{\tilde{\mathit{F}}}_t,\mathcal{D})$ is given as follows:

$$\begin{array}{c}\hfill p\left({\tilde{\mathit{y}}}_t\right|{\tilde{\mathit{F}}}_t,\mathcal{D})=\int p\left({\tilde{\mathit{y}}}_t\right|{\tilde{\mathit{F}}}_t,\mathit{\theta })p\left(\mathit{\theta }\right|\mathcal{D})d\mathit{\theta },\end{array}$$

(10)

where $p\left({\tilde{\mathit{y}}}_t\right|{\tilde{\mathit{F}}}_t,\mathit{\theta })$ is the conditional probability distribution of the event occurrence probability ${\tilde{\mathit{y}}}_t={({\tilde{y}}_t^{\left(n\right)},{\tilde{y}}_t^{\left(p\right)})}^{\top }$ given the model parameter $\mathit{\theta }$. Here, ${\tilde{y}}_t^{\left(n\right)}$ and ${\tilde{y}}_t^{\left(p\right)}$ are the negative and positive probability predictions of the unknown data at time step t, respectively. $p\left(\mathit{\theta }\right|\mathcal{D})$ is the posterior probability distribution of $\mathit{\theta }$ given the training data $\mathcal{D}$ including V soccer videos as follows:

$$\begin{array}{c}\hfill \mathcal{D}=({\mathcal{D}}_x,{\mathcal{D}}_y)=(\{{\mathit{F}}_{t,1},\dots ,{\mathit{F}}_{t,V}\},\{{\mathit{y}}_{t,1},\dots ,{\mathit{y}}_{t,V}\}),\end{array}$$

(11)

where ${\mathcal{D}}_x$ and ${\mathcal{D}}_y$ are the input features and target labels of the training data, respectively. The estimation of $p\left(\mathit{\theta }\right|\mathcal{D})$ based on Bayes’ theorem is calculated from the following equation:

$$\begin{array}{c}\hfill p\left(\mathit{\theta }\right|\mathcal{D})=\frac{p\left({\mathcal{D}}_y\right|{\mathcal{D}}_x,\mathit{\theta })p\left(\mathit{\theta }\right)}{{\int }_{\theta }p\left({\mathcal{D}}_y\right|{\mathcal{D}}_x,\mathit{\theta })p\left(\mathit{\theta }\right)d\mathit{\theta }}.\end{array}$$

(12)

Since neural networks have large number of parameters and non-convex probability distributions due to their complexity [54], it is difficult to compute the marginal likelihood ${\int }_{\theta }p\left({\mathcal{D}}_y\right|{\mathcal{D}}_x,\mathit{\theta })p\left(\mathit{\theta }\right)d\mathit{\theta }$. To handle this problem, it is common to avoid direct computation of the marginal likelihood and instead use approximation techniques such as Markov chain Monte Carlo (MCMC) and variational inference (VI). Since MCMC is also computationally expensive and appropriate sampling becomes challenging in high-dimensional spaces, the proposed method adopts Bayes by Backprop [55], which is one of the VI methods. Specifically, Bayes by Backprop approximates the posterior probability distribution $p\left(\mathit{\theta }\right|\mathcal{D})$ by minimizing the following equation:

$$\begin{array}{cc}\hfill {\mathit{\lambda }}^{\ast }& =\underset{\mathit{\lambda }}{arg\; min}\mathrm{KL}\left[q\left(\mathit{\theta }\right|\mathit{\lambda }\left)\right|\left|p\right(\mathit{\theta }\left|\mathcal{D}\right)\right]\hfill \\ \hfill & =\underset{\mathit{\lambda }}{arg\; min}\int q\left(\mathit{\theta }\right|\mathit{\lambda })log\frac{q\left(\mathit{\theta }\right|\mathit{\lambda })}{p\left(\mathit{\theta }\right)p\left(\mathcal{D}\right|\mathit{\theta })}d\mathit{\theta }\hfill \\ \hfill & \approx \underset{\mathit{\lambda }}{arg\; min}\sum _{k=1}^K[logq\left({\mathit{\theta }}_k\right|\mathit{\lambda })-logp\left({\mathit{\theta }}_k\right)-logp\left(\mathcal{D}\right|{\mathit{\theta }}_k)],\hfill \end{array}$$

(13)

where $\lambda$ is the variational parameter, and ${\mathit{\theta }}_k$ represents the $\mathit{k}$-th Monte Carlo sample taken from $q\left(\mathit{\theta }\right|\mathit{\lambda })$. To make the variational distribution $q\left(\mathit{\theta }\right|\mathit{\lambda })$ closer to the true posterior distribution $p\left(\mathit{\theta }\right|\mathcal{D})$, we minimize the Kullback–Leibler (KL) Divergence. The marginal likelihood is a constant value independent of $\lambda$ and does not affect the minimization; thus, it can be ignored. The final equation transformation is approximated by a Monte Carlo method, which eliminates the need to compute integrals and simplifies the computational process. Furthermore, the variational posterior parameters $\mathit{\lambda }=(\mu ,{\sigma }^2)$ are updated using the reparameterization trick in conjunction with gradient descent. The reparameterization trick is a critical component in Bayes by Backprop for facilitating efficient backpropagation through stochastic weights. It involves expressing the sampled weights $\mathit{\theta }$ as a deterministic function of the variational parameters $\mathit{\lambda }$ and independent noises $\mathit{ϵ}$, typically sampled from a standard normal distribution. Specifically, the reparameterization is formulated as $\mathit{\theta }=g(\mathit{ϵ},\mathit{\lambda })$, where $g(·)$ is a differentiable function. This transformation allows the gradient of the loss function with respect to the variational parameters $\mathit{\lambda }$ to be computed directly and enables standard gradient-based optimization methods to be applied. The reparameterization trick thus addresses the challenge of backpropagating gradients through random variables by transforming the randomness into an external and independent noise source. This technique significantly reduces variance in the gradient estimates and leads to more stable and efficient optimization in BNNs. Each term in this approximation cost is dependent on a specific weight obtained from the variational posterior, which ensures efficient and precise adjustments at each step of model learning. By employing the reparameterization trick, Bayes by Backprop optimizes the variational parameters in a way that the variational distribution $q\left(\mathit{\theta }\right|\mathit{\lambda })$ closely approximates the true posterior $p\left(\mathit{\theta }\right|\mathcal{D})$, thereby capturing the uncertainty in the model weights effectively.

In the proposed method, we involve multiple forward passes at each training step to obtain various prediction results. These results are then aggregated to form a comprehensive prediction distribution. The mean of this distribution represents the expected value of the predictions, while the variance indicates the prediction uncertainty. Furthermore, this predictive uncertainty can be decomposed into two distinct components [56]: aleatoric uncertainty ${\mathit{u}}_t^{alt}$ and the epistemic uncertainty ${\mathit{u}}_t^{ept}$ as follows:

$$\begin{array}{cc}\hfill {\mathit{u}}_t^{alt}& =\frac{1}{M}\sum _{m=1}^M[\mathrm{diag}\left({\tilde{\mathit{y}}}_{t,m}\right)-{\tilde{\mathit{y}}}_{t,m}{\tilde{\mathit{y}}}_{t,m}^{\top }],\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathit{u}}_t^{ept}& =\frac{1}{M}\sum _{m=1}^M({\tilde{\mathit{y}}}_{t,m}-{\overline{\mathit{y}}}_t){({\tilde{\mathit{y}}}_{t,m}-{\overline{\mathit{y}}}_t)}^{\top },\hfill \end{array}$$

(15)

where M is the number of forward passes. ${\tilde{\mathit{y}}}_{t,m}$ is the output value at time step t of the m-th layer of BNNs, and ${\overline{\mathit{y}}}_t$ is the mean value of the output from each layer. Aleatoric uncertainty reflects the inherent noise and randomness present in the data, which cannot be reduced even with more data or a more complex model. In contrast, epistemic uncertainty stems from the model itself, such as the uncertainty in the model parameters, and can be reduced as the model learns from more data. By distinguishing between these two types of uncertainty, Bayes by Backprop provides valuable insights into the confidence and reliability of the model’s predictions. This differentiation is particularly useful in applications where understanding the nature of uncertainty is crucial for decision-making processes.

2.3.2. Loss Function

In the proposed method, the variational distribution $q\left(\mathit{\theta }\right|\mathit{\lambda })$ is optimized to be close to the true posterior distribution $p\left(\mathit{\theta }\right|\mathcal{D})$, as shown in Equation (13). Hence, we first employ the loss functions ${\mathcal{L}}_{\mathrm{PP}}$ given by the following equation:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{PP}}=\sum _{k=1}^K[logq\left({\mathit{\theta }}_k\right|\mathit{\lambda })-logp\left({\mathit{\theta }}_k\right)].\end{array}$$

(16)

This loss function is the aggregation of the first and second terms in Equation (13), that is, the difference between the variational posterior distribution and the prior distribution. Furthermore, to minimize the third term in Equation (13), which is the negative log-likelihood, it is equal to minimizing the mean-square error. Therefore, we adopt the weighted binary cross-entropy loss ${\mathcal{L}}_{\mathrm{BCE}}$ as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{BCE}=\sum _{t=1}^T-[e^{-\max (0,\frac{T_{\mathrm{s}}-t}{\mathrm{FPS}})}{y}^{\left(p\right)}log{\widehat{y}}_t^{\left(p\right)}+(1-y^{\left(p\right)})log(1-{\widehat{y}}_t^{\left(p\right)})],\end{array}$$

(17)

where $T_{\mathrm{s}}$ represents the time of event occurrence and $\mathrm{FPS}$ indicates the frame rate of input video v. $\mathit{y}={(y^{\left(n\right)},y^{\left(p\right)})}^{\top }$ and ${\widehat{\mathit{y}}}_t={({\widehat{y}}_t^{\left(n\right)},{\widehat{y}}_t^{\left(p\right)})}^{\top }$ are the target labels at the video level and the prediction probability at the time step level of training data, respectively. Here, in a positive video containing the event occurrence, it is considered that the probability of the event occurrence may be low at the beginning of a video. Therefore, by introducing the weight $e^{-\max (0,\frac{T_{\mathrm{s}}-t}{\mathrm{FPS}})}$, we gradually decrease the importance of the loss in the pre-event time instances as time moves away from the event moment. By using the above loss functions, we can achieve the acquisition of optimal variational approximations.

In addition, we introduce the loss function ${\mathcal{L}}_{\mathrm{RANK}}$, which considers the prediction uncertainty as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{RANK}=\max (0,\mathrm{trace}({\mathit{u}}_t^{ept}-{\mathit{u}}_{t-1}^{ept})).\end{array}$$

(18)

The epistemic uncertainty ${\mathit{u}}_t^{ept}$ represents a measure of uncertainty about the model’s predictions. Thus, by introducing this loss, it is expected that the epistemic uncertainty does not increase with time, that is, the uncertainty is reduced or stabilized as the model learns new information. As for the aleatoric uncertainty ${\mathit{u}}_t^{alt}$, since it is a measure of the inherent variability of the latent feature’s output from the GCRNN, we do not include it in the loss function in the proposed method.

Eventually, the proposed method optimizes the prediction model by training with the combination loss function $\mathcal{L}$, which is expressed as:

$$\begin{array}{c}\hfill \mathcal{L}={\mathcal{L}}_{\mathrm{BCE}}+w_1·{\mathcal{L}}_{\mathrm{PP}}+w_2·{\mathcal{L}}_{\mathrm{RANK}},\end{array}$$

(19)

where $w_1$ and $w_2$ are employed to balance respective loss function values. In this way, the proposed method achieves accurate shot prediction while considering prediction uncertainty.

3. Experiments

In this section, we validate the effectiveness of the proposed method for shot prediction. Subsequently, Section 3.1 explains the experimental settings and Section 3.2 discusses the experimental results.

3.1. Settings

In this experiment, we used a dataset consisting of video clips derived from scouting videos of 31 matches in the J1 League for the 2019 and 2020 seasons. Video clips that included a shot event were defined as positive samples, and video clips that included attack scenes without a shot event were defined as negative samples. As a result, there were 400 video clips with 200 positive and 200 negative samples. Each clip consisted of 6 s of a video acquired at 10 fps, with the frame size being $1280\times 720$ pixels. We used 80% of the video clips as a training dataset and the remaining 20% as a test dataset, and 5-fold cross-validation was conducted.

The dimension D of each image feature ${\mathit{x}}_t^{\mathrm{player}}$, ${\mathit{x}}_t^{\mathrm{frame}}$, and ${\mathit{x}}_t^{\mathrm{audio}}$ applied to fully connected layers after being extracted from ResNet-101 was 256 dimensions, that is, the graph node feature ${\mathit{X}}_t$ that combined them was 768 dimensions. The maximum number of detected players was set at 20, corresponding to the number of field players excluding the goalkeeper. Furthermore, the GCN had two layers, and the first layer transformed graph node ${\mathit{X}}_t$ into 256-dimensional features. In the second layer, by inputting the hidden state ${\mathit{H}}_{t-1}$ of the previous time step in addition to the output of the first GCN layer, the latent feature ${\mathit{F}}_t$ was obtained. Here, the dimensions of ${\mathit{H}}_{t-1}$ and ${\mathit{F}}_t$ were both 256. As for the GCRN, the hidden state ${\mathit{H}}_t$ was updated using a graph whose nodes were 1024-dimensional features concatenated from graph node feature ${\mathit{X}}_t$ and latent feature ${\mathit{F}}_t$ in the same manner as the hidden state ${\mathit{H}}_{t-1}$ at the previous time step. Lastly, by flattening and then inputting the latent feature ${\mathit{Z}}_t$, the prediction results were calculated with two-layer BNNs that output 64 and 2 dimensions, respectively, that is, the prediction probabilities of positive and negative event occurrences. In the training phase, two forward passes were conducted ($M=2$), and weights were sampled twice ($K=2$) in each pass by Monte Carlo sampling to learn the probability distribution of weights. Thereby, the proposed model captured the uncertainty and improved the reliability of the prediction results. In the inference phase, the proposed model made eleven forward passes ($M=11$) (10 stochastic forward passes and a deterministic forward pass) and ten samplings of weights ($K=10$). In this way, by utilizing the probability distribution of the weights and the prediction uncertainty acquired in the training phase, our model provided more robust predictions for unknown input data. This setting emphasized computational efficiency in training and more accurate uncertainty evaluation and prediction accuracy in inference.

The event occurrence time $T_{\mathrm{s}}$ of positive samples varied with each video clip and was manually determined. As shown in Figure 4, the distribution of $T_{\mathrm{s}}$ is concentrated in the latter half of the video. The video clips included in the training dataset were given target labels $\mathit{y}={(y^{\left(n\right)},y^{\left(p\right)})}^{\top }$ for the entire video clip, which were defined as $\mathit{y}={(0,1)}^{\top }$ for positive and $\mathit{y}={(1,0)}^{\top }$ for negative. The $w_a$ in Equation (3) was set experimentally to 1050, which is the length of the touchline of the field. Moreover, $w_1$ and $w_2$ in Equation (19) were set to 0.001 and 10, respectively, in order to balance the edge weights and loss values. In the training phase, the learning rate was initialized to $5\times {10}^{-4}$, and we employed Adam [57] and ReduceLROnPlateau as the optimizer and scheduler, respectively. The batch size was set to 16, and it was trained in 60 epochs.

To validate the effectiveness of the proposed model (PM) for shot prediction from soccer videos, we first conducted ablation studies that excluded the novelty from the PM. Specifically, we compared our model with the models without audio feature ${\mathit{x}}_t^{\mathrm{audio}}$ from graph nodes, xG from graph edge weights, and both of them, which were defined as Ablations1-3, respectively. We also compared with Ablation4, which employed a simple neural network instead of a BNN. Furthermore, we adopted the following models for the comparison.

DSA [58]:

This is a method for traffic accident prediction from dashcam videos. This model learns temporal relationships through RNN and the importance of detected objects in the video through Dynamic Spatial Attention (DSA).

DSTA [59]:

This is a video-based prediction model for traffic accidents, which uses GRU or LSTM to learn temporal relationships and Dynamic Spatial–Temporal Attention (DSTA) to learn the spatial and temporal importance of detected objects in the dashcam video.

ViViT [60]:

This is a video recognition model known as a Video Vision Transformer (ViViT), which learns spatial and temporal dynamics by tokenizing videos into a series of image frames or patches.

To the best of our knowledge, our previous studies [38,39,40,41] are the only ones that use visual information to predict the shot event occurrence in soccer matches. Therefore, we employed DSA and DSTA as comparative methods although they are video-based traffic accident prediction models that consider spatial–temporal relationships between objects in the video. In addition, since ViViT has the ability to classify the existence of events by recognizing various factors from the entire video image, it is considered to have the same ability as an accurate prediction of event occurrences. Hence, we adopted ViViT as a valid standard of comparison for event prediction methods.

In order to validate the prediction accuracy, we used Average Precision (AP) and F1-score (F1), which were defined as follows:

$$\begin{array}{cc}\hfill \mathrm{AP}& =\sum _{th}({\mathrm{Recall}}_{th+1}-{\mathrm{Recall}}_{th}){\mathrm{Precision}}_{th},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \mathrm{F}{1}_{th}& =\frac{2\times {\mathrm{Recall}}_{th}\times {\mathrm{Precision}}_{th}}{{\mathrm{Precision}}_{th}+{\mathrm{Recall}}_{th}},\hfill \end{array}$$

(21)

where

$$\begin{array}{cc}\hfill {\mathrm{Recall}}_{th}& =\frac{{\mathrm{TP}}_{th}}{{\mathrm{TP}}_{th}+{\mathrm{FN}}_{th}},\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill {\mathrm{Precision}}_{th}& =\frac{{\mathrm{TP}}_{th}}{{\mathrm{TP}}_{th}+{\mathrm{FP}}_{th}}.\hfill \end{array}$$

(23)

The prediction of event occurrence by the model is dependent on the threshold $th$ being exceeded. At the threshold $th$, ${\mathrm{TP}}_{th}$ denotes the count of video clips correctly predicted as events, ${\mathrm{FP}}_{th}$ indicates the number of video clips incorrectly predicted as events, and ${\mathrm{FN}}_{th}$ refers to the number of video clips that were events but not predicted as events. Additionally, to measure the prediction earliness for positive samples, we evaluated event prediction performance with mean Time to Event (mTTE) calculated by the following equation:

$$\begin{array}{cc}\hfill \mathrm{mTTE}& =\mathbb{E}\left[{\mathrm{TTE}}_{th}\right],\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill {\mathrm{TTE}}_{th}& =\max \{T_{\mathrm{s}}-t|\beta \left(y_t\right)\ge th,1\le t\le T_{\mathrm{s}}\},\hfill \end{array}$$

(25)

where $\beta (·)$ represents the sigmoid function. $\mathrm{TTE}$ is a common metric in prediction tasks and is used in various studies [39,58,59,61]. In this experiment, we verified the effectiveness of the proposed model in the above experimental settings.

3.2. Results and Discussion

3.2.1. Quantitative Results

Table 1. Comparison of AP and F1 across varying video lengths for PM and other models. The time step t indicates the length of video clips from the beginning. The best performance for each length is shown in bold, and the second-best performance is underlined.

	AP						F1
	t = 10	t = 20	t = 30	t = 40	t = 50	t = 60	t = 10	t = 20	t = 30	t = 40	t = 50	t = 60
Ablation1	0.881	0.921	0.939	0.944	0.948	0.947	0.729	0.782	0.823	0.848	0.874	0.897
Ablation2	0.912	0.917	0.934	0.942	0.958	0.964	0.672	0.790	0.829	0.849	0.860	0.854
Ablation3	0.905	0.923	0.933	0.940	0.942	0.942	0.671	0.750	0.795	0.823	0.842	0.855
Ablation4	0.915	0.929	0.937	0.939	0.942	0.940	0.601	0.753	0.800	0.824	0.831	0.832
DSA [58]	0.882	0.880	0.888	0.929	0.936	0.932	0.742	0.777	0.802	0.818	0.838	0.838
DSTA [59]	0.912	0.923	0.929	0.928	0.939	0.946	0.674	0.763	0.815	0.829	0.844	0.859
ViViT [60]	0.736	0.744	0.766	0.770	0.760	0.755	0.679	0.682	0.676	0.679	0.675	0.682
PM	0.913	0.923	0.938	0.946	0.959	0.966	0.757	0.806	0.825	0.860	0.876	0.891

and

Table 2. AP, Recall, Precision, F1, and mTTE for all methods except ViViT at time step $T_{\mathrm{s}}$. Note that Recall and Precision adopt the values at the threshold $th$ when F1 achieves the highest accuracy. Best and second-best performances are highlighted in bold and underlined, respectively.

	AP	Recall	Precision	F1	mTTE
Ablation1	0.934	0.807	0.882	0.839	3.46
Ablation2	0.941	0.781	0.891	0.824	3.63
Ablation3	0.927	0.797	0.849	0.801	3.46
Ablation4	0.926	0.766	0.884	0.816	3.61
DSA [58]	0.920	0.786	0.859	0.820	3.71
DSTA [59]	0.925	0.820	0.856	0.836	3.47
PM	0.948	0.828	0.896	0.858	3.82

show the quantitative results for each method. Concretely, in Table 1, we show the prediction accuracy at different lengths of video clips, and the performance of PM is better than other models at almost all lengths of video clips. Note that in all video clips, the time $T_{\mathrm{s}}$ of the shot occurrence is situated in the second half of the video clip. Compared with PM and Ablations1-3, we confirm that Ablation3 has the lowest and PM the highest performance when the length of video clips is long, that is, video clips including around the shot occurrence time $T_{\mathrm{s}}$. Since spectators cheered loudly at important moments, it is considered that the audio feature is effective for shot prediction. xG can reflect the importance of the player’s position in more detail, and thus, it is assumed that the introduction of xG into the graph edge weights is also useful by comparing PM with Ablation2. In comparison between PM and Ablation4, it is observed that although Ablation4 has a slightly higher AP for only the first half of the video, PM shows higher accuracy as the video length increases. From this result, we confirm that Ablation4 does not correctly capture the match situations, which indicates the effectiveness of using the BNN instead of a general neural network since the shot occurred in the second half of the video clip. Furthermore, the comparison of PM with DSA and DSTA shows that it is more effective in shot prediction in soccer matches than video-based shot prediction models, which achieved high performance in a different task. In the comparison between PM and ViViT, we confirm that PM significantly outperforms ViViT at all lengths of video clips. The above results show that PM achieves higher prediction accuracy than that of the state-of-the-art video prediction model. Here, DSA and DSTA achieve high prediction accuracy, although not as high as PM, in contrast to ViViT, which has a considerably less accurate performance than that of the other comparison methods. The reason for the low performance of ViViT could be attributed to the crucial difference that ViViT does not perform object detection. With the self-attention of the vision transformer, ViViT learns the interaction of each position in a video frame with the other positions. However, it is considered to be difficult to determine the focus points in a scouting video, which is captured from a bird’s-eye view of the field, without prior information. On the other hand, DSA, DSTA, and PM utilize object detection results to learn the extent to which each detected object is attributable to the prediction results. This difference may have affected the prediction accuracy. Moreover, PM utilizes a graph that introduces more detail on the importance of each player, which is assumed to be one of the factors behind its higher performance than that of DSA and DSTA.

Since the shot occurrence time $T_{\mathrm{s}}$ is different for each video clip, we evaluate the prediction performance at time step $T_{\mathrm{s}}$. In Table 2, we present the prediction accuracy at time step $T_{\mathrm{s}}$ and the prediction earliness for the positive sample. The values of Recall and Precision are determined at threshold $th$ in which F1 achieved the best performance. Note that in this experiment, ViViT, which classifies the existence of events from the entire video, is excluded from the comparison method since it is not possible to make predictions at each time step. As shown in Table 2, PM achieves the highest performance in all evaluation metrics than those of the other models. The comparison between PM and Ablations1–4 indicates the effectiveness of introducing each novelty in terms of prediction accuracy. DSA and DSTA also performed as well as the Ablation studies, although not as well as PM. Furthermore, we confirm that PM correctly predicts positive samples earlier than other models. Since PM has a high mTTE while ensuring high accuracy, it may be able to support players and coaching staff in making timely decisions based on the results. In summary, it is verified that PM is a quantitatively outstanding model for shot prediction tasks in soccer.

3.2.2. Qualitative Results

To qualitatively evaluate the prediction performance, visualized examples of the shot prediction results obtained by PM are shown in Figure 5. Each example result represents positive and negative outcomes of accurate predictions (true-positive and true-negative, respectively), followed by negative outcomes of incorrect predictions (false-positive). A threshold value $th=0.5$ was set to determine whether each prediction result was correct. In the figure of each example, several characteristic frames in the video clip are displayed at the top. For the bottom part of the figure, the prediction probability at each time step (black lines) and associated aleatoric and epistemic uncertainties (orange and cyan regions, respectively) are depicted. The larger region of each uncertainty at each time step indicates greater uncertainty regarding the prediction outcome. The true positive example shown in Figure 5a is a scene that includes the sequences of pass to near goal and shot, where the prediction probability increases rapidly after the pass into the penalty area and reaches the target threshold value 1.3 s before the shot moment. In this example, as the player dribbling approaches the penalty area gradually, the model provides a rising prediction probability while maintaining uncertainty. The fact that the model continues to offer significant uncertainty right up to the moment before the shot suggests that it recognizes the situation as ambiguous regarding whether the pass will succeed or not. The following example of a true negative shown in Figure 5b is a scene where a cross is made from the side but no one touches the ball. The prediction result does not exceed the threshold throughout the video, yet it suggests significant uncertainty. This indicates that while there are few attacking players near the goal, which leads to a low probability of a shot occurring, there is a potentially high possibility of it happening. Finally, we observe a false positive scenario where an attack from the side culminates in a centering attempt within the penalty area shown in Figure 5c. The predictive outcomes of this scene frequently reach the threshold in the video, accompanied by considerable uncertainty. Although this particular scene does not result in an actual shot, thus categorizing it as a negative instance in our study, it can arguably be considered a scene with a highly probable chance of leading to a shot or creating a shot opportunity. In a real soccer match, the shot opportunity often exists even in situations where the player does not choose the shot. This observation suggests that future research may need to reconsider the definitions within the dataset.

3.2.3. Limitations

There are some limitations to our study. First, a data-specific limitation is that it is challenging to identify the ball carrier in the scouting videos used in this study. This may lead to scenarios where a shot event occurs without the predicted probability reaching the threshold value. Next, there is a limitation on the number of datasets. In this experiment, we adopted ViViT as one of the comparison methods; however, its performance was lower compared to other methods. As one of the factors, it is considered that this may be the insufficient scale of the dataset. Since ViViT is a transformer-based model with a weak inductive bias, it has been reported that generalization performance is poor for small datasets and requires large datasets to achieve a high-performance model [62]. In this experiment, the total number of datasets was 400, which may not have been sufficient to achieve the same level of accuracy as our method. Furthermore, we applied the xG provided by MLS to the match data of the J1 League, which is a different league. Although xG for individual players and teams is provided in the J1 League and European leagues, the xG for each position on the field is not publicly available. In this study, we adopted the xG provided by MLS instead, considering that xG for each field position in soccer does not vary significantly across different leagues.

4. Conclusions

In this paper, we propose a model for multimodal shot prediction by utilizing spatial–temporal interactions between players in soccer videos. The proposed model constructs a graph that considers the relationship and importance of players with various audio-visual features acquired from the video as nodes. Then, we extract latent features from the graph based on GCRNN considering both spatial and temporal interactions. Finally, we predict the shot occurrence by using BNNs, which is a probabilistic deep-learning method. The BNN provides uncertainty about the prediction results and thus users can analyze and evaluate the results with high confidence. In experiments using video clips extracted from actual professional soccer league matches, our method achieved the highest accuracy in all evaluation metrics at and around the shot moment. Specifically, the results of an AP of 0.948 and an F1 score of 0.858 indicate that it is a high-performance prediction model, demonstrating its superiority. From the experiment, we confirmed the effectiveness of the novelty of our study, that is, the audio feature, and the xG for considering the importance based on players’ positions.

This approach is expected to improve predictive accuracy in diverse scenarios and environments. We also believe our shot prediction model can be easily adapted to other sports. Given that soccer is a sport with a large number of players and high degrees of freedom for each player, it is considered arguably more challenging than other sequential sports such as basketball and rugby. Therefore, as our model has successfully captured the more complex inter-player relationships in soccer, it is highly likely to provide accurate event predictions in similar competitive sports. Consequently, our model demonstrates significant applicability and potential benefits for a range of sports, underscoring its utility in diverse sporting contexts.