Automated Tennis Player and Ball Tracking with Court Keypoints Detection (Hawk Eye System) Desu Venkata Manikanta, Syed Fawaz Ali, Sunny Rathore Abstract This study presents a complete pipeline for automated tennis match analysis. Our framework integrates multiple deep learning models to detect and track players and the tennis ball in real time, while also identifying court keypoints for spatial reference. Using YOLOv8 for player detection, a custom-trained YOLOv5 model for ball tracking, and a ResNet50-based architecture for court keypoint detection, our system provides detailed analytics including player movement patterns, ball speed, shot accuracy, and player reaction times . The experimental results demonstrate robust performance in varying court conditions and match scenarios. The model outputs an annotated video along with detailed performance metrics, enabling coaches, broadcasters, and players to gain actionable insights into the dynamics of the game. 1 Introduction and Problem Statement Tennis is a sport characterized by rapid movements, split-second decisions, and complex strategies. The ability to analyze these elements quantitatively has be- come increasingly important for players, coaches, and broadcasters. Traditional manual analysis methods are time-consuming, subjective, and limited in their precision. This creates a significant demand for automated systems that can provide objective, comprehensive, and immediate analysis of tennis matches. In professional tennis, systems like Hawk-Eye [1] have revolutionized officiating and broadcast ex-periences by providing accurate ball tracking. However, these systems typically require multiple high-speed cameras placed at precise locations around the court. There is a clear need for more accessible solutions that can work with standard video equipment while still providing valuable analytical insights. Tennis coaches and players increasingly rely on quantitative metrics to identify strengths, weak-nesses, and areas for improvement. Broadcasters seek enhanced visualizations to enrich viewer experi-ence. Tournament organizers require effi- cient tools for match statistics and automated line calling. All these stakeholders would benefit from an integrated system that can analyze matches comprehen- sively from standard video input. Our study addresses these needs by developing an end-to-end framework for tennis match analysis that integrates player tracking, ball detection, court map- ping, and performance metrics calculation. The main objectives of this work is:

Tennis is a fast-paced sport where split-second decisions can determine match outcomes, yet traditional analysis methods are often slow and subjective. This research introduces an automated system that delivers immediate, objective data on player movements and ball dynamics, making advanced analytics accessible with standard video equipment. For coaches, players, and fans, this means deeper insights into performance without the high costs of specialized setups like Hawk-Eye.

The key finding is that this system accurately tracks tennis players and the ball in real time, while also mapping court positions to calculate metrics such as speed, reaction times, and movement patterns. By integrating multiple machine learning models, it identifies active players, filters out non-players like referees, and provides visualizations like heatmaps and mini-court animations. For instance, it can show where a player spends most time on the court or how quickly they react to an opponent's shot, offering a comprehensive view of game dynamics.

Methodology involves a four-component pipeline: player detection, ball tracking, court keypoint identification, and performance metrics calculation. The system uses YOLOv8 for player detection, initialized with pre-trained weights from the COCO dataset to recognize persons. A custom-trained YOLOv5 model handles ball detection, optimized for small, fast-moving objects with techniques like data augmentation to improve generalization. For court mapping, a ResNet50-based architecture detects 14 key landmarks, such as baseline corners and net posts, achieving an average error of 3.8 pixels. Algorithms filter players based on court boundaries and interpolate missing ball detections using linear and Kalman smoothing methods to ensure continuous tracking.

Results analysis shows the system maintains strong performance under various conditions. In experiments, player identification was reliable during normal rallies, though it occasionally struggled with close interactions like handshakes. Ball detection achieved 89% mean average precision on validation data, with interpolation handling occlusions and blur. Visual outputs include annotated videos with metrics: for example, shot speeds are calculated by converting pixel distances to real-world measurements using known court dimensions, and reaction times are defined as the interval between an opponent's shot and the player's first significant movement. These data reveal patterns, such as players taking less time to react at the start of a rally compared to later stages.

Contextually, this system matters because it democratizes high-level sports analytics. Coaches can use it to identify strengths and weaknesses without expensive hardware, broadcasters can enrich viewer experiences with real-time stats, and tournament organizers gain efficient tools for officiating and statistics. It builds on prior work in computer vision but stands out by integrating tracking, mapping, and analytics into a single framework, potentially applicable to other sports with similar dynamics.

Limitations include reduced accuracy with unusual camera angles, such as overhead views, and challenges in handling prolonged occlusions, like when the ball is hidden behind a player. Additionally, the system does not classify shot types, such as forehands or backhands, which could enhance analytical depth. Future work may focus on adding shot classification, integrating multiple cameras for 3D tracking, and incorporating pose estimation for biomechanical analysis.