Inter-observer Reliability of a Real-time Observation Tool in Handball

Background: The analysis of the competition in real time is currently one of the most important aspects to develop the sport. The purpose of the analysis should be creating valid and reliable knowledge for coaches to make the best decisions in a situation of competition. Objectives: This study was to determine the inter-observer reliability of the real-time observation tool for handball. Methods: Two groups of two observers each one were required to analyze the men's handball final of the London 2012 Olympic Games (average age: 23.72 ± 2.16 years; experience as handball players: 14.69 ± 1.92 years; experience as coaches: 4.64 ± 4.04 years). The process of training of the observers lasted 22 days, accounting twelve hours of work distributed in 12 training sessions. Results: The reliability showed a very good agreement between the independent observers (Kappa values were 0.96 and 0.90) in the registered events of both teams, and a very good agreement (Kappa values were 0.85 and 0.94) of the registered actions of the goalkeepers. The high coefficient of intra-class correlation with a value of 0.98 and the low value of the standard error with a 0.11 of the actions of the players of both teams showed a high level of inter-observer reliability. Conclusions: These results showed that the tool of observation in handball is reliable for registering the events of a real time match by well-trained observers. With the help of the HandballTAS and using technology, large volumes of real-time data were collected in a simple and easily usable.

Other studies in handball have validated: i) a system for observing the players´ decision making (Martín et al., 2013); ii) a tool for the study of the players and teams´ dynamics during single games (Prudente, Garganta, & Anguera, 2004); iii) a system about the observation of coaches' behaviour (Tzioumakis et al., 2009); or iv) the explosive strength of the shot from different zones of the field (Vuleta et al., 2010).
Most of the methods of performance analysis were not based on automated techniques of data entry. Human errors during the compilation of data can limit the reliability of the methods used (O´Donoghue, 2007). Validity and reliability in performance analysis is essential to meet its purposes with efficiency. However, this scientific base has not been fully established and controlled for in many of the studies (Tenga et al., 2009). Reliability refers to the reproducibility of values of a test, assay or other measurement in repeated trial on the same individuals (Hopkins, 2000a). It is defined as a characteristic of the measurement or the experimental procedure in which the same results take place in two or more separate occasions (Kent, 1994).
The tests of reliability in the analysis of sports performance are used to assess the agreement between the observers and to guarantee the objectivity of the process of data gathering (Berry, Johnston, & Mielke, 2008;Choi, O'Donoghue, & Hughes, 2007;Cooper et al., 2007;Hughes, Cooper, & Nevill, 2004;Lames & McGarry, 2007;Nevill et al., 2002;O'Donoghue, 2007;Robinson & O'Donoghue, 2007). This is essential for the development of studies in sport contexts, bringing the validity and objectivity that are necessary in the scientific field.
Inter-observer consistency allows establishing the objectivity of the system and proving that it can be used to collect data from observations independently of the individual perception of the encoder. There are two crucial prerequisites to obtain a sufficient level of reliability of the data collected in the research. On the one hand, measuring reliability and validity among observers with a suitable instrument; and on the other hand, training the observers in a systematic and coherent way to achieve good results in these measurements (O´Donoghue, 2010). Therefore, the aim of this study is to determine the reliability inter-observer of the real-time observation tool in handball HandballTAS (Handball Tactic Analysis System).

Sample
The match chosen as the subject of the current inter-observer reliability study was the men's handball final of the London 2012 Olympic Games that took place on the 12 th of August, 2012 between Sweden and France. A total of 22 players from both teams were involved in the match. The criteria proposed by Heinemann (2003), by which the object of the research study is accessible using the recording of matches for the data collection, were taken into account.
The observation was carried out by four observers (average age: 23.72 ± 2.16 years). They had experience as handball players for 14.69 ± 1.92 years and experience as coaches in lower categories for 4.64 ± 4.04 years. The process of training of the observers lasted 22 days, from 14/11/2013, up to 5/12/2013 accounting twelve hours of work distributed in 12 training sessions.
In every training session the observers were subjected to the same conditions: (i) the observer is isolated in a room to keep the intra-sessional connection; and (ii) at the same time and in the same place in stable conditions and without the presence of any person with the intention to avoid interferences, with the exception of the researcher.
Four observers were required to analyze the game independently: two observer registered the team of Sweden (observer 1 and observer 2), and two observers registered the team of France (observer 3 and observer 4).

Software and taking of data
The software has been designed to register the individual actions of the players during a handball match in real time. The process to evaluate the inter-observer reliability was similar to the study of Liu et al. (2013) used to evaluate the inter-observers reliability of a live football match using Opta Sportdata. Previous research has studied the effectiveness of handball teams through record of offensive and defensive players actions (Balint & Curiţianu, 2012;Bilge, 2012;Meletakos, Vagenas, & Bayios, 2011;Rogulj, Srhoj, & Srhoj, 2004;Srhoj et al., 2001;Yamada et al., 2014).
To analyze the reliability of the software, the individual actions of the players were divided in two groups: (1) actions related to the attack: received free-throw, assist, turnover, throw-in, committed steps, committed illegal dribble, committed offensive foul, committed entering the goal area goal, throw block, caused yellow card, caused exclusion, caused disqualification, shot with opposition, and shot without opposition; (2) actions related with the defense: committed free-throw, overcome by direct opponent, steal, committed throw-in, caused steps, caused illegal dribble, caused offensive foul, caused entering the goal area, blocked shot, committed yellow card, committed exclusion, and committed disqualification. El registro de estas acciones durante un partido tiene como ventaja el aumento del conocimiento del juego y la disposición de esa información inmediatamente para que el entrenador pueda aumentar la eficacia de sus intervenciones. The record of these actions during a match has the advantage of increasing the knowledge of the game and the disposition of such information immediately in order that the coach could increase the effectiveness of their interventions.
The software is presented on a touchscreen mobile device (tablet) in order to register all events in real time as quickly and easily as possible (see Figure 1). Before beginning with the record of events, the general information of the match is annotated: name of observer, date, hour, place, championship, phase, match time and period. To take the record the individual offensive and defensive actions of systematic form every action is associated with a number of a player. If the player throws the ball, the degree of opposition (with opposition or without opposition), the result (goal, save, out or post), the location and the area of the field from where the throw occurs are noted down. All actions are associated with a numerical situation (equality, superiority, inferiority or forewarning signal passive play) at a particular time of a match.

Actions registered with the tool
The following actions were defined and described in order to clarifying the data gathering: Yellow card: Awarded by the referee to a player as a result of a fault or unsportsmanlike behavior.
Assist: Technical action of an attacking player who passes the ball to a teammate without opposition in a way that leads to score a goal.
Attack: The team has the possession of the ball in its own field or in the opponet´s field, and the attack phase begins.
Forewarning signal for passive play: The referee indicates the warning of passive play when a team does not have the intention of attacking or to throw to the goal.
Blocked shot: Technical action of a defending player who intercepts the trajectory of the ball in a throwing on goal.
Defense: The team loses the ball possession, beginning their defense phase.
Disqualification: Red card awarded by the referee to a player due to an unsportsmanlike behavior. Exclusion: Exclusion of a player for 2 minutes indicated by the referee due to repeated fouls, unsporting conduct, an incorrect change, or as a result of a disqualification.
Offensive foul: Infraction made by a player as a result of a foul on a defender indicated by the referee. The team loses the possession of the ball.
Throw-in: A player sends the ball out of the field of play.
Out: The throw on goal finishes out of the goal boundaries.
Goal: Throw on goal that crosses the goal line.
Free-throw: A defender commits an infraction on an attacker, and then the referees stop the game indicating a freethrow.
Equality: Game situation in which the same number of players on both teams are present on the field of play.
Inferiority: Game situation in which a smaller number of players of the team observed are present on the field of play.
Interruption: Game situation in which the attack or defense of the team observed is stopped as a result of a free-throw, out of play, a time-out or any circumstance in which the game is stopped.
Entering the goal area: Action in which a player invades the goal area.
Passive play: The referee indicates passive game when the team in possession of the ball does not change its way of attacking or does not execute a throwing after the referee has indicated the warning of passive play.
Throw block: Throwing directed to goal but neutralized following the rules by a defender before the goalkeeper can intercept it.
Shot with opposition: Action of throwing to goal following the rules with at least a defender in the shot line.
Shot without opposition: Action of throwing to goal following the rules without any defenders in the shot line.
Save: Throwing that the goalkeeper neutralizes so that the ball does not cross the goal line.
Steps: Offensive action in which the team loses the possession of the ball due to an infraction of the player with the ball. The referee indicates the steps hand-signal. Overcome by direct opponent: Defensive action in which the defensive player is overcome by an attacker with ball possession using a displacement, feint or fixation. Surpassing an adversary in the one to one action is considered to be overcome by the direct opponent.
Superiority: Game situation in which there is a higher number of players of the team observed on the field of play. 7-meter throw: Action of a defender against the rules that destroys a clear goal chance of an attacker. The referee then indicates the corresponding infraction.

Statistical analysis
The events of both teams and the actions of the goalkeepers were compared between the two groups of observers using the Cohen's Kappa (k), which determines the proportion of cases in which there is agreement among observers once excluded the proportion of cases in which the agreement between them is a consequence of chance (Robinson & O´Donoghue 2007). The Kappa values can range from -1.0 to 1.0. The agreement in the interpretation of the Kappa value was valued as follows: <0 less than the possibility of agreement; 0.01-0.20 poor agreement; 0.21-0.40 fair agreement; 0.41-0.60 moderate agreement; 0.61-0.80 good agreement; 0.81-0.99 very good agreement (Altman, 1991;O´Donoghue, 2010;Viera & Garret, 2005).
Absolute reliability values (mean, change in the mean, standardized typical error and the intra-class correlation coefficient) of different individual actions of the players were calculated using the spreadsheet developed by Hopkins (2000b). Each team was registered by two independent observers. Therefore, there were two groups of values of absolute reliability, on the one hand Swedish players and on the other hand, French players. The results presented below were the average of the two groups. The value of the standardized typical error should be doubled and their levels of disagreement are as follows: <0.20 trivial; 0.21-0.60 small; 0.61-1.20 moderate; 1.21-2.00 large; 2.01-4.00 very large; >4.00 extremely large (Hopkins, 2000a;Smith & Hopkins, 2011). Table 1 shows 864 events agreed by the two groups of independent observers, 437 for Sweden and 427 for France. The Kappa values of the two teams' events were 0.90 and 0.96 respectively, which showed a very good agreement between independent observers.  Standardized typical errors of the different individual actions of the players from both teams were recorded by independent observers with a value of 0.11 (Table 3). The intra-class correlation index, with a value of 0.98, shows high levels of reliability.  Table 4 showed the absolute reliability of performance indicators of the players who took part in the observed match. Standardized typical errors are located in a range from 0 to 0.55, and the intra-class correlation coefficients varied from 0.77 to 1.00, showing a good level of reliability.

Discussion
The high Kappa values, intra-class correlation coefficients, and the low standardized typical errors show a high level of inter-observer reliability using the HandballTAS. The degree of reliability and validity among independent observers show that the instrument is suitable for the record of the actions in real time. Likewise, observers have undergone systematic and consistent training to achieve good results during the observational measurements (O´Donoghue, 2010).
Another technological tool used in handball is the one presented by Martín et al. (2011), and Martín et al. (2013) developing a program to observe decision making of the player based on the instrument validated by Oslin et al. (1998). Those authors developed and validated the Game Performance Assessment Instrument (GPAI). Validity and reliability are examined through three separate studies using middle school physical education specialists and their sixth-grade classes. The stability-reliability coefficient showed, as this study does, a high level of reliability. The soccer, basketball, and volleyball correlation coefficients ranged from 0.84 to 0.97, from 0.84 to 0.99, and from 0.85 to 0.97, respectively. Lozano & Camerino (2012), designed an instrument of observation with syntactic rules and codes (Fernández et al., 2009;Jönsson et al., 2006) to know the variables involved in offensive efficiency in handball. The degree of interobserver agreements using the Cohen's Kappa coefficient was of 0.95, a similar value to the present study.
It is necessary to know the actions that generate a greater disagreement by the independent observers (Liu et al., 2013). According to Bradley et al. (2007), the disagreed events mainly came from misrecognizing individual players. For example, during defensive situation it is not easy to differentiate who commits an area of goal invasion foul when there are several players inside the goal area. One of the actions with greater disagreement among the observers is the offensive foul. In addition, the intra-class correlation coefficients of committed steps, enters of goal area and caused offensive foul were the lowest ones, while their standardized typical errors were the highest ones. These findings may be due to the fact that their records were much lower compared to other actions and performance indicators.

Conclusion
The present study demonstrates that the method of record using the instrument of observation HandballTAS has a high inter-observer reliability. The statistics generated by the instrument of observation can be valid for subsequent research studies. With the help of the observational instrument and using technology, large volumes of real-time data are collected in a simple and easily notation system. The data obtained showed that only two observers are able to make valid and reliable data form using the HandballTAS for a handball match.