CONTENTS Introduction Methods Results and Discussion References
Northeast Fisheries Science Center Reference Document 06-28
Sandra J. Sutherland, Nina L. Shepherd, and Sarah E. Pregracke
Precision Exercises Associated with SARC 42 Production Aging
National Marine Fisheries Serv., Woods Hole Lab., 166 Water St., Woods Hole MA 02543-1026
Web version posted January 23, 2007Citation: Sutherland SJ, Shepherd NL, Pregracke SE. 2006. Precision exercises with SARC 42 production aging. US Dep Commer, Northeast Fish Sci Cent Ref Doc 06-28; 6 p.
Information Quality Act Compliance: In accordance with section 515 of Public Law 106-554, the Northeast Fisheries Science Center completed both technical and policy reviews for this report. These predissemination reviews are on file at the NEFSC Editorial Office.
In production aging programs, age reader accuracy can be thought of as how often the “right” age is obtained, and precision as how often the “same” age is obtained (Campana 2001). It is possible that, over time, an age reader may inadvertently change the criteria that are used for determining ages, thereby introducing a bias into the age data. This bias can be measured with accuracy tests, which consist of the age reader blindly examining known- or consensus-aged fish from established reference collections. An age reader may also make periodic mistakes, which introduces random errors into the data. The degree of this error can be measured with precision tests, which consist of the age reader blindly re-aging fish which they have already aged. Both accuracy and precision must be considered within a quality-control monitoring program.
Acceptable levels of aging accuracy and precision are influenced by factors such as species, age structure, and age reader experience. Although percent agreement is strongly affected by these differences, the staff of the Fishery Biology Program at the Northeast Fishery Science Center (NEFSC) have long considered levels above 80% to be acceptable. The total coefficient of variation (CV) is less affected by these differences and, thus, is a better measure of aging error. In many aging labs around the world, total CVs of under 5% are considered acceptable among species of moderate longevity and aging complexity (Campana 2001), such as the species considered here.
At the NEFSC Fishery Biology Program, the approach to age-data quality control and assurance has historically been a two-reader system. In this approach, there are both a primary and a secondary age reader for each species. The primary age reader conducts all production aging, in which a large number of samples are aged over a short period of time using established methods (Penttila and Dery 1988). The secondary age reader then ages a portion of those same samples using similar methods. The ages determined by the two readers are compared, and if they agree sufficiently (above 80% agreement), the production ages are considered valid. If not, the sources of disagreement must first be resolved. This interreader approach is still used in the course of training new readers in order to ensure consistency in application of aging criteria and in inter-laboratory sample exchanges. Budgetary and staffing constraints have made this approach less feasible, however, by reducing the number of species for which there are two competent age readers at this laboratory.
In response, the NEFSC Fishery Biology Program has updated our approach to quality control and assurance. Intrareader tests of aging accuracy and precision, as described above, allow us to quantify the amount of inherent aging error and bias in the ages determined by each of our staff members. These values provide a measure of the reliability of the production age data used in stock assessments, and they may be directly incorporated into population models as a source of variability.
For the 42nd Northeast Regional Stock Assessment Review Committee (SARC 42) meeting (NEFSC 2006), exercises were undertaken to estimate the precision of production aging by the Fishery Biology Program for silver hake (Merluccius bilinearis) and Atlantic mackerel (Scomber scombrus). This report lists the results of those exercises. No accuracy tests were conducted, as the NEFSC aging laboratory does not yet have reference collections for these species.METHODS
For precision tests on both species, subsamples were randomly selected from the production sample and re-aged by the same age reader. When re-aging fish, the age reader had knowledge of the same data as during production aging (i.e. fish length, date captured, and area captured) but no knowledge of previous age estimates. During age-testing exercises, no attempts were made to improve results with repeated readings. There was also no attempt to revise the production ages in cases where differences occurred.
Results are presented in terms of percentage agreement, total coefficient of variation (CV), age-bias plots, and age-frequency tables (Campana et al. 1995; Campana 2001). Also, Bowker’s test of symmetry (Bowker 1948; Hoenig et al. 1995) was used in cases where the percent agreement was less than 90%. This statistic tests whether there was a systematic difference between the two readings.
For mackerel, random subsamples were drawn from the 2002 NEFSC spring bottom trawl survey, and NEFSC commercial port samples from the second quarter of 2005 and the fourth quarter of 2002. For the silver hake exercises, a subsample was selected from the 2004 NEFSC spring bottom trawl survey.
The SARC 42 scheduling of both mackerel and silver hake, which are normally aged by the same primary age reader, required that the secondary age reader perform production aging for silver hake. Therefore, an interreader comparison was undertaken for this species to compare the production ages from the secondary reader against test ages by the primary reader, using 2004 NEFSC spring bottom trawl survey samples.RESULTS AND DISCUSSION
The total sample sizes associated with the precision exercises were N = 100 for mackerel and N = 99 for silver hake. Results are summarized in Table 1.
For mackerel (Figure 1), a high level of precision was attained, with 95% agreement and a total CV of 0.7%. No bias was apparent. This indicated an adequate level of consistency in age determinations for this species.
For silver hake (Figure 2), an agreement level of 92% was attained, with a low total CV (1.8%). No bias was apparent. This indicated an adequate level of precision by this age reader.
The comparison between the two silver hake age readers (Figure 3, N = 99) resulted in lower consistency, with 77% agreement and a 5.2% CV. A Bowker’s test of symmetry revealed a significant bias (χ2 = 19.0, P < 0.005, 5 df), primarily due to disagreements at ages 3 and 4. There was no trend in bias, but ages determined by the two age readers differed significantly at age 4.
Recent age determinations appear to have been reliably precise for the species in SARC 42 assessments. Despite this high level of precision, ages generated for silver hake collected on the NEFSC surveys between autumn 2002 and spring 2005 (inclusive) may differ from ages determined for samples from previous years. This type of uncertainty can be reduced or eliminated once the Fishery Biology Program has assembled reference sample collections for all species which we age regularly. These collections will then be available to train new age readers, to refresh current age readers’ skills, and to measure the accuracy of each reader’s ages on a regular basis.REFERENCES
Bowker AH. 1948. A test for symmetry in contingency tables. Journal of the American Statistical Association 43:572–574.
Campana SE. 2001. Accuracy, precision, and quality control in age determination, including a review of the use and abuse of age validation methods. Journal of Fish Biology 59:197-242.
Campana SE, Annand MC, McMillan JI. 1995. Graphical and statistical methods for determining the consistency of age determinations. Transactions of the American Fisheries Society 124:131-138.
Hoenig JM, Morgan MJ, Brown CA. 1995. Analysing differences between two age determination methods by tests of symmetry. Canadian Journal of Fisheries and Aquatic Science 52:364–368.
Northeast Fisheries Science Center (NEFSC). 2006. 42nd northeast regional stock assessment workshop stock assessment report. NEFSC Reference Document 06-09. 308 p. Available at http://nefsc.noaa.gov/publications/crd/crd0609/
Penttila J and Dery LM. 1988. Age determination methods for northwest Atlantic species. National Oceanic and Atmospheric Administration Technical Report NMFS 72; 135 p. Available at http://www.nefsc.noaa.gov/fbi/age-man.html