# ASTM E3064-16

Designation: E3064 − 16Standard Test Method forEvaluating the Performance of Optical Tracking Systemsthat Measure Six Degrees of Freedom (6DOF) Pose1This standard is issued under the fixed designation E3064; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon (´) indicates an editorial change since the last revision or reapproval.1. Scope1.1 Purpose—This test method presents metrics and proce-dures for measuring, analyzing, and reporting the relative poseerror of optical tracking systems that compute the pose (that is,position and orientation) of a rigid object while the object ismoving.1.2 Usage—System vendors may use this test method todetermine the performance of their Six Degrees of Freedom (6DOF) optical tracking system which measures pose. This testmethod also provides a uniform way to report the measurementerrors and measurement capability of the system. System usersmay use this test method to verify that the system’s perfor-mance is within the user’s specific requirements and within thesystem’s rated performance.1.3 Test Location—The procedures defined in this standardshall be performed in a facility in which the environmentalconditions are within the optical tracking system’s ratedconditions.1.4 Test Volume—This standard shall be used for testing anoptical tracking system working volumes of 3000 mm long by2000 mm wide by 2000 mm high, 6000 mm long by 4000 mmwide by 2000 mm high, or 12 000 mm long by 8000 mm wideby 2000 mm high.1.5 Units—The values stated in SI units are to be regardedas standard. No other units of measurement are included in thisstandard.1.6 This standard does not purport to address all of thesafety concerns, if any, associated with its use. It is theresponsibility of the user of this standard to establish appro-priate safety and health practices and determine the applica-bility of regulatory limitations prior to use.2. Referenced Documents2.1 ASTM Standards:2E2919 Test Method for Evaluating the Performance ofSystems that Measure Static, Six Degrees of Freedom(6DOF), PoseE177 Practice for Use of the Terms Precision and Bias inASTM Test Methods2.2 ASME Standard:3B89.4.19 Performance Evaluation of Laser-Based SphericalCoordinate Measurement Systems2.3 ISO/IEC Standards:4ISO/IEC Guide 99:2007 International Vocabulary ofMetrology—Basic and General Concepts and AssociatedTerms (VIM: 2007)ISO/IEC Guide 98–3:2008 Uncertainty of measurement—Part 3: Guide to the expression of uncertainty in measure-ment (GUM: 1995)IEC 60050-300:2001 International Electro technicalVocabulary—Electrical and electronic measurements andmeasuring instrumentsJCGM 200:2012 International Vocabulary of Metrology Ba-sic and General Concepts and Associated Terms (VIM),3rd edition3. Terminology3.1 Definitions:3.1.1 degrees of freedom, DOF, n—any of the minimumnumber of translation or rotation components required tospecify completely the pose of a rigid object. E29193.1.1.1 Discussion—1This test method is under the jurisdiction of ASTM Committee E57 on 3DImaging Systems and is the direct responsibility of Subcommittee E57.02 on TestMethods.Current edition approved June 1, 2016. Published June 2016. DOI: 10.1520/E3064-162For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at service@astm.org. For Annual Book of ASTMStandards volume information, refer to the standard’s Document Summary page onthe ASTM website.3Available from American Society of Mechanical Engineers (ASME), ASMEInternational Headquarters, Two Park Ave., New York, NY 10016-5990, http://www.asme.org.4Available from American National Standards Institute (ANSI), 25 W. 43rd St.,4th Floor, New York, NY 10036, http://www.ansi.org.Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United States1(1) In a 3D space, a rigid object can have at most 6DOF,three translations and three rotations.(2) The term “degree of freedom” is also used with regardto statistical testing. It will be clear from the context in whichit is used whether the term relates to a statistical test or therotation/translation aspect of the object.3.1.2 measurement error, error of measurement, and error,n—measured quantity value minus a reference quantity value.(JCGM 200:2012)3.1.3 metrology bar, n—a rod of a known length havingmarkers (active or passive) attached to both ends and used toestimate the errors of an optical tracking system.3.1.4 optical tracking system, n—a tracking system that usesmeasurements obtained from camera images.3.1.5 pose, n—a 6DOF vector whose components representthe position and orientation of a rigid object with respect to acoordinate frame. E29193.1.6 precision, n—the closeness of agreement betweenindependent test results obtained under stipulated conditions.E1773.1.7 rated conditions, n—manufacturer-specified limits onenvironmental, utility, and other conditions within which themanufacturer’s performance specifications are guaranteed atthe time of installation of the instrument. ASME B89.4.193.1.8 reference system, n—a measurement instrument orsystem used to generate a reference value or quantity. E29193.1.9 relative pose, n—change of an object’s pose betweentwo poses measured in the same coordinate frame. E29193.1.10 repeatability, n—precision under repeatabilityconditions. E1773.1.11 repeatability conditions, n—conditions where inde-pendent test results are obtained with the same method onidentical test items in the same laboratory by the same operatorusing the same equipment within short intervals of time. E1773.1.12 tracking system, n—a system that is used for mea-suring the pose of moving objects and supplies the data as atimely ordered sequence.3.1.13 work volume, n—a physical space, or region within aphysical space, that defines the bounds within which a rigidobject tracking system is acquiring data. E29194. Summary of Test Method4.1 This test method provides a set of statistically basedperformance metrics and a test procedure to quantitativelyevaluate the performance of an optical tracking system.4.2 The measurement errors include the positional andorientation error components. Specifically, the test proceduremeasures the relative pose between two marker sets rigidlyattached to the opposing ends of a fixed-length metrology baras shown in Fig. 1. The relative pose is then decomposed intopositional and angular components. Measurement errors arecalculated from the positional and angular components as theartifact is moved about the work volume.5. Significance and Use5.1 Optical tracking systems are used in a wide range offields including: video gaming, filming, neuroscience,biomechanics, flight/medical/industrial training, simulation,robotics, and automotive applications.5.2 This standard provides a common set of metrics and atest procedure for evaluating the performance of opticaltracking systems and may help to drive improvements andinnovations of optical tracking systems.5.3 Potential users often have difficulty comparing opticaltracking systems because of the lack of standard performancemetrics and test methods, and therefore must rely on the claimsof a vendor regarding the system’s performance, capabilities,and suitability for a particular application. This standard makesit possible for a user to assess and compare the performance ofcandidate optical tracking systems, and allows the user todetermine if the measured performance results are within thespecifications with regard to the application requirements.6. Apparatus6.1 Artifact:6.1.1 A 300 mm long bar with markers rigidly attached toeach end of the metrology bar shall be used as the 6DOFartifact. The bar shall have stiffness and thermal expansioncharacteristics such that the deflection is less than or equal to0.01 mm. For example, the metrology bar shown in Fig. 2satisfies these requirements.6.1.2 A constant relative 6DOF pose is formed between thetwo clusters of markers located at the ends of the metrologybar. All markers shall be contained within hemisphericalvolumes with a maximum radius of 100 mm from the ends ofthe bar (see Fig. 1). Examples of metrology bars that can beused to evaluate optical tracking systems are shown in Fig. 2(Ref (1))5.5The boldface numbers in parentheses refer to a list of references at the end ofthis standard.FIG. 1 Drawing of the artifact showing critical dimensions of thebar length and the maximum hemispherical volume inside whichthe markers can be placed.E3064 − 1627. Measurement Procedure7.1 Introduction:7.1.1 This section describes the basic procedure for deter-mining the pose measurement error of an optical trackingsystem.7.2 Pose Measurement:7.2.1 The X and Y axes are aligned with the work volume inthe horizontal plane as shown in Fig. 3, and Z is aligned withthe vertical axis. Move the metrology bar throughout the workvolume along two regular patterns: (X pattern) parallel, straightline segments back-and-forth along the X axis with the pathsseparated by at most, the metrology bar length as shown in Fig.3 (a) and (Y pattern) parallel, straight line segments back-and-forth along the Y axis with the paths separated by at most, themetrology bar length as shown in Fig. 3 (b). The distancebetween the boundary lines and the limits of the work volumeshall be at most, one-half of the metrology bar length. Thecenter of the metrology bar shall traverse the X patternfollowed by the Y pattern with artifact orientation #1 as shownin Fig. 3 (c) in a continuous smooth motion.This traversal shallbe repeated two more times, once with artifact orientation #2and once with artifact orientation #3. The data from all threetraversals shall be combined into a single data set.7.2.2 The metrology bar paths and orientations shall bechosen as described in 7.2.1. The height of the centroid of themetrology bar shall remain approximately 1000 mm above thebottom of the test volume.FIG. 2 Examples of artifacts for evaluating optical tracking systems having a 300 mm long metrology bar (a) with six passive, reflectivemarkers, within 100 mm radius from the bar end, on each end, and (b) with a reduced pose ambiguity cuboctahedron, Ref (1), within100 mm radius from the bar end, on each end.FIG. 3 The (a) X pattern and (b) Y pattern are combined to make a single path along which the metrology bar is moved throughout thework volume. (c) Artifact (shown with axes on bar center) orientations with respect to the path: 1) perpendicular to the path segmentsin the plane of motion, 2) perpendicular to the path segments and normal to the plane of motion, and 3) in-line with the path segmentsin the plane of motion.NOTE 1—Example artifact shown in (a) and (b) is oriented with respect to the path as in 1) perpendicular to the path segments in the plane of motion.E3064 − 1637.2.3 The centroid of the metrology bar shall be moved at arelatively constant walking speed of 1200 6 700 mm/s.8. Pose Measurement Error8.1 This section describes methods for computing posemeasurement errors of an optical tracking system (OTS) usingthe artifact. For each instance of time t, the optical trackingsystem measures the pose of an object as:OTSHˆObject~t! 5FOTSRˆObject~t!0OTSTˆObject~t!1G(1)Here,OTSRˆObject~t! isa3×3matrix describing the orientationof the object andOTSTˆObject~t! is a 3-dimensional vector de-scribing the position of the object in the optical tracking sys-tem coordinate frame. In our artifact, two objects are consid-ered corresponding to the left and right ends of themetrology bar. The optical tracking system measures theposes of the left and right ends, and the corresponding4×4matrices are defined respectively as:OTSHˆLeft~t! 5FRˆLeft~t!0TˆLeft~t!1Gand(2)OTSHˆRight~t! 5FRˆRight~t!0TˆRight~t!1GBecause the bar is rigid, the relative pose between the leftobject and the right object is constant in time (see Fig. 4)and can be defined as:LeftHˆRight~t! 5OTSHˆLeft21OTSHˆRight5FRˆLeft~t!0TˆLeft~t!1G21FRˆRight~t!0TˆRight~t!1G5FRˆ~t!0Tˆ~t!1G(3)The angle of rotation is calculated as:θˆ~t! 5 2* asin ~=qˆx2~t!1qˆy2~t!1qˆz2~t!! (4)Here, (qˆw(t),qˆx(t),qˆy(t),qˆz(t))Tis the unit quaternion represen-tation of Rˆ~t!, Ref (2), where qˆw~t! is the scalar componentof the quaternion.8.1.1 A reference system measurement is used to measurethe relative pose between two groups of markers.The referencesystem measurement shall have an uncertainty that is at leastten times smaller than the uncertainty of the optical trackingsystem under test. The relative pose between the left object andright object at the ends of the metrology bar is measured by thereference system measurement as:LeftHRight5FR0T1G5FI0T1G(5)The coordinate frames associated with the right and left endsof the bar are rotationally aligned using the reference sys-tem. Where R=I, and I is the identity matrix.8.1.2 The following sections describe two methods forevaluating the optical tracking system. In 8.2, measurementsare taken relative to the measured relative pose of a testartifact, which is measured by a more accurate system, and in8.3 the measurements are taken relative to the mean value ofthe collected data.8.2 Error Statistics using a Reference System:8.2.1 This section describes the computation of system errorstatistics relative to a precisely characterized test artifact, suchas the one described in Section 6.8.2.2 The relative measured poseLeftHˆRight~t! (see Eq 3)attime t can be compared to the reference system measured poseLeftHRight(see Eq 5). Specifically, the positional error at time tcan be calculated as:ep~t!5 ǁTˆ~t!ǁ22 ǁTǁ2(6)and the orientation error at time t can be calculated as:eo~t!5 θˆ~t! 2 0 5 θˆ~t! (7)where θˆ~t! is calculated using Eq 4 and ǁǁ2denotes the2-norm of the vector.8.2.3 The statistics on these errors include: the root meansquare error, the maximum error, and the percentile error. Theroot mean square error is calculated as:Root Mean Square Error 5 Œ1N(t51Net2(8)The maximum of the errors is defined as:emax5 max~?e1?,?e2?,.?eN?! (9)Here, etis the (positional or orientation) error at time t, andN is the number of data samples collected.8.2.4 A method for estimating the percentile error of a dataset is described in Ref (3). For a series of errors {|e1|,|e2|,.,|eN|}, a new ordered set {E1, E2, ., EN} is constructed wherethe errors are ordered by increasing value. The percentile errorof the data can be estimated from N measurements as follows:for the pth percentile, setp100~N 1 1! equal to k+d for k,aninteger, and d, a fraction greater than or equal to 0 and less than1. The estimated error percentile E(p) is defined as follows:E~p! 5HEk1d~Ek112 Ek! 0,k,NE1, k 5 0EN, k $ N(10)In this standard, E(99.7), E(95) and E(50) shall be reported.8.3 Repeatability without Reference System Measurement:8.3.1 This section describes the computation of systemrepeatability statistics using a test artifact not meas