upsp.cam_cal_utils.external_calibrate

check_external_calibrate_two_stage_inputs(img, rmat_init_guess, tvec_init_guess, incal, tgts, test_config, vis_checker, debug)[source]

Check that the inputs to external calibrate are valid

Helper function to external_calibrate_two_stage(). Checks that the input are of the proper type, size, format, and have the relevant attributes.

Parameters

  • img (numpy.ndarray, shape (h, w)) – Numpy 2D array of the image

  • rmat_init_guess (np.ndarray (3, 3), float) – Initial guess of rotation matrix from camera to object

  • tvec_init_guess (np.ndarray (3, 1), float) – Initial guess of translation vector from camera to object

  • incal (tuple) –

    Camera internal calibration.

    • cameraMatrix (numpy.ndarray, shape (3, 3)): The (OpenCV formatted) camera matrix for the camera

    • distCoeffs (numpy.ndarray, shape (1, 5): The (OpenCV formatted) distortion coefficients for the camera

  • tgts (list of dict) – Each target is a dict and has, at a minimum, the keys ‘tvec’, ‘target_type’, ‘norm’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i]

  • test_config (dict) – Processing parameters with, at a minimum, a key for ‘min_dist’, ‘max_dist’, and 2 keys for the primary target type in targets and img_targets. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target. The primary target type is ‘dot’ if there are 4+ dots in tgts. Otherwise the primary target type is ‘kulite’. The keys for the primary target type are target_type + ‘_pad’ and target_type + ‘_blob_parameters’. The first is the padding around the img target center location to use to sub-pixel localize. The second is the blob detection parameters for that type of target

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • debug (string, optional) – Name for all debug images (potentially camera name, test an camera name, etc).

Returns

valid – True if the all inputs are valid, False if any input is invalid

Return type

bool

compare_poses(pose0, pose1, more_info=False)[source]

Returns the angle and distance in which the two poses differ

Any two rotation matrices are related by a single rotation of theta about a given axis. Any two translation vectors are related by an [X, Y, Z] translation vector. This function returns the angle between the two poses, as well as the distance formatted as [theta, dist] where theta is in degrees and dist is in the units of tvec. If more_info is True, [theta, axis, tvec_rel] is returned where axis is the axis of rotation.

Parameters

  • pose0 (tuple) – Pose to compare: (rmat, tvec), where rmat is the rotation matrix from camera to object (numpy.ndarray with shape (3, 3)) and tvec is the translation vector from camera to object (numpy.ndarray with shape (3, 1)).

  • pose1 (tuple) – Pose to compare: (rmat, tvec), where rmat is the rotation matrix from camera to object (numpy.ndarray with shape (3, 3)) and tvec is the translation vector from camera to object (numpy.ndarray with shape (3, 1)).

  • more_info (bool, optional) – Changes the return value. Function returns (theta, distance) if more_info is False, function returns (rvec, tvec) if more_info is True

Returns

If more_info is False, returns (theta, distance). If more_info is True, return is (rvec, tvec)

Return type

tuple

external_calibrate(img, rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, vis_checker, test_config, isMatched=False, max_localize_delta=None, reprojectionError=6.0)[source]

Set up and run solvePnPRansac to get the external calibration and inlier targets

Parameters

  • img (numpy.ndarray, shape (height, width)) – Image to use for calibration

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • cameraMatrix (numpy.ndarray, shape (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (1, 5), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (list of dict) – Each target is a dict that has, at a minimum, the keys ‘tvec’, and ‘target_type’. If isMatched is False, ‘norm’ is additionally needed as a key. ‘tvec’ has a np.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i] for i from 0 to num_matches. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model coordinate system for its associated value

  • img_targets (list of dict) – Matched image targets. Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string). img_targets[i] is associated with tgts[i]

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • test_config (dict) – dict with, at a minimum, a key for ‘min_dist’, ‘max_dist’, and each target type in targets and img_targets. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target. The key for each target type is target_type + ‘_pad’. This is the padding around the img target center location to use to sub-pixel localize

  • isMatched (bool) – If True, denotes that tgts[i] is associated with img_targets[i] and all targets are visible to the camera. If False, denotes tgts and img_targets are not in any particular order. If False, targets are checked for visibility and match_targets() is used to match the tgts to the image targets

  • max_localize_delta (float, optional) – Parameter passed to subpixel_localize()

  • reprojectionError (float, optional) – Maximum reprojection error between a target and image target to be considered an inlier. ReprojectionError is often smaller than test_config['max_dist'] since it is the optimized distance between the target and image target.

Returns

  • rmat_opt – optimized rotation matrix from the camera to the model

  • tvec_opt – optimized translation vector from the camera to the model

  • tgt_inliers – list of inlier targets of the optimization

  • img_target_inliers – list of the inlier image targets of the optimization

Notes

tgt_inliers[i] is associated with img_target_inliers[i]

external_calibrate_RANSAC(incal, tgts, img_targets, vis_checker, max_iter=0.999, max_dist=8, match_thresh=0.8)[source]

Use RANSAC to find an external calibration with brute force

To find an external calibration, select at random 3 targets and 3 img targets. Solve the associated P3P problem to get the external calibration(s). For each of the external calibrations (P3P can yield up to 4), find the visible targets, project them, and match them to the image targets. If there is sufficient consensus amoung the matches (i.e. the randomly selected targets and image targets yields a solution where many other unselected targets project to a location close to an image target), return that external calibration. If there is not sufficient consensus, repeat this process.

Parameters

  • incal (tuple) –

    Camera internal calibration.

    • cameraMatrix (numpy.ndarray, shape (3, 3)): The (OpenCV formatted) camera matrix for the camera

    • distCoeffs (numpy.ndarray, shape (1, 5): The (OpenCV formatted) distortion coefficients for the camera

  • tgts (list of dict) – Each target is a dict with (at a minimum) ‘tvec’, ‘norm’, and ‘target_type’ attributes. The ‘tvec’ attribute gives the target’s location and the ‘norm’ attribute gives the target’s normal vector. ‘target_type’ is a string denoting the type of target (most commonly ‘dot’ or ‘kulite’)

  • img_targets (list of dict) – Each img_target is a dict with (at a minimum) a ‘center’ attribute. The ‘center’ attribute gives the pixel position of the target’s center in the image. img_targets[i] is associated with tgts[i]

  • vis_checker (VisibilityChecker) – VisibilityChecker object with the relevant BVH and oblique viewing angle

  • max_iter (int or float, optional) – If int, it represents the maximum number of iterations for RANSAC. If float must be greater than 0 but less than 1. Float means it represents the number of estimate number of iterations in order to have that probability of finding a solution should one exist (0.999 means there’s a 99.9% chance of finding a solution should one exist)

  • max_dist (int or float, optional) – Maximum matching distance between image targets and projected target location

  • match_thresh (int or float, optional) – If int, this is the number of matches that must be found to deem it a consensus. If float, must be greater than 0 but less than 1. If float, this is the proportion of tgts or img_targets (whichever is lower) that must be matched. I.e. If match_thresh is 0.8 and there are 50 target and 20 img_targets, there must be 16+ matches for it to be deemed a consensus.

Returns

  • rmat (numpy.ndarray, shape (3, 3)) – Rotation matrix. May be None if the maximum number of iterations is met without finding a valid solution.

  • tvec (numpy.ndarray, shape (3, 1)) – Translation vector. May be None if the maximum number of iterations is met without finding a valid solution.

external_calibrate_one_step(img, rmat_coarse, tvec_coarse, incal, tgts, test_config, vis_checker, debug=None)[source]

Performs external calibration from a seim-accurate coarse guess

The coarse guess of rmat and tvec should project each target’s within 1 pixel of the associated image target

Parameters

  • img (numpy.ndarray, shape (h, w)) – Numpy 2D array of the image

  • rmat_coarse (numpy.ndarray, shape (3, 3), float) – Coarsely refined rotation matrix from camera to object

  • tvec_coarse (numpy.ndarray, shape (3, 1), float) – Coarsely refined translation vector from camera to object

  • incal (tuple) –

    Camera internal calibration.

    • cameraMatrix (numpy.ndarray, shape (3, 3)): The (OpenCV formatted) camera matrix for the camera

    • distCoeffs (numpy.ndarray, shape (1, 5): The (OpenCV formatted) distortion coefficients for the camera

  • tgts (list of dict) – Each target is a dict and has, at a minimum, the keys ‘tvec’, ‘target_type’, ‘norm’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i]

  • test_config (dict) – Processing parameters with, at a minimum, a key for ‘min_dist’, ‘max_dist’, and 2 keys for the primary target type in targets and img_targets. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target. The primary target type is ‘dot’ if there are 4+ dots in tgts. Otherwise the primary target type is ‘kulite’. The keys for the primary target type are target_type + ‘_pad’ and target_type + ‘_blob_parameters’. The first is the padding around the img target center location to use to sub-pixel localize. The second is the blob detection parameters for that type of target

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • debug (string or None. Optional, default None) – Name for all debug images (potentially camera name, test an camera name, etc)

Returns

  • rmat (numpy.ndarray, shape (3, 3), float) – camera-to-model external calibration rotation matrix

  • tvec (numpy.ndarray, shape (3, 1), float) – camera-to-model external calibration translation vector

external_calibrate_two_stage(img, rmat_init_guess, tvec_init_guess, incal, tgts, test_config, vis_checker, debug=None)[source]

Performs external calibration from an inaccurate initial guess

Runs blob detection to find the img targets, then matches and filters the 3D targets to the img targets. Performs a coarse optimization to improve the intial guess. Then calls external_calibrate_one_step() to get the refined optimization

The initial guess of rmat and tvec should project each target’s within ~5 pixels of the associated image target

Parameters

  • img (numpy.ndarray, shape (h, w)) – Numpy 2D array of the image

  • rmat_init_guess (np.ndarray (3, 3), float) – Initial guess of rotation matrix from camera to object

  • tvec_init_guess (np.ndarray (3, 1), float) – Initial guess of translation vector from camera to object

  • incal (tuple) –

    Camera internal calibration.

    • cameraMatrix (numpy.ndarray, shape (3, 3)): The (OpenCV formatted) camera matrix for the camera

    • distCoeffs (numpy.ndarray, shape (1, 5): The (OpenCV formatted) distortion coefficients for the camera

  • tgts (list of dict) – Each target is a dict and has, at a minimum, the keys ‘tvec’, ‘target_type’, ‘norm’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i]

  • test_config (dict) – Processing parameters with, at a minimum, a key for ‘min_dist’, ‘max_dist’, and 2 keys for the primary target type in targets and img_targets. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target. The primary target type is ‘dot’ if there are 4+ dots in tgts. Otherwise the primary target type is ‘kulite’. The keys for the primary target type are target_type + ‘_pad’ and target_type + ‘_blob_parameters’. The first is the padding around the img target center location to use to sub-pixel localize. The second is the blob detection parameters for that type of target

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • debug (string or None. Optional, default None) – Name for all debug images (potentially camera name, test an camera name, etc)

Returns

  • rmat (numpy.ndarray, shape (3, 3)) – Valid solution camera-to-model rotation matrix

  • tvec (numpy.ndarray, shape (3, 1)) – Valid solution camera-to-model translation vector

external_calibrate_two_stage_from_wtd(img, tunnel_vals, camera_tunnel_cal, tgts, test_config, vis_checker, debug=None)[source]

Wrapper function to external_calibrate_two_stage().

The tunnel_vals plus test_config are used to estimate an initial guess of rmat and tvec. That initial guess should project each target’s within ~5 pixels of the associated image target

Parameters

  • img (numpy.ndarray, shape (h, w)) – Numpy 2D array of the image

  • tunnel_vals (dict) – Wind tunnel data as a dict with (at a minimum) the keys ‘ALPHA’, ‘BETA’, ‘PHI’, and ‘STRUTZ’. ALPHA, BETA, and PHI are tunnel angles in degrees. STRUTZ is the offset of the tunnel center of rotation for the z axis in inches

  • camera_tunnel_cal (tuple) –

    Camera-tunnel calibration

    • rmat_camera_tunnel (numpy.ndarray, shape (3, 3)): Rotation matrix from camera to tunnel at wind off condition

    • tvec_camera_tunnel (numpy.ndarray, shape (3, 1)): Translation vector from camera to tunnel at wind off condition

    • cameraMatrix (numpy.ndarray, shape (3, 3)): The (OpenCV formatted) camera matrix for the camera

    • distCoeffs (numpy.ndarray, shape (1, 5)): The (OpenCV formatted distortion coefficients for the camera

  • tgts (list of dict) – 3D targets. Each target is a dict and has, at a minimum, the keys ‘tvec’, ‘target_type’, ‘norm’. ‘tvec’ has a np.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i]

  • test_config (dict) – Processing parameters with, at a minimum, a key for ‘min_dist’, ‘max_dist’, and 2 keys for the primary target type in targets and img_targets. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target. The primary target type is ‘dot’ if there are 4+ dots in tgts. Otherwise the primary target type is ‘kulite’. The keys for the primary target type are target_type + ‘_pad’ and target_type + ‘_blob_parameters’. The first is the padding around the img target center location to use to sub-pixel localize. The second is the blob detection parameters for that type of target

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • debug (string, optional) – Name for all debug images (potentially camera name, test an camera name, etc)

Returns

  • rmat (numpy.ndarray, shape (3, 3), float) – camera-to-model external calibration rotation matrix

  • tvec (numpy.ndarray, shape (3, 1), float) – camera-to-model external calibration translation vector

filter_bifilter(rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, num_matches, max_dist)[source]

Filters the targets and image target for ambiguity

Check that only 1 img target is within a radius of max_dist from every projected target location. If more than 1 img target is near a target, do not use that target

Check that only 1 projected target location is within a radius of max_dist from every img target. If more than 1 projected target location is near an img target, do not use that img target

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • cameraMatrix (numpy.ndarray, shape (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (1, 5), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (list) – 3D targets. Each target is a dict and has, at a minimum, the keys ‘tvec’, and ‘target_type’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list) – Matched image targets. Each dict has, at a minimum, the key ‘center’ which has a value of is the image target center location (tuple/np.ndarray of floats). img_targets[i] is associated with tgts[i] for i from 0 to num_matches

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

  • max_dist (float) – The maximum distance between two image targets. Any image targets farther than this distance are filtered out.

Returns

  • tgts_bifilter (list) – Target positions (tgts) that have been bifiltered.

  • img_targets_bifilter (list) – Target positions in the image.

  • num_matches_bifilter (int)

Notes

tgts_matched_bifilter[i] is associated with img_targets_bifilter[i] for i from 0 to num_matches_bifilter.

filter_matches(rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, num_matches, test_config, debug=None)[source]

Wrapper to run multiple filtering functions

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • cameraMatrix (numpy.ndarray, shape (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (1, 5), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (list) – 3D targets. Each target is a dict that has, at a minimum, the keys ‘tvec’, and ‘target_type’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list) – Matched image targets. Each dict has, at a minimum, the key ‘center’ which has a value of is the image target center location (tuple/np.ndarray of floats). img_targets[i] is associated with tgts[i] for i from 0 to num_matches

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

  • test_config (dict) – Processing parameters with, at a minimum, a key for ‘min_dist’, ‘max_dist’. ‘min_dist’ is the minimum distance between two image targets. ‘max_dist’ is the maximum allowable distance between an image target and the projection of a 3D target

  • debug (tuple, optional) – tuple of length 2. First item is the image to use as the background. Second item is the name of the debug image

Returns

  • tgts_filtered (list) – Targets filtered to remove None values, matches greater than the maximum matching distance, matches that are not one-to-one, matches that do not pass a “bifilter” test, and matches that have image locations that are too close together

  • img_targets_filtered (list) – Associated image locations of tgts_filtered

filter_max_dist(rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, num_matches, max_dist)[source]

Filters the targets and image target to pairs are not very far apart

Any match where the distance between the projected target pixel position and the img target center is greater than max_dist is removed.

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • cameraMatrix (np.ndarray (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (1, 5), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (list) – 3D targets. Each target is a dict and has, at a minimum, the keys ‘tvec’, and ‘target_type’. ‘tvec’ has a np.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i] for i from 0 to num_matches.

  • img_targets (list) – Matched image targets. Each dict has, at a minimum, the key ‘center’ which has a value of is the image target center location (tuple/np.ndarray of floats). img_targets[i] is associated with tgts[i] for i from 0 to num_matches

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

  • max_dist (float) – The maximum matching distance between a 3D target’s project and an image targets center. Pairs with matching distance greater this distance are filtered out.

Returns

  • tgts_filtered (list) – Target positions (tgts) that have been filtered such that the matching distance is less than max_dist.

  • img_targets_filtered (list) – Target positions in the image.

  • num_matches_filtered (int)

Notes

tgts_filtered[i] is associated with img_targets_filtered[i] for i from 0 to num_matches_filtered.

filter_min_dist(tgts, img_targets, num_matches, min_dist=8)[source]

Filters the targets and image target that are too close together in the image

If the image locations of any two targets are below the min_dist threshold, remove both targets from the set of matched targets. This is to avoid ambiguity in target matching.

Parameters

  • tgts (list) – 3D targets. Each target should be a dict. The only strict requirement set by this function is tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list) – Matched image targets. Each dict has, at a minimum, the key ‘center’ which has a value of the image target center location (tuple/np.ndarray of floats). img_targets[i] is associated with tgts[i] for i from 0 to num_matches

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

  • min_dist (float, optional) – The minimum distance between two image targets. Any image targets closer than this distance are filtered out.

Returns

  • tgts_filtered (list) – Target positions (tgts) that have been filtered so that their image locations are separated by a distance larger than min_dist.

  • img_targets_filtered (list) – Target positions in the image.

  • num_matches_filtered (int)

Notes

tgts_filtered[i] is associated with img_targets_filtered[i] for i from 0 to num_matches_filtered.

filter_nones(tgts, img_targets, num_matches)[source]

Filters the targets and image target to remove None objects

match_obj_and_img_pts() matches each target to the closest img target. Therefore, it is possible that not every img target will have a match. Additionally, if max_dist is given to match_obj_and_img_pts(), it is possible that not every target will have a match. match_obj_and_img_pts() solves this by matching unmatched items to None.

That causes something to fail, so this function reformats that output. Instead of using Nones, this function reorders the list of targets to the first n target have a match, and correspond to the first n items of the img targets list. n is additionally given as num_matches

Parameters

  • tgts (list) – 3D targets. Each target should be a dict. The only strict requirement set by this function is tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list) – Image targets. Each image target should be a dict, but this function does not set any strict requirements other than that img_targets[i] is associated with tgts[i]

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

Returns

  • tgts_filtered (list) – Target positions (tgts) that have been filtered to remove None values.

  • img_targets_filtered (list) – Target positions in the image.

  • num_matches_filtered (int)

Notes

tgts_filtered[i] is associated with img_targets_filtered[i] for i from 0 to num_matches_filtered.

filter_one2one(tgts, img_targets, num_matches)[source]

Filters the targets and image target to ensure matches are one-to-one

Ensures that each target is matched with at most 1 img target, and that each img target is matched with at most 1 target. Remove anything that is not 1:1 as stated

Parameters

  • tgts (list) – 3D targets. Each target should be a dict. The only strict requirement set by this function is tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list) – Image targets. Each image target should be a dict, but this function does not set any strict requirements other than that img_targets[i] is associated with tgts[i] for i from 0 to num_matches

  • num_matches (int) – An integer of the number of matches. Must be less than or equal to min(len(tgts), len(img_targets))

Returns

  • tgts_filtered_one2one (list) – Target positions (tgts) that have been filtered for pairs that are one-to-one.

  • img_targets_one2one (list) – Target positions in the image.

  • num_matches_one2one (int)

Notes

tgts_one2one[i] is associated with img_targets_one2one[i] for i from 0 to num_matches_one2one.

filter_partially_occluded(rmat, tvec, focal_length, tgts, img_targets, vis_checker, test_config)[source]

Checks corners of cropped area used for sub-pixel localization for occlusion

If the corners of the crop used for sub-pixel localization jumps surfaces, then the target is likely partially occluded. This most commonly occurs when a booster partially occludes a target on the core of a launch vehicle

To get the 3D positions of corners of the cropped area, start at the target tvec. Approximate the physical distance (in inches) of the cropped area (which is done in pixels) using the focal length and distance from camera to model. Take steps along the camera image plane to get to the approximate corner locations in 3D.

With the corner locations, ensure that they are not inside the model (since the model is curved the steps along the image plane may put the corners slightly inside the model. Then check for occlusions. If the corner is occluded, the target is deemed partially occluded. If none of the corners are occluded, the target is deemed not occluded (but is still potentially partially occluded). Only the corners are checked to reduce computation

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • focal_length (float) – Focal length of the camera. Most easily accessible from a cameraMatrix[0][0]

  • tgts (list) – 3D targets. Each target is a dict and has, at a minimum, the keys ‘tvec’, ‘target_type’, ‘norm’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘norm’ has a numpy.ndarray (3, 1) representing the normal vector of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i]

  • img_targets (list) – Image targets. Each image target should be a dict, but this function does not set any strict requirements other than that img_targets[i] is associated with tgts[i]

  • vis_checker (VisibilityChecker) – Visibility checker object with the relevant BVH and oblique viewing angle

  • test_config (dict) – Processign parameters with, at a minimum, a key for each target type of the targets in tgts and img_targets. They key is target_type + ‘_pad’. This is the padding around the img target center location to use to sub-pixel localize

Returns

  • tgts_filtered (list) – Target positions (tgts) that have not been filtered out

  • img_targets_filtered (list) – Target positions in the image.

match_obj_and_img_pts(rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, max_dist=inf)[source]

Matches 3D targets to the image targets

Projects the 3D targets into the image, then finds the closest image target. If the closest image target is less than max_dist pixels away, it is matched. If it is farther than max_dist pixels, it is matched to None. This matching scheme does not ensure matches are one-to-one.

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3,), float) – Translation vector from camera to object

  • cameraMatrix (numpy.ndarray, shape (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (5,), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (numpy.ndarray, shape (n, 3)) – List of 3D targets.

  • img_targets (list of dict) – Matched image target information. Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string). img_targets[i] is associated with tgts[i]

  • max_dist (float) – The maximum matching distance between a 3D target’s project and an image targets center. Pairs with matching distance greater this distance are filtered out.

  • Returns –

  • ----------- –

  • matching_img_targets (list) – List of the items from the img_targets input such that matching_img_targets[i] is the closest image target to the projected position of tgts[i]. If the closest image target is farther than max_dist, matching_img_targets[i] is None

match_targets(rmat, tvec, cameraMatrix, distCoeffs, tgts, img_targets, max_dist=inf, debug=None)[source]

Matches each target to the closest img target

If max_dist is given, distance between target and img target must be less than max_dist. By default this value is infinite, so any match is valid.

Parameters

  • rmat (numpy.ndarray, shape (3, 3), float) – Rotation matrix from camera to object

  • tvec (numpy.ndarray, shape (3, 1), float) – Translation vector from camera to object

  • cameraMatrix (numpy.ndarray, shape (3, 3), float) – The (openCV formatted) camera matrix for the camera

  • distCoeffs (numpy.ndarray, shape (1, 5), float) – The (openCV formatted) distortion coefficients for the camera

  • tgts (list) – 3D targets. Each target is a dict and has, at a minimum, the keys ‘tvec’, and ‘target_type’. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value. tgts[i] is associated with img_targets[i] for i from 0 to num_matches

  • img_targets (list of dict) – Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string). img_targets[i] is associated with tgts[i]

  • max_dist (float) – The maximum matching distance between a 3D target’s project and an image targets center. Pairs with matching distance greater this distance are filtered out.

  • debug (tuple, optional) – Debug option. For no debugging, give None. To generate debugging images, give a tuple of length 3. First item is the image to use as the background. Second item is the name of the debug image. Third item can be a dict or None. If dict, matches will be filtered before the debug image is created. Dict needs to follow test_config input requirements of filter_matches. If None, matches will not be filtered in the debug image

Returns

  • tgts_matched (list) – 3D targets that are matched

  • matching_img_targets (list) – Image locations of matched targets

  • num_matches (int)

Notes

tgts_matched[i] is matched with matching_img_targets[i] for i from 0 to num_matches.

post_filter_append(tgts, tgts_filtered, img_targets, img_targets_filtered)[source]

Adds tgts that were filtered back into the list

The match-and-filter scheme we use is to have two ordered lists, where tgts_filtered[i] is matched to img_targets_filtered[i] for i from 0 to num_matches. For i greater than num_matches, the tgt and img_target are not matched. We leave all unmatched targets in the list, because some filtering operations need them. Ex the the filters for min_dist and bi_filter.

This function takes in the whole (randomly ordered) tgts and img_targets, as well as the (ordered) tgts_filtered and img_targets_filtered. Any tgts not in tgts_filtered are appended at the end, and similar for the img_targets

Parameters

  • tgts (list of dict) – List of all targets, order is not important. Each target is a dict that has, at a minimum, the keys ‘tvec’, and ‘target_type’. If isMatched is False, ‘norm’ is additionally needed as a key. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value.

  • tgts_filtered (list of dict) – List of filtered targets, order is important. tgts_filtered[i] is associated with img_targets_filtered[i] for i from 0 to num_matches. Each target is a dict that has, at a minimum, the keys ‘tvec’, and ‘target_type’. If isMatched is False, ‘norm’ is additionally needed as a key. ‘tvec’ has a numpy.ndarray (3, 1) representing the position of the target relative to the model origin for its associated value. ‘target_type’ has a string representing the type of target (most commonly ‘dot’) for its associated value.

  • img_targets (list of dict) – List of all matched image targets, order is not important. Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string).

  • img_targets_filtered (list of dict) – List of filtered image targets, order is important. img_targets_filtered[i] is associated with tgts_filtered[i] for i from 0 to num_matches. Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string).

subpixel_localize(img, tgts, img_targets, test_config, max_localize_delta=None)[source]

Find the sub-pixel localized position of the image targets

Find the location of the image target centers with sub-pixel accuracy. This method filters common bad localization solutions. I.e The localized position is too far initial guess to make sense, invalid optimizer solution (None flag), outside the cropped region or outside the image

Parameters

  • img (numpy.ndarray, shape (h, w)) – Numpy 2D array of the image

  • tgts (list) – Matched 3D targets. Each target should be a dict. The only strict requirement set by this function is tgts[i] is associated with img_targets[i]

  • img_targets (list) – Matched image targets. Each dict has, at a minimum, keys ‘center’, and ‘target_type’. ‘center’ has a value of the image target center location (tuple/np.ndarray of length 2 of floats) and ‘target_type’ has the key of the type of target (string). img_targets[i] is associated with tgts[i]

  • test_config (dict) – Processing parameters with, at a minimum, a key for each target type in targets and img_targets. The key is target_type + ‘_pad’. This is the padding around the img target center location to use to sub-pixel localize

  • max_localize_delta (float, optional) – The maximum allowable distance that subpixel_localize can change the img_target position. If None, the max allowable distance will be set to the padding distance minus 2.

Returns

  • targets (list) – Target positions (tgts) that have not been filtered out.

  • img_targets (list) – Refined (sub-pixel localized) target positions in the image.

Notes

targets[i] is associated with img_targets[i]. Return lists may not be the same length as input tgts and/or img_targets inputs (some targets/img_targets from the input may be rejected and thus are not included in the output)