You are here: Start » Program Examples » Hand-Eye Calibration - On Filters

Hand-Eye Calibration - On Filters

Aim:

The goal of this application is to perform a complete calibration, which includes image distortion removal, perspective removal, and hand-eye calibration, using filters instead of a dedicated editor. This functionality is crucial when performing calibration directly from HMI is needed. A typical application of this process is hand-eye calibration in a production environment.

A robotics engineer should be able to select a number of appropriate grid point coordinates and for each of them provide the Real World Points (here called Calibration Points) obtained from the robot's interface for each of them. Based on this input data and an image of the calibration grid, the program should prepare a RectificationMap with all calibration data stored inside. Finally, after clicking Exit Application, the RectificationMap will be saved to disk. Additionally, after calibration, the program should compute a set of Test points, in which a corresponding Robot Point should be provided for each Grid point. The aforementioned requirements refer to the content and functionalities of the Calibration tab. The program should be implemented in the Main worker task. Moreover, the HMI should also contain an Inspection tab as well as a corresponding Inspection worker task, implementing the typical usage of the Rectification Map.

Input:

An image of the circular calibration grid.

An image with objects.

Output:

A rectified image with world coordinate points.

Hints:

Create a RectificationMap file in the Main Worker Tasks using filters from camera calibration filters. Use this file later in the Inspection Worker Task.

Labeling connections is explained in this article.

Solution (AVS):

Load calibration image and create the HMI:

  1. Go to the ACQUIRE section. Add the LoadImage filter and specify the images directory in the inFile.

  2. Add the Repeat filter.

  3. Create the HMI, which will be used in this application. Open the HMI layout designer as shown in the image below. You should now be able to edit the content and layout of the HMI window.

  4. Open the "Containers" category, then drag and drop the TabControl control into the HMI designer. Resize this control to fit the HMI panel. Change the tabs description in the Properties window and change tab names from tabPage1 and tabPage2 to Calibration and Inspection.

  5. In TabControl, switch to the Calibration tab. Drag and drop the View2DBox control from the "Indicators" category into the HMI window. Resize it to suit the design of your HMI and change the InitialSizeMode from OriginalSize to FitToWindow in the Properties window.

  6. Based on the image above, create sections next to the View2DBox control. To create sections, use the Panel controls from the "Container" category.

  7. Create boxes for numerical values using the NumericUpDown controls from the "Controls" category as shown in the image above. In the "Robot point coordinate" and the "Errors" sections, use the Properties window to set the DecimalPlaces to 1, the Maximum to 5000 and the Minimum to -5000.

  8. In the second "Test points" section ("Robot point coordinate") insert the TextBox controls from the "Controls" category.

  9. Label data as shown in the image above with the Label controls from the "Controls" category.

  10. Assign the following annotations to the NumericUpDown controls in the first "Calibration points" section ("Grid point coordinate"):

    • X:1 Y:1

    • X:32 Y:2

    • X:2 Y:22

    • X:26 Y:17

  11. Assign the following Real World values to the NumericUpDown controls in the second "Calibration points" section ("Robot point coordinate"):

    • X:133,6 Y:493,8

    • X:1064,8 Y:477,1

    • X:172,4 Y:-136,6

    • X:890,8 Y:24,3

  12. Assign the following annotations to the NumericUpDown controls in the first "Test points" section ("Grid point coordinate"):

    • X:22 Y:2

    • X:11 Y:13

    • X:17 Y:23

  13. Go to the "Inspection" tab in the TabControl. Drag and drop the View2DBox control from the "Indicators" category and resize it to suit your preference. Then, change the InitialSizeMode from OriginalSize to FitToWindow in the Properties window.

  14. Use the TextBox controls from the "Controls" category to create boxes for calculated values as shown in the image above.

  15. Label data, as shown in the image above, using the Label controls from the "Controls" category.

  16. Add the Impulse Button from the "Controls" category below the TabControl. Change the default text to Exit Application in the Properties window.

Now that the HMI for setting Real World Points is ready, we can proceed to creating the Main program for the calibration.

  1. Go to the PROCESS section, then add a Step Macrofilter to the main program and name it "GetRobotPointsFromHMI".

  2. Add the Formula inside the previously created macrofilter.

  3. Drag and drop the NumericUpDown control's outValue output to the Formula and create the Real type inputs. The Formula inputs should be named as shown on image below. Create connections between HMI and the Formula inputs so that:

    • outValue of grid point coordinate X is connected with the inRobotAnnotationX Formula's input,
    • outValue of grid point coordinate Y is connected with the inRobotAnnotationY Formula's input,
    • outValue of robot point coordinate X is connected with the inRobotPointX Formula's input,
    • outValue of robot point coordinate Y is connected with the inRobotPointY Formula's input.

  4. Use the code from the image above to create the Formula's outputs. Copy created Formula three times and connect each grid point coordinate and robot point coordinate from HMI with the Formula.

  5. Add the CreateArray filter and connect the outRobotGridPoint output from each Formula with the inValue of the CreateArray filter.

  6. Create a macrofilter output with the AnnotatedPoint2DArray type and name it outRobotGrid. Connect the outArray output of the CreateArray filter with the created output.

  7. Go back to Main. Add the DetectCalibrationGrid_Circles filter.

  8. Connect the outImage output from the LoadImage filter with the inImage input of the DetectCalibrationGrid_Circles. Set the inCircleRadius to 15.

  9. Make a new AnnotatedPoint2DArray type input in the created macrofilter and name it inImageGrid.

  10. Connect the outImageGrid output from the DetectCalibrationGrid_Circles filter with the newly created macrofilter input.

  11. Go to the "GetTestPointsFromHMI" macrofilter and create a new Formula inside.

  12. Create an AnnotatedPoint2DArray Formula's input, name it inImageGrid and connect it with the macrofilter's input. In this step, grid coordinates are combined into test annotation points with the Point2D data type, and then for each Grid Point, its Image equivalent is being located.

  13. Create 6 Real type Formula inputs and name them as shown in the image above.

  14. Connect Grid point coordinate (test points) from HMI to the Formula so that:

    • point's 4 outValue of Grid point coordinate X is connected with the inTPAnnotation0X Formula's input,
    • point's 4 outValue of Grid point coordinate Y is connected with the inTPAnnotation0Y Formula's input,
    • point's 5 outValue of Grid point coordinate X is connected with the inTPAnnotation1X Formula's input,
    • point's 5 outValue of Grid point coordinate Y is connected with the inTPAnnotation1X Formula's input,
    • point's 6 outValue of Grid point coordinate X is connected with the inTPAnnotation2X Formula's input,
    • point's 6 outValue of Grid point coordinate Y is connected with the inTPAnnotation2X Formula's input.

  15. Use the code from the image above to create Formula outputs. outTPAnnotationPoint outputs should have the Point2D data type, and outTPImagePoint outputs should have the Point2D? data type assigned to them.

  16. Add the MergeArrays filter to the current macrofilter and connect Formula's outTPImagePoint outputs with its inputs. This step allows the indices on HMI to stay synchronized.

  17. Create a macrofilter output with the Point2D?Array data type and name it outTestPoints. Connect the outJoinedArray output from the MergeArrays filter with the created macrofilter output.

  18. Go back to Main. Add a Step Macrofilter and name it "DrawPoints".

  19. Add the following inputs to the created macrofilter:

  20. Connect the created inputs with following this scheme:

    • Connect the outImage output from the LoadImage filter with the inImage input of the "DrawPoints" macrofilter,
    • Connect the outImageGrid output from the DetectCalibrationGrid_Circles filter with the inImageGrid input of the "DrawPoints" macrofilter,
    • Connect the outRobotGrid output from the "GetRobotPointsFromHMI" macrofilter with the inRobotGrid input of the "DrawPoints" macrofilter,
    • Connect the outTestPoints output from the "GetTestPointsFromHMI" macrofilter with the inTestPoints input of the "DrawPoints" macrofilter.

  21. Go to the "DrawPoints" macrofilter and create a new Formula inside.

  22. Create the AnnotatedPoint2DArray Formula's input, name it inImageGrid and connect it with the macrofilter's inImageGrid input.

  23. Create the AnnotatedPoint2DArray Formula's input, name it inRobotGrid and connect it with the macrofilter's inRobotGrid input.

  24. Use the code from the image above to create Formula's outputs. The outRobotPoints output should have the Point2D?Array data type assigned to it.

  25. Click on the Show/hide Ports > outRobotPoints > Count to display Formula's outRobotPoints.Count output.

  26. Add a Step Macrofilter and name it "DrawCalibrationPoints".

  27. Add the following inputs to the created macrofilter:

    • inImage input with the Image data type,
    • inColor input with the Pixel data type,
    • inImagePoints input with the Point2D?Array data type,
    • inOffset input with the Integer data type.

  28. Go to the "DrawCalibrationPoints" macrofilter.

  29. Add the ArrayIndices filter, connect macrofilter's inImagePoints input with the inArray input.

  30. Add the DrawPoints_SingleColor filter, connect its inputs using the following scheme:

  31. Add the Formula block and create its inputs using following scheme:

  32. Connect the created inputs using the following scheme:

    • Connect the outArray output from the ArrayIndices filter with the "inPointIndex" Formula input,
    • Connect the inImagePoints macrofilter's input with the "inPointLocation" Formula input,
    • Connect the inOffset macrofilter's input with the "inOffset" Formula input.

  33. Use the code from the image above to create a Formula's output. Labels output should have the StringLabel?Array data type assigned to it.

  34. Add the DrawStringLabels_SingleColor filter and create connections using the following scheme:

  35. Create the Image data type macrofilter output and name it outImage, connect the outImage with that output.

  36. Go to the "DrawPoints" macrofilter. Copy the "DrawCalibrationPoints" macrofilter to have 2 of them inside the "DrawPoints" macrofilter.

  37. Make connections for the first "DrawCalibrationPoints" macrofilter using the following scheme:

    • Connect the "DrawPoints" macrofilter's inImage input with the inImage input of the "DrawCalibrationPoints" macrofilter,
    • Set the inColor input to {0,000;128,000;0,000;0,000},
    • Connect the Formula's outRobotPoints output with the inImagePoints input,
    • Set inOffset to 0.

  38. Make connections for the second "DrawCalibrationPoints" macrofilter using the following scheme:

    • Connect the first "DrawCalibrationPoints" macrofilter's outImage output with the inImage input of the "DrawCalibrationPoints" macrofilter,
    • Set the inColor input to {0,000;0,000;255,000;0,000},
    • Connect the "DrawPoints" macrofilter's inTestPoints input with the inImagePoints input of the "DrawCalibrationPoints" macrofilter,
    • Connect the Formula's outRobotPoints.Count output with the inOffset input,
    • Connect the second "DrawCalibrationPoints" macrofilter's outImage output with the inImage input of the View2DBox control in the "Calibration" tab.

  39. Create an Image data type "DrawPoints" macrofilter output and name it outImage. Connect the outImage output of the second "DrawCalibrationPoints" macrofilter with this created output.

  40. Go to the Main macrofilter and add the CalibrateCamera_Pinhole filter.

  41. Make connections with the added filter:

  42. Add the CalibrateWorldPlane_Labeled filter.

  43. Make connections with the added filter:

Hand-Eye Calibration is now ready. Run the program and test the created calibration while also checking for RMS Errors. Create a rectification map and save it to use in the Inspection task.

  1. Go to the PROCESS section in the Main macrofilter and add a new "Step Macrofilter", name it "FindWorldCoordinatesForTestPoints".

  2. Add the following inputs to the created macrofilter:

  3. Connect the created inputs using the following scheme:

    • Connect the outTransform from the CalibrateWorldPlane_Labeled filter with the inTransform input of "FindWorldCoordinatesForTestPoints" macrofilter,
    • Connect the outTestPoints output from "GetTestPointsFromHMI" macrofilter with theinTestPoints input of the "FindWorldCoordinatesForTestPoints" macrofilter.

  4. Go to the "FindWorldCoordinatesForTestPoints" macrofilter.

  5. Add the ImagePointToWorldPlane filter and connect its inputs using the following scheme:

  6. Add the FormatPoint2DToString filter and connect its inPoint input with the outWorldPoints output of the ImagePointsToWorldPlane filter.

  7. Add the MergeDefault filter with the String data type variant.

  8. Connect the outString output from the FormatPoint2DToString filter with the inConditionalObject input of the MergeDefault filter.

  9. Add the GetArrayElements_OrNil filter and connect its inArray input with the outObject output of the MergeDefault filter.

  10. Connect the outValues from the GetArrayElements_OrNil to the corresponding TextBox controls placed in the HMI "Test Points" section, which are labeled as "Robot point coordinates".

  11. Go to the Main macrofilter and add the CreateRectificationMap_PixelUnits filter.

  12. Make connections with the added filter:

  13. Add the Exit filter and connect its inCondition input with the outValue of the ImpulseButton in the HMI, labeled as "Exit Application".

  14. Go to the FINALIZE section. Add the SaveObject filter and connect its inObject input with the outRectificationMap output of the CreateRectificationMap_PixelUnits filter.

  15. Set the inFile to RectificationMap.avdata.

Hand-Eye Calibration is ready and it was tested. Now it can be used in inspection:

  1. Create a new Worker Task from the Project Explorer window and name it as "Inspection".

  2. Go to the the INITIALIZE section in the newly created Worker Task macrofilter.

  3. Add the LoadObject filter with the RectificationMap data type variant. Set the inFile to the RectificationMap.avdata.

  4. Go to the ACQUIRE section and add the Repeat filter.

  5. Add the LoadImage filter and set the inFile to the Inspection.png.

  6. Add the RectifyImage filter and create connections with it:

  7. Go to the PROCESS section and add the LastNotNil filter with the Point2D? data type.

  8. Right click on the View2DBox indicator in the "Inspection" tab, then click on the Edit Ports and Events Visibility and check in outLeftClick. Connect the created output from the View2DBox to the inObjects input of the LastNotNil filter.

  9. Connect the outObject output from the LastNotNil filter with the inData1 input of the View2DBox indicator.

  10. Add the FormatPoint2DToString filter and connect its inPoint input with the outObject output of the LastNotNil filter.

  11. Connect the outString output from the FormatPoint2DToString filter with the TextBox control labeled as "Image Coordinates:".

  12. Add the ImagePointsToWorldPlane filter and connect its inputs using the following scheme:

  13. Add the FormatPoint2DToString filter and connect its inPoint input with the outWorldPoints output of the ImagePointsToWorldPlane filter.

  14. Connect the outString output from the FormatPoint2DToString with the inText input of the TextBox control labeled as "World Coordinates".

  15. Add the Exit filter and connect its inCondition input with the outValue output of the ImpulseButton labeled as "Exit Application".

Hand-Eye Calibration and inspection are now ready to use. Test the created program.

Used Filters

Icon Name Description
ArrayIndices Creates an array of element indices, i.e. {0, 1, 2, ..., N-1}, where N is the length of the input array.
CalibrateCamera_Pinhole Computes camera parameters which need to be calculated only once for each camera, and are a prerequisite for other calibration filters.
CalibrateWorldPlane_Labeled Image to world coordinates transformations, also useful for image rectification where exact bounds of output image are important, e.g. stitching.
CreateArray Creates an array from up to 8 individual objects.
CreateRectificationMap_PixelUnits Specification of fixed output geometry allows for constant environment even when recalibration is performed.
DetectCalibrationGrid_Circles Detects an arbitrary size symmetric circle pattern on the image.
DrawPoints_SingleColor Draws points on an image with a single color.
DrawStringLabels_SingleColor Draws strings (text) on an image with a single color.
Exit If the specified condition is true, exits the macrofilter loop.
FormatPoint2DToString Converts a 2D point to a string of format "(X, Y)".
GetArrayElements_OrNil Extracts up to 8 individual elements from an array or NIL for indices out of range.
ImagePointToWorldPlane Finds the world coordinates of image Point.
LastNotNil Returns the last value passed that existed.
LoadImage Loads a single image from a file.
LoadObject Loads an object from a file.
MergeArrays Concatenates conditional arrays one after another. Any array that is Nil becomes skipped.
MergeDefault Usually used to create a definite result for the special cases represented by the Nil value.
RectifyImage Applies a spatial map to distorted image transforming it to rectified image defined in world coordinates.
Repeat Put this filter into a task that should have a loop, but does not have any other loop generators (e.g. GrabImage).
SaveObject Saves an object to a file.

Further Readings