You are here: Start » Program Examples » Hand-Eye Calibration - On Filters

Hand-Eye Calibration - On Filters

Aim

The goal of this application is to perform a complete calibration, which includes image distortion removal, perspective removal and hand-eye calibration, using filters instead of dedicated editor. This functionality is crucial when calibration performed directly from HMI is needed. A typical application of this process is Hand-Eye calibration in the production environment.

A robotics engineer should be able to select a number of appropriate Grid point coordinates and for each of them provide the Real World Points (here called Calibration points) obtained from the robot's interface. Based on this input data and image of calibration grid, the program should prepare a RectificationMap with all calibration data stored inside. Finally after clicking Exit Application the RectificationMap will be saved to disk. Additionally, after calibration, the program should compute a set of Test points, in which for each Grid point a corresponding Robot Point should be provided. The above mentioned requirements refer to the content and functionalities of the Calibration tab. The program should be implemented in Main worker task. Moreover, the HMI should also contain an Inspection tab as well as corresponding Inspection worker task, implementing a typical usage of the Rectification Map.

Input

An image of the circle calibration grid.

Image with objects.

Output

Rectified image with world coordinates point.

Hints

Create a RectificationMap file in the Main Worker Tasks using filters from camera calibration filters. Use this file later in the Inspection Worker Task.

Labeling connections is explained in this article.

Solution (AVS)

Load calibration image and create the HMI:

  1. Go to the ACQUIRE section. Add a LoadImage filter and specify the images directory in inFile.

  2. Add a Loop filter.

  3. Create the HMI, which will be used in this application. Open the HMI layout designer as shown on image below. You should now be able to edit the content and layout of the HMI window.

  4. Open the "Containers" category, next drag and drop the TabControl control into the HMI designer. Resize this control to the size of the HMI panel. Change tabs description in the Properties window and change tab names from "tabPage1" and "tabPage2" to "Calibration" and "Inspection".

  5. In TabControl switch to the "Calibration" tab. Drag and drop View2DBox control from the "Indicators" category into the HMI window. Resize its size to suit the design of your HMI and change the InitialSizeMode from OriginalSize to FitToWindow in the Properties window.

  6. Basing on the image above, create sections next to View2DBox control. To create sections use Panel controls from "Container" category.

  7. Create boxes for numerical values using NumericUpDown controls from "Controls" category as shown in the image above. In case of "Robot point coordinate" and "Errors" sections use Properties window to set DecimalPlaces to 1, Maximum to 5000 and Minimum to -5000.

  8. In the second "Test points" section ("Robot point coordinate") insert TextBox controls from the "Controls" category.

  9. Label data as shown in the image above with Label controls from "Controls" category.

  10. Assign the following annotations to the NumericUpDown controls in the first "Calibration points" section ("Grid point coordinate"):

    • X:1 Y:1

    • X:32 Y:2

    • X:2 Y:22

    • X:26 Y:17

  11. Assign the following Real World values to the NumericUpDown controls in the second "Calibration points" section ("Robot point coordinate"):

    • X:133,6 Y:493,8

    • X:1064,8 Y:477,1

    • X:172,4 Y:-136,6

    • X:890,8 Y:24,3

  12. Assign the following annotations to the NumericUpDown controls in the first "Test points" section ("Grid point coordinate"):

    • X:22 Y:2

    • X:11 Y:13

    • X:17 Y:23

  13. Go to the "Inspection" tab in the TabControl. Drag and drop View2DBox control from the "Indicators" category and resize this control to suit your preference. Next, change InitialSizeMode from OriginalSize to FitToWindow in the Properties window.

  14. Use TextBox controls from "Controls" category to create boxes for calculated values as shown in the image above.

  15. Label data, as shown in the image above, using Label controls from "Controls" category.

  16. Add an Impulse Button from "Controls" category below the TabControl. Change default text to Exit Application in the Properties window.

Now that the HMI for setting Real World Points is ready, we can proceed to creating Main program for the calibration.

  1. Go to the PROCESS section, next add Step Macrofilter to the main program and name it "GetRobotPointsFromHMI".

  2. Add a Formula inside the previously created macrofilter.

  3. Drag and drop NumericUpDown control's outValue output to the Formula and create Real type inputs. Formula inputs should be named as shown on image below. Create connections between HMI and the Formula inputs so that:

    • outValue of grid point coordinate X is connected with inRobotAnnotationX Formula's input,
    • outValue of grid point coordinate Y is connected with inRobotAnnotationY Formula's input,
    • outValue of robot point coordinate X is connected with inRobotPointX Formula's input,
    • outValue of robot point coordinate Y is connected with inRobotPointY Formula's input.

  4. Use the code from image above to create Formula's outputs. Copy created Formula three times and connect each grid point coordinate and robot point coordinate from HMI with the Formula.

  5. Add CreateArray filter and connect outRobotGridPoint output from each Formula with the inValue of CreateArray filter.

  6. Create a macrofilter output with AnnotatedPoint2DArray type and name it outRobotGrid. Connect outArray output of CreateArray filter with the created output.

  7. Go back to Main. Add DetectCalibrationGrid_Circles filter.

  8. Connect outImage output from LoadImage filter with the inImage input of DetectCalibrationGrid_Circles. Set inCircleRadius to 15.

  9. Make a new AnnotatedPoint2DArray type input in the created macrofilter and name it inImageGrid.

  10. Connect outImageGrid output from DetectCalibrationGrid_Circles filter with the newly created macrofilter input.

  11. Go to the "GetTestPointsFromHMI" macrofilter and create a new Formula inside.

  12. Create an AnnotatedPoint2DArray Formula's input, name it inImageGrid and connect it with the macrofilter's input. In this step grid coordinates are combined into test annotation points with Point2D data type and then for each Grid Point its Image equivalent is being located.

  13. Create 6 Real type Formula inputs and name them as shown in the above image.

  14. Connect Grid point coordinate (test points) from HMI to the Formula so that:

    • point's 4 outValue of Grid point coordinate X is connected with inTPAnnotation0X Formula's input,
    • point's 4 outValue of Grid point coordinate Y is connected with inTPAnnotation0Y Formula's input,
    • point's 5 outValue of Grid point coordinate X is connected with inTPAnnotation1X Formula's input,
    • point's 5 outValue of Grid point coordinate Y is connected with inTPAnnotation1X Formula's input,
    • point's 6 outValue of Grid point coordinate X is connected with inTPAnnotation2X Formula's input,
    • point's 6 outValue of Grid point coordinate Y is connected with inTPAnnotation2X Formula's input.

  15. Use the code from image above to create Formula outputs. outTPAnnotationPoint outputs should have Point2D data type and outTPImagePoint outputs should have Point2D? data type assigned to them.

  16. Add a MergeArrays filter to the current macrofilter and connect Formula's outTPImagePoint outputs with its inputs. This step allows to keep the indices on HMI synchronized.

  17. Create a macrofilter output with Point2D?Array data type and name it outTestPoints. Connect outJoinedArray output from MergeArrays filter with the created macrofilter output.

  18. Go back to Main. Add a Step Macrofilter and name it "DrawPoints".

  19. Add following inputs to the created macrofilter:

  20. Connect created inputs with following this scheme:

    • Connect outImage output from LoadImage filter with the inImage input of "DrawPoints" macrofilter,
    • Connect outImageGrid output from DetectCalibrationGrid_Circles filter with the inImageGrid input of "DrawPoints" macrofilter,
    • Connect outRobotGrid output from "GetRobotPointsFromHMI" macrofilter with the inRobotGrid input of "DrawPoints" macrofilter,
    • Connect outTestPoints output from "GetTestPointsFromHMI" macrofilter with the inTestPoints input of "DrawPoints" macrofilter.

  21. Go to "DrawPoints" macrofilter and create a new Formula inside.

  22. Create AnnotatedPoint2DArray Formula's input, name it inImageGrid and connect it with the macrofilter's inImageGrid input.

  23. Create AnnotatedPoint2DArray Formula's input, name it inRobotGrid and connect it with the macrofilter's inRobotGrid input.

  24. Use the code from image above to create Formula's outputs. outRobotPoints output should have Point2D?Array data type assigned to it.

  25. Click on the Show/hide Ports > outRobotPoints > Count to display Formula's outRobotPoints.Count output.

  26. Add a Step Macrofilter and name it "DrawCalibrationPoints".

  27. Add following inputs to the created macrofilter:

    • inImage input with Image data type,
    • inColor input with Pixel data type,
    • inImagePoints input with Point2D?Array data type,
    • inOffset input with Integer data type.

  28. Go to the "DrawCalibrationPoints" macrofilter.

  29. Add an ArrayIndices filter, connect macrofilter's inImagePoints input with the inArray input.

  30. Add a DrawPoints_SingleColor filter, connect its inputs using the following scheme:

  31. Add a Formula block and create its inputs using following scheme:

  32. Connect created inputs using following scheme:

    • Connect outArray output from ArrayIndices filter with the "inPointIndex" Formula input,
    • Connect inImagePoints macrofilter's input with the "inPointLocation" Formula input,
    • Connect inOffset macrofilter's input with the "inOffset" Formula input.

  33. Use the code from the image above to create a Formula's output. Labels output should have StringLabel?Array data type assigned to it.

  34. Add a DrawStringLabels_SingleColor filter and create connections using following scheme:

  35. Create a Image data type macrofilter output and name it outImage, connect outImage with that output.

  36. Go to the "DrawPoints" macrofilter. Copy the "DrawCalibrationPoints" macrofilter to have 2 of them inside the "DrawPoints" macrofilter.

  37. Make connections for the first "DrawCalibrationPoints" macrofilter using following scheme:

    • Connect "DrawPoints" macrofilter's inImage input with the inImage input of "DrawCalibrationPoints" macrofilter,
    • Set inColor input to {0,000;128,000;0,000;0,000},
    • Connect Formula's outRobotPoints output with the inImagePoints input,
    • Set inOffset to 0.

  38. Make connections for the second "DrawCalibrationPoints" macrofilter using following scheme:

    • Connect (first) "DrawCalibrationPoints" macrofilter's outImage output with the inImage input of "DrawCalibrationPoints" macrofilter,
    • Set inColor input to {0,000;0,000;255,000;0,000},
    • Connect "DrawPoints" macrofilter's inTestPoints input with the inImagePoints input of "DrawCalibrationPoints" macrofilter,
    • Connect Formula's outRobotPoints.Count output with the inOffset input,
    • Connect (second) "DrawCalibrationPoints" macrofilter's outImage output with the inImage input of View2DBox control in the "Calibration" tab.

  39. Create a Image data type "DrawPoints" macrofilter output and name it outImage. Connect outImage output of the (second) "DrawCalibrationPoints" macrofilter with this created output.

  40. Go to the Main macrofilter and add a CalibrateCamera_Pinhole filter.

  41. Make connections with the added filter:

  42. Add a CalibrateWorldPlane_Labeled filter.

  43. Make connections with the added filter:

Hand-Eye Calibration is now ready. Run the program and test the created calibration while also checking for RMS Errors. Create a rectification map and save it in order to use it in the Inspection task.

  1. Go to the PROCESS section in Main macrofilter and add a new "Step Macrofilter", name it "FindWorldCoordinatesForTestPoints".

  2. Add following inputs to the created macrofilter:

  3. Connect created inputs using following scheme:

    • Connect outTransform from CalibrateWorldPlane_Labeled filter with the inTransform input of "FindWorldCoordinatesForTestPoints" macrofilter,
    • Connect outTestPoints output from "GetTestPointsFromHMI" macrofilter with theinTestPoints input of "FindWorldCoordinatesForTestPoints" macrofilter.

  4. Go to the "FindWorldCoordinatesForTestPoints" macrofilter.

  5. Add a ImagePointToWorldPlane filter and connect its inputs using following scheme:

  6. Add a FormatPoint2DToString filter and connect its inPoint input with the outWorldPoints output of ImagePointsToWorldPlane filter.

  7. Add a MergeDefault filter with String data type variant.

  8. Connect outString output from FormatPoint2DToString filter with the inConditionalObject input of MergeDefault filter.

  9. Add a GetArrayElements_OrNil filter and connect its inArray input with the outObject output of MergeDefault filter.

  10. Connect outValues from GetArrayElements_OrNil to the corresponding TextBox controls placed in the HMI "Test Points" section, which are labeled as "Robot point coordinates".

  11. Go to the Main macrofilter and add a CreateRectificationMap_PixelUnits filter.

  12. Make connections with the added filter:

  13. Add an Exit filter and connect its inCondition input with the outValue of the ImpulseButton in the HMI, labeled as "Exit Application".

  14. Go to the FINALIZE section. Add a SaveObject filter and connect its inObject input with the outRectificationMap output of CreateRectificationMap_PixelUnits filter.

  15. Set inFile to RectificationMap.avdata.

Hand-Eye Calibration is ready and it was tested. Now it can be used in inspection:

  1. Create new Worker Task from the Project Explorer window and name it as "Inspection".

  2. Go to the the INITIALIZE section in the new created Worker Task macrofilter.

  3. Add a LoadObject filter with the RectificationMap data type variant. Set inFile to the RectificationMap.avdata.

  4. Go to the ACQUIRE section and add a Loop filter.

  5. Add a LoadImage filter and set inFile to the Inspection.png.

  6. Add a RectifyImage filter and create connections with it:

  7. Go to the PROCESS section and add a LastNotNil filter with Point2D? data type.

  8. Right click on the View2DBox indicator in the "Inspection" tab, then click on the Edit Ports and Events Visibility and check in outLeftClick. Connect created output from the View2DBox to inObjects input of LastNotNil filter.

  9. Connect outObject output from the LastNotNil filter with the inData1 input of the View2DBox indicator.

  10. Add a FormatPoint2DToString filter and connect its inPoint input with the outObject output of the LastNotNil filter.

  11. Connect outString output from FormatPoint2DToString filter with the TextBox control labeled as "Image Coordinates:".

  12. Add a ImagePointsToWorldPlane filter and connect its inputs using following scheme:

  13. Add a FormatPoint2DToString filter and connect its inPoint input with the outWorldPoints output of ImagePointsToWorldPlane filter.

  14. Connect outString output from FormatPoint2DToString with the inText input of TextBox control labeled as "World Coordinates".

  15. Add an Exit filter and connect its inCondition input with the outValue output of ImpulseButton labeled as "Exit Application".

Hand-Eye Calibration and inspection are now ready to use. Test the created program.

Macrofilter Main

Used Filters

Icon Name Description
LoadImage Loads a single image from a file.
ArrayIndices Creates an array of element indices, i.e. {0, 1, 2, ..., N-1}, where N is the length of the input array.
RectifyImage Image undistortion, perspective removal.
SaveObject Saves an object to a file.
DrawPoints_SingleColor Draws points on an image with a single color.
GetArrayElements_OrNil Extracts up to 8 individual elements from an array or NIL for indices out of range.
MergeArrays Concatenates conditional arrays one after another. Any array that is Nil becomes skipped.
CalibrateWorldPlane_Labeled Image to world coordinates transformations.
MergeDefault Usually used to create a definite result for the special cases represented by the Nil value.
DrawStringLabels_SingleColor Draws strings (text) on an image with a single color.
DetectCalibrationGrid_Circles Camera calibration, image to world coordinates transformations.
LoadObject Loads an object from a file.
Exit If the specified condition is true, exits the macrofilter loop.
CreateRectificationMap_PixelUnits Image undistortion, perspective removal.
CreateArray Creates an array from up to 8 individual objects.
Repeat Put this filter into a task that should have a loop, but does not have any other loop generators (e.g. GrabImage).
FormatPoint2DToString Converts a 2D point to a string of format "(X, Y)".
ImagePointToWorldPlane Undistortion, image to world coordinate transformation.
LastNotNil Returns the last value passed that existed.
CalibrateCamera_Pinhole Camera calibration, image to world coordinates transformations.

Further Readings