Updated Blocks Sample TensorFlow Object Detection Op Mode (markdown)

FTC Engineering
2020-10-23 21:45:37 -04:00
parent 62366eb004
commit 114ab17f37

@ -4,20 +4,20 @@ You can use the sample "ConceptTensorFlowObjectDetection" as a template to creat
* If you are using a REV Control Hub with an externally connected webcam as your Robot Controller, select "ConceptTensorFlowObjectDetection" as the sample op mode from the dropdown list on the Create New Op Mode dialog box. * If you are using a REV Control Hub with an externally connected webcam as your Robot Controller, select "ConceptTensorFlowObjectDetection" as the sample op mode from the dropdown list on the Create New Op Mode dialog box.
* If you are using an Android smartphone as your Robot Controller, select "ConceptTensorFlowObjectDetection" as the sample op mode from the dropdown list on the Create New Op Mode dialog box. * If you are using an Android smartphone as your Robot Controller, select "ConceptTensorFlowObjectDetection" as the sample op mode from the dropdown list on the Create New Op Mode dialog box.
Press "OK" to create the new Op Mode. Press "OK" to create the new op mode.
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksConceptTensorFlowWebcam.png]]<br/>Create an Op Mode with ConceptTensorFlowObjectDetection as its template.<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksConceptTensorFlowWebcam.png]]<br/>Create an Op Mode with ConceptTensorFlowObjectDetection as its template.<p>
Your new Op Mode should appear in the editing pane of the Blocks Development Tool screen. Your new op mode should appear in the editing pane of the Blocks Development Tool screen.
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksMyExample.png]]<br/>Your newly created Op Mode will have the ConceptTensorFlowObjectDetection blocks included.<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksMyExample.png]]<br/>Your newly created op mode will have the ConceptTensorFlowObjectDetection blocks included.<p>
### Initializing the System ### Initializing the System
Let's take a look at the initial blocks in the Op Mode. The first block in the op mode (excluding the comment blocks) initializes the Vuforia library on the Android Robot Controller. This is needed because the TensorFlow Lite library will receive image data from the Vuforia library. Also, in the screenshot below, the Vuforia system will use an externally connected webcam named "Webcam 1" (which should match the camera name in your robot's configuration file). Let's take a look at the initial blocks in the op mode. The first block in the op mode (excluding the comment blocks) initializes the Vuforia library on the Android Robot Controller. This is needed because the TensorFlow Lite library will receive image data from the Vuforia library. Also, in the screenshot below, the Vuforia system will use an externally connected webcam named "Webcam 1" (which should match the camera name in your robot's configuration file).
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksInit.png]]<br/>Initialize the Vuforia and TensorFlow libraries.<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksInit.png]]<br/>Initialize the Vuforia and TensorFlow libraries.<p>
In the screenshot shown above, the sample Op Mode disables the camera monitoring window on the Robot Controller. If you are using a REV Robotics Control Hub as your Robot Controller (which lacks a touch screen) you normally want to disable the camera monitoring window and use the Camera Stream function instead to view the output of the webcam. If you are using an Android phone as your Robot Controller, however, you can enable the camera monitoring window so you can see the camera output on the Robot Controller's touch screen. In the screenshot shown above, the sample op mode disables the camera monitoring window on the Robot Controller. If you are using a REV Robotics Control Hub as your Robot Controller (which lacks a touch screen) you normally want to disable the camera monitoring window and use the Camera Stream function instead to view the output of the webcam. If you are using an Android phone as your Robot Controller, however, you can enable the camera monitoring window so you can see the camera output on the Robot Controller's touch screen.
You can initialize both the Vuforia and the TensorFlow libraries in the same op mode. This is useful, for example, if you would like to use the TensorFlow library to determine the ring stack and then use the Vuforia library to help the robot autonomously navigate on the game field to navigate to the appropriate target zone from its starting position. You can initialize both the Vuforia and the TensorFlow libraries in the same op mode. This is useful, for example, if you would like to use the TensorFlow library to determine the ring stack and then use the Vuforia library to help the robot autonomously navigate on the game field to navigate to the appropriate target zone from its starting position.
@ -27,7 +27,7 @@ Also note that the Minimum Confidence level is set to 70%. This means that the
If a camera monitor window is enabled for the TensorFlow library, then the confidence level for a detected target will be displayed near the bounding box of the identified object (when the object tracker is enabled). For example, a value of "0.92" indicates a 92% confidence that the object has been identified correctly. If a camera monitor window is enabled for the TensorFlow library, then the confidence level for a detected target will be displayed near the bounding box of the identified object (when the object tracker is enabled). For example, a value of "0.92" indicates a 92% confidence that the object has been identified correctly.
When an object is identified by the TensorFlow library, the Op Mode can read the "Left", "Right", "Top" and "Bottom" values associated with the detected object. These values correspond to the location of the left, right, top and bottom boundaries of the detection box for that object. These values are in pixel coordinates of the image from the camera. When an object is identified by the TensorFlow library, the op mode can read the "Left", "Right", "Top" and "Bottom" values associated with the detected object. These values correspond to the location of the left, right, top and bottom boundaries of the detection box for that object. These values are in pixel coordinates of the image from the camera.
The origin of the coordinate system is in the upper left-hand corner of the image. The horizontal (x) coordinate value increases as you move from the left to the right of the image. The vertical (y) coordinate value increases as you move from the top to the bottom of the image. The origin of the coordinate system is in the upper left-hand corner of the image. The horizontal (x) coordinate value increases as you move from the left to the right of the image. The vertical (y) coordinate value increases as you move from the top to the bottom of the image.
@ -36,7 +36,7 @@ The origin of the coordinate system is in the upper left-hand corner of the imag
In the landscape image above, the approximate coordinate values for the Left, Top, Right, and Bottom boundaries are 455, 191, 808, and 547 respectively (pixel coordinates). The width and height for the landscape image above is 1280 and 720 respectively. In the landscape image above, the approximate coordinate values for the Left, Top, Right, and Bottom boundaries are 455, 191, 808, and 547 respectively (pixel coordinates). The width and height for the landscape image above is 1280 and 720 respectively.
### Activating TensorFlow ### Activating TensorFlow
In this example, the Op Mode activates the TensorFlow object detector before waiting for the start command from the Driver Station. This is done so that the user can access the "Camera Stream" preview from the Driver Station menu while it waits for the start command. Also note that in this example, the Op Mode does not activate the Vuforia tracking feature, it only activates TensorFlow object detection. If you want to incorporate Vuforia image detection and tracking you will also need to activate (and later deactivate when you are done) the Vuforia tracking feature. In this example, the Op Mode activates the TensorFlow object detector before waiting for the start command from the Driver Station. This is done so that the user can access the "Camera Stream" preview from the Driver Station menu while it waits for the start command. Also note that in this example, the op mode does not activate the Vuforia tracking feature, it only activates TensorFlow object detection. If you want to incorporate Vuforia image detection and tracking you will also need to activate (and later deactivate when you are done) the Vuforia tracking feature.
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksTensorFlowActivate.png]]<br/>Activate TensorFlow.<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/blocksTensorFlowActivate.png]]<br/>Activate TensorFlow.<p>
@ -52,11 +52,11 @@ If a zoom factor has been set, then the Camera Stream preview on the Driver Stat
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/magnifiedArea.png]]<br/>Camera stream preview indicating magnified area (at a distance of about 4 feet or 1.2 meters).<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/magnifiedArea.png]]<br/>Camera stream preview indicating magnified area (at a distance of about 4 feet or 1.2 meters).<p>
### Iterating and Processing List of Recognized Objects ### Iterating and Processing List of Recognized Objects
The Op Mode will then iterate until a Stop command is received and it will check with the object detector to see how many objects it recognizes in its field of view. In the screenshot below, the variable "recognitions" is set to a list of objects that were recognized using the TensorFlow technology. The op mode will then iterate until a Stop command is received. At the beginning of each iteration, the op mode will check with the object detector to see how many objects it recognizes in its field of view. In the screenshot below, the variable "recognitions" is set to a list of objects that were recognized using the TensorFlow technology.
<p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/whileLoop.png]]<br/>The op mode gets a list of recognized objects with each iteration of the while loop.<p> <p align="center">[[/images/Blocks-Sample-TensorFlow-Object-Detection-Op-Mode/whileLoop.png]]<br/>The op mode gets a list of recognized objects with each iteration of the while loop.<p>
If the number of detected objects equals three (i.e., if three Minerals are in the camera's field of view) then the Op Mode will loop through the list of recognized objects and look at the label for each recognized object. If an object has a label that indicates it's a Gold Mineral, then the "Left" coordinate value of its bounding box will be assigned to the variable "goldMineralX". Similarly, if the other two objects in the list have labels that indicate that they are Silver Minerals, then their Left coordinate values will be assigned to the variables "silverMineral1X" and "silverMineral2X". If the number of detected objects equals three (i.e., if three Minerals are in the camera's field of view) then the op mode will loop through the list of recognized objects and look at the label for each recognized object. If an object has a label that indicates it's a Gold Mineral, then the "Left" coordinate value of its bounding box will be assigned to the variable "goldMineralX". Similarly, if the other two objects in the list have labels that indicate that they are Silver Minerals, then their Left coordinate values will be assigned to the variables "silverMineral1X" and "silverMineral2X".
<p align="center"><img src="https://github.com/FIRST-Tech-Challenge/WikiSupport/blob/master/tensorflow/images/blocksFindingLeftValues.png" width="600"><br/>Keep track of the Left coordinate values of the detected objects.<p> <p align="center"><img src="https://github.com/FIRST-Tech-Challenge/WikiSupport/blob/master/tensorflow/images/blocksFindingLeftValues.png" width="600"><br/>Keep track of the Left coordinate values of the detected objects.<p>