SCDS/FtcRobotControllerSDK
7
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

First draft of TFOD entry page, updated for Freight Frenzy. Not linked yet, still private.

Westside Robotics
2021-10-21 18:02:36 -07:00
parent c8b119953f
commit 79371e198a
1 changed files with 64 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
Using-TensorFlow-in-Freight-Frenzy.md Normal file

@ -0,0 +1,64 @@
### What is TensorFlow?
FTC teams can use [TensorFlow Lite](https://www.tensorflow.org/lite/), a lightweight version of Google's [TensorFlow](https://www.tensorflow.org/) machine learning technology that is designed to run on mobile devices such as an Android smartphone. A _trained TensorFlow model_ was developed to recognize game elements for the 2021-2022 Freight Frenzy challenge.
_<p align="center">[[https://raw.githubusercontent.com/wiki/WestsideRobotics/FTC-training/Images/010-TFOD-Cube-Duck-crop-2.png]]<br/>
This season's TFOD model can recognize Freight elements._
<br/>
TensorFlow Object Detection (TFOD) has been integrated into the FTC control system software, to identify and track these game pieces during a match. The FTC software (SDK version 7.0) contains TFOD Sample Op Modes that can recognize the Freight elements Duck, Box (or Cube), and Cargo (or Ball).
Also, FTC teams can soon use a new tool to train their own TFOD models, to recognize their custom Team Shipping Element (TSE) and/or to improve recognition of Freight elements. This training could take into account certain conditions of distance, angle, lighting and background.
This new tool, FTC Machine Learning Toolchain, was [announced 10/7/2021](http://firsttechchallenge.blogspot.com/2021/10/new-machine-learning-tool-beta-testing.html) for upcoming beta testing by interested FTC teams.
### How Might a Team Use TensorFlow in Freight Frenzy?
For this season's challenge, during the pre-Match stage a single die is rolled and the field is randomized.
_<p align="center">[[https://raw.githubusercontent.com/wiki/WestsideRobotics/FTC-training/Images/020-TFOD-Barcode.png]]<br/>
Randomization_
<br/>
At the beginning of the match's Autonomous period, a robot can use TensorFlow to "look" at the **Barcode** area and determine whether the Duck or optional Team Shipping Element (TSE) is in position 1, 2 or 3. This indicates the preferred scoring level on the **Alliance Shipping Hub**. A bonus is available for using the TSE instead of a Duck.
_<p align="center">[[https://raw.githubusercontent.com/wiki/WestsideRobotics/FTC-training/Images/030-TFOD-levels.png]]<br/>
Alliance Shipping Hub_
<br/>
### Important Note on Phone Compatibility
TensorFlow Lite runs on Android 6.0 (Marshmallow) or higher, a requirement met by all currently allowed FTC devices. If you are a Blocks programmer using an older/disallowed Android device that is not running Marshmallow or higher, TFOD Blocks will automatically be missing from the Blocks toolbox or design palette.
### Sample Op Modes
The FTC software (SDK version 7.0 and higher) contains sample Blocks and Java op modes that demonstrate TensorFlow **recognition** of Freight elements Duck, Box (cube) and Cargo (ball). The sample op modes also show **where** in the camera's field of view a detected object is located.
Click on the following links to learn more about these sample Op Modes.
* [Blocks TensorFlow Object Detection Example](https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Blocks-Sample-Op-Mode-for-TensorFlow-Object-Detection)
* [Java TensorFlow Object Detection Example](https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Java-Sample-Op-Mode-for-TensorFlow-Object-Detection)
### Using a Custom Inference Model
Teams have the option of using a custom inference model with the FIRST Tech Challenge software. As in the past, some teams may want to use the [TensorFlow Object Detection API](https://github.com/tensorflow/models/tree/master/research/object_detection) to create an enhanced model of the Freight elements or TSE, or to create a custom model to detect other entirely different objects. Other teams might also want to use an available pre-trained model to build a robot that can detect common everyday objects (for demo or outreach purposes, for example).
The FTC software includes sample op modes (Blocks and Java versions) that demonstrate how to use a **custom inference model**:
* [Using a Custom TensorFlow Model with Blocks](https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Using-a-Custom-TensorFlow-Model-with-Blocks)
* [Using a Custom TensorFlow Model with Java](https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Using-a-Custom-TensorFlow-Model-with-Java)
These tutorials use examples from a previous FTC season (Skystone), but the process remains generally valid for Freight Frenzy.
As noted above, soon FTC teams will have a streamlined tool for training their own TFOD models. Watch for announcements regarding the FTC Machine Learning Toolchain, already scheduled for beta testing.
### Detecting Everyday Objects
You can use a pretrained TensorFlow Lite model to detect **everyday objects**, such as a clock, person, computer mouse, or cell phone. The following advanced tutorial shows how you can use a free, pretrained model to recognize numerous everyday objects.
* [Using a TensorFlow Pretrained Model to Detect Everyday Objects](https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Using-a-TensorFlow-Pretrained-Model-to-Detect-Everyday-Objects)
_<p align="center">[[https://raw.githubusercontent.com/wiki/FIRST-Tech-Challenge/FtcRobotController/images/Using-a-TensorFlow-Pretrained-Model-to-Detect-Everyday-Objects/tfliteDemo.png]]<br/>TensorFlow can recognize everyday objects._
<br/>
_<p align="right">updated 10/21/21</p>_