Department |
People |
Curriculum |
Research |
Resources |
Faculty Current Students Alumnae |
|
|
||||||||||
|
|||||||||||
 This page refers to the vision system prior to Pyro 2.2.0. For the newer version please see PyroModuleVisionSystem.
Using a Pioneer with a Camera
The camera should be plugged into the Pioneer, and the green light on the front of the camera should be on. If the light isn't on, check to make sure the power switch on the back of the camera is on.
Pyro Vision FunctionsThe following functions can be used in Pyro with the Video for Linux (V4L) frame grabber on the Pioneers. They are defined in /usr/local/pyrobot/camera/v4l/__init.py__.
To start the V4L service:
self.getRobot().startService("V4LCamera")
self.camera = self.getRobot().getService("V4LCamera")
self.camera.makeWindow()
The image that the camera captures is dumped to a shared memory buffer. This buffer can be accessed by both Pyro and C. For speed reasons, the vision processing is done in C. All the changes to the image are done by manipulating this shared memory buffer, which is then read by pyro to display the newly modified image. The frame grabber grabs images in a different thread then what the program is running in, so it updates itself reguardless of where the program is executing. This can cause wierd things to happen. Lets say you run a color filter to keep only red items in the image, and halfway through the filter process, the framegrabber updates the image. This would cause the filtering that has just been done to be over written, and the filter will continue to filter the bottom half of the image. Therefore, you would be left with an image with the top half not filtered and the bottom half being filtered for red. To stop this, the camera has a flag called lockCamera. If this flag is set, it will stop the framegrabber from being able to update the shared memory so that all the filtering, blobbing, etc can be done on the same image. Here is an example:
self.camera.lockCamera = 1 #lock the buffer
self.camera.superColor("red",2)
self.camera.maxBlobs(2, 1, 255, "mass")
self.camera.lockCamera = 0 #unlock the buffer
Once you are done manipulating the image and have the information you need, setting lockCamera to 0 will allow the framegrabber to update the images again. Another flag that the camera has is the sleepFlag. If set, the sleepFlag will cause the program to sleep, for what sleepTime is set to, after filtering processes have been completed. This allows the effects to be seen in the video window. This is mostly for testing purposes to make sure that the filter does what you want it to do. Here is an example:
self.camera.sleepTime=2.0 #sleep for 2 seconds after filters
self.camera.sleepFlag = 1 #turn sleep flag on
self.camera.lockCamera = 1 #lock the buffer
self.camera.superColor("red",0)
self.camera.maxBlobs(0, 1, 255, "mass")
self.camera.lockCamera = 0 #unlock the buffer
Recall from Introduction to Computer Vision that the processing for the filter and blob functions requires you to specify a channel (0 for red, 1 for green, and 2 for blue) to which the results will be written. All of the filtering functions default to the green channel, but you have to specify a channel for the blob function. See descriptions of the individual filters below.
camera.colorFilterHiLowThis function filters for color values within specified ranges. Points that have values between the low and high ranges specified for red, green and blue will be marked in the buffer as containing the color. After using this function, you would probably call maxBlob. Parameters:
Return Values:
Example calls:
self.camera.colorFilterHiLow(70,255,30,60,30,60)
self.camera.colorFilterHiLow(0,255,30,60,30,60)  
camera.colorFilterOneTolThis function filters the image for a specific color with one tolerance being used for each of the RGB values. Parameters:
Return Values:
Example calls:
self.camera.colorFilterOneTol(215,215,70,40,2)
camera.colorFilterThreeTolThis function also filters the image for a specific color, but the user has the option of setting a specific individual tolerance for R,G and B. Parameters:
Return Values:
Example calls:
self.camera.colorFilterThreeTol(255,255,0,30,30,200,0)
camera.superColorLets say you wanted to filter for red. If you just look at the red component and filter and color that has a red value over 180, for instance, then any white pixel will be filtered as a red pixel, because white has a red value of 255. What super color does, is subtracts out the secondary colors from the main color. So if you want to filter on red, the green and blue values get subtracted from the red value resulting in an overall value from 0-255. Super Color is only for Red, Green and Blue (the three primary colors). The return value is between 0-255 so you have the ability to choose how red, green, or blue you want to filter for.
Parameters:
Return Values:
Example calls:
self.camera.superColor("red",0)
self.camera.superColor("red",0,1)
camera.trainColorThe train color function takes a histogram of the colors in a 50x50 pixel box in the center of the image. It takes the average color of the peak and that is the color that it returns. This function is useful for determining what overall RGB value an object has. It is also good for passing into a filter function to be able to filter for the color that was just trained on. It is good practice to call this function a few times, then average the resulting values. In practice, we have found that calling trainColor five times and averaging the return values provides good results. Parameters:
Return Values:
Example call(s).
self.camera.trainColor() self.camera.colorFilterOneTol(self.camera.histogram.red,self.camera.histogram.green, self.camera.histogram.blue, 30)
camera.gaussianBlurThe gaussianBlur function blurs the image by changing the current pixel's values by weighting the values of the pixels around it. The kernel size is 3, and cannot be changed. Parameters:
Return Values:
Example call:
self.camera.gaussianBlur()
camera.meanBlurThis function blurs the image by changing the color of the pixel being looked at to the mean value of the pixels surrounding it. The number of surrounding pixels being looked at is defined by the kernel parameter. If kernel is 3, then the pixel being looked at is the center of a 3x3 box, shown in the diagram.
Parameter:
Return Values:
Example calls:
self.camera.meanBlur()
self.camera.meanBlur(9)
camera.medianBlurThis function blurs the image by changing the color of the pixel being looked at to the median value of the pixels surrounding it. The number of surrounding pixels being looked at is defined by the kernel parameter. If kernel is 3, then the pixel being looked at is the center of a 3x3 box, shown in the diagram.
Parameter:
Return Values:
Example calls:
self.camera.medianBlur(5)
self.camera.medianBlur(9)
self.camera.medianBlur(11)
camera.maxBlobsThis function finds upto the five largest blobs of an image by either mass or area. The color channel and the high and low values are passed in. MaxBlobs searches the image and keeps track of the all the blobs in the image. It then sets the upper left and lower right coordinates, as well as the mass of these one to five blobs into a list named blob. Optionally, white boxes are drawn around these blobs for identification purposes during testing. Parameter:
Return Values:
Example calls:
self.camera.maxBlobs(1,1,255,"mass")
self.camera.maxBlob(2,1,255,"mass")
Edge DetectionThis function uses the sobel method to detect the edges of an image.
self.camera.edgeDetection()
camera.saveImageThis function will save the image that is in the processing buffer to a ppm file. Seeing as how ppm files are quite large, you might want to convert the ppm files to a different image format. The best way to do this is to use an image editor (such as xv) and use Save As to save the image in a different format. Another way is to use xview with the following command:
xview -dump jpeg yourfile.jpeg yourfile.ppm
The unix command ppmtogif can also be used to convert your ppm file to the gif format, however, if you try to use ppmtogif to convert your file you may get an error that there are too many colors. You may try the following, but the colors in the resulting image may not match the colors of the original image in some spots, which is why we recommend using an image editing program.
ppmquant 256 yourfile.ppm | ppmtogif > yourfile.gif
Parameters:
Return Values:
Example call:
self.camera.saveImage("myPicture.ppm")
Examples1. This is a demonstration of the robot blurring an image, training on the color in the center of the image, filtering for that color, and then finding the max blob.
from pyrobot.brain import Brain
from pyrobot.brain.behaviors.core import * # Stop
import time
class CameraTest(Brain):
def __init__(self, name, robot):
Brain.__init__(self, name, robot)
print "initializing camera ..."
self.getRobot().startService("V4LCamera")
self.camera = self.getRobot().getService("V4LCamera")
self.camera.makeWindow()
self.getRobot().startService("ptz")
self.ptz = self.getRobot().getService("ptz")
self.ptz.panTilt(0, -10) #look down slightly
print "done initializing camera"
def step(self):
self.camera.lockCamera = 1
self.camera.sleepFlag = 1
self.camera.sleepTime = 3
print "Blurring Image"
self.camera.gaussianBlur()
print "Training on a color"
self.camera.trainColor()
print "Trained on color r:", self.camera.histogram.red,
print " g:", self.camera.histogram.green,
print " b:", self.camera.histogram.blue
print "Filtering Image"
self.camera.colorFilterOneTol(self.camera.histogram.red,self.camera.histogram.green,self.camera.histogram.blue )
print "Blobbing Image"
self.camera.maxBlobs(1, 1, 255, "mass", 1)
self.camera.lockCamera = 0
sleep(0.5)
print "\n"
def INIT(robot):
return CameraTest('CameraTest', robot)
2. In this example, the robot uses superColor to find red, blobs the image, then tracks the largest blob by moving the camera using the PTZ service.
from pyrobot.brain import Brain
from pyrobot.brain.behaviors.core import * # Stop
import time
class CameraTest(Brain):
def __init__(self, name, robot):
Brain.__init__(self, name, robot)
print "initializing camera ..."
self.getRobot().startService("V4LCamera")
self.camera = self.getRobot().getService("V4LCamera")
self.camera.makeWindow()
self.getRobot().startService("ptz")
self.ptz = self.getRobot().getService("ptz")
self.ptz.pan(0)
self.ptz.tilt(0)
self.ptz.zoom(0)
self.pan = 0
self.tilt = 0
print "done initializing camera"
def step(self):
self.camera.lockCamera = 1
self.camera.superColor("red",2)
print "Blobbing Image"
# channel, low val, high val, "mass" or "area", num blobs, draw box = 1
self.camera.maxBlobs(2, 1, 255, "mass", 1) # change "draw box" param to 0 to increase speed, see maxBlobs function specification
self.camera.lockCamera = 0
if( self.camera.maxBlob.mass > 10 ):
self.pan = (( self.camera.blob[0].min_x + self.camera.blob[0].max_x ) / 2 -
(self.camera.width/2)) / 10
self.tilt = -1*(( self.camera.blob[0].min_y + self.camera.blob[0].max_y ) / 2 -
(self.camera.height/2)) / 10
self.ptz.panTiltRel(self.pan, self.tilt)
print "PAN IS ", self.pan
print "TILT IS ", self.tilt
print "\n"
def INIT(robot):
return CameraTest('CameraTest', robot)
Exercises1. Comparing filters.
Experiment with the different blurring filters before filtering on a particular color.
Experiment with writing your results out to different channels. 2. Follow that Ball.
Write code for your robot that will search for a ball of a specified color. Once the robot finds the ball, it should move towards it. If you move the ball, the robot should relocate the ball and move towards it again.
Extension to the exercise: Robot Soccer. Have the robot push the ball towards a goal to play soccer. 3. Color training.
Write a program that will train on the color of an object (for example, the color of your pants) when a bumper is hit. Once the robot has finished training, it should maintain a constant distance from the trained color. For example, if you trained on the color of your pants, the robot should move towards you when you move away from it and it should back up when you move towards it. You may want to use sonars as well as vision for this exercise. 4. Pick Up the Trash.
Develop a system that performs the task from the AAAI 1994 Pick Up the Trash Competition (find reference). Your robot will need to find trash of a particular color (you can choose a can of soda, a cup, or some other object that your gripper can pick up). Once it finds the trash, the robot should move towards it and pick it up with the gripper. Then the robot should find the trash can and deliver the object.
Start with one color trash and one trash can. As an extension, create an environmentally conscious robot that will deliver trash to the trash can and recyclables to the recycling bin. (It will be easiest to complete this task using four distinct colors.)
Pyro Modules Table of Contents
Modules
Additional Resources
Reference: PyroSiteNotes
ViewWiki | EditWiki | Webmaster@wiki.cs |
Beyond Legos