Intelligent Materials and Systems Lab - User contributions [en]

File:Momentum.jpg

2015-02-07T21:57:24Z

Moment92: Moment92 uploaded a new version of "File:Momentum.jpg"

User:Moment92

2014-08-21T11:23:11Z

Moment92:

<h1>Andres Traumann</h1>

E-mail: andres.traumann.01@gmail.com

Mobiil: 56817325

Skype: moment9292

<h2>Projects</h2>
[[Secure:Andres_MSc|MSc Work]]

File:Momentum.jpg

2014-08-21T11:07:11Z

Moment92: Moment92 uploaded a new version of "File:Momentum.jpg"

VirtualMannequinResearch

2014-07-03T18:49:43Z

Moment92: Blanked the page

VirtualMannequinResearch

2014-07-03T17:35:05Z

Moment92: Created page with "== Overview == The purpose of this wiki is to document research about virtual clothes fitting. The aim of this project is somewhat similar to the goal of the Robot Mannequin p..."

== Overview ==
The purpose of this wiki is to document research about virtual clothes fitting. The aim of this project is somewhat similar to the goal of the Robot Mannequin project, but instead of robots, the fitting is done virtually by using 3D-models. The research is being done by Andres Traumann and supervised by Gholamreza Anbarjafari.

User:Moment92

2013-07-24T18:03:35Z

Moment92:

<h1>Andres Traumann</h1>

E-mail: andres.traumann.01@gmail.com

Mobiil: 56817325

Skype: moment9292

Nao localization

2013-06-20T13:59:23Z

Moment92: /* Position estimation */

== Overview ==
We plan to use a [http://en.wikipedia.org/wiki/Particle_filter particle filter] that gets information from our line detection algorithm and from the estimated position change while moving. To estimate the position change, we found a useful function in Aldebaran's API: [http://www.aldebaran-robotics.com/documentation/naoqi/motion/control-walk-api.html#ALMotionProxy::getRobotPosition__bCR ALMotionProxy::getRobotPosition()].

== Position change estimation ==
The position change estimation is not perfect and it is not possible to use it to walk "blindly", but it is decent enough for our needs, because we also use information from the camera. Initial tests show that after a few meters of walking, this method usually accumulates to mistakes around 30 centimeters.

Setup Nao

2013-06-19T19:30:01Z

Moment92: /* Setting up SDK */

== First steps ==
Mostly people like to start with the Choreographe.. It is easy to use and it helps people to get excited about Naos. You can install it from the DVD that was included with the Nao. You can also download it from [http://users.aldebaran-robotics.com http://users.aldebaran-robotics.com], but you need to have an account there to do it.
Here is a quickstart Choreographe [https://developer.aldebaran-robotics.com/resources/tutorial/nao-first-steps/ tutorial].

== Setting up SDK ==
To do some more serious programming, it is better to use some general-purpose programming language.

Our team uses mainly C++ and Python. To start programming in those languages, it is neccessary to set up the SDKs. You can get all the needed software from the DVD included with the Nao, or from the [https://community.aldebaran-robotics.com/ https://community.aldebaran-robotics.com/] website. The website is preferred, because it is more up to date. The website requires logging in, but you can get the username and password from the members of your team. 

The SDKs are fairly easy to set up, as Aldebaran has adequate documentation for that. For C++ you can follow the instructions [http://www.aldebaran-robotics.com/documentation/dev/cpp/install_guide.html#cpp-install-guide here] and for Python you can follow the instructions [http://www.aldebaran-robotics.com/documentation/dev/python/install_guide.html#python-install-guide here].

== qiBuild ==
qiBuild is the Aldebaran tool for managing projects, using different toolchains and carrying out the C++ building proccess. Basically, it is a collection of Python scripts. The qibuild script is the most important of them, it calls cmake to generate the required files for the building proccess using the CMakeLists.txt file, and then builds the generated Makefile. Aldebaran has a brief explenation on using qiBuild [http://www.aldebaran-robotics.com/documentation/dev/cpp/tutos/using_qibuild.html here].

Nao segmentation

2013-06-02T10:18:18Z

Moment92: /* Pixel classification */

== Overview ==
We need to use some criteria to detect real-world objects. The easiest method to distinguish different objects is to use their colour. As many objects on the football field are colour-coded, it is reasonable to use it, as it is also computationally very effective. It means that we claasify every pixel into a colour class and after this is done, we form the blobs of the similar pixels.

== Pixel classification ==
We use a lookup table to classify pixels into colour classes. We have a special program that let's us easily construct new lookup tables and this method has worked well enough. The lookup table format is described [http://www.cs.cmu.edu/~jbruce/cmvision/papers/JBThesis00.pdf here] on pages 15-16. 

When using lookup tables, it is crucial to stop camera's from being too "smart". Things like [http://en.wikipedia.org/wiki/Color_balance white balance] and [http://en.wikipedia.org/wiki/Exposure_%28photography%29#Automatic_exposure auto-exposure] might make the picture look more natural, but it will also make lookup tables harder to use. That's why we turned them off with [http://www.aldebaran-robotics.com/documentation/naoqi/vision/alvideodevice-api.html#ALVideoDeviceProxy::setCameraParameter__ssCR.iCR.iCR ALVideoDeviceProxy::setCameraParameter], specifically we set kCameraAutoWhiteBalanceID and kCameraAutoExpositionID to zero.

== Blob formation ==
After we have classified pixels, blobs are formed from pixels of the same colour class. Blob formation is done using the CMVision algorithms with slight modifications to work better with our own data structures. The algorithms are described in detail [http://www.cs.cmu.edu/~jbruce/cmvision/papers/JBThesis00.pdf here] on pages 16-19.

== End result ==
Applying those techiques on a frame gives us an array that consists of blob lists. In every blob list we have blobs that belong to the same colour class. They are also sorted by their area, so that it's easy to find the largest blobs of some specific colour.

Nao localization

2013-06-02T10:00:33Z

Moment92: Created page with "== Overview == We plan to use a [http://en.wikipedia.org/wiki/Particle_filter particle filter] that gets information from our line detection algorithm and from the estimated p..."

== Overview ==
We plan to use a [http://en.wikipedia.org/wiki/Particle_filter particle filter] that gets information from our line detection algorithm and from the estimated position change while moving. To estimate the position change, we found a useful function in Aldebaran's API: [http://www.aldebaran-robotics.com/documentation/naoqi/motion/control-walk-api.html#ALMotionProxy::getRobotPosition__bCR ALMotionProxy::getRobotPosition()].

== Position estimation ==
The position estimation is not perfect and it is not possible to use it to walk "blindly", but it is decent enough for our needs, because we also use information from the camera. Initial tests show that after a few meters of walking, this method usually accumulates to mistakes around 30 centimeters.

Nao

2013-06-02T09:48:42Z

Moment92: /* Our solutions to some important problems */

== Summary ==

We are a team of people who are interested in robotics, software part of robotics to be more precise. After Robotex 2012 competition we wanted to do something cool and challenging, so we ended up playing around with 4 Nao robots who just waited to be found :)

Our goal is to prepare the Naos for [http://www.robocup.org/robocup-soccer/standard-platform/ Robocup Standard Platform] competition. But, whether or not we are able to succeed, we will make an effort to learn as much as possible through making the Naos do something useful and pleasant.

== Our solutions to some important problems ==
*[[Nao_segmentation|Image segmentation and blob detection]]
*[[Nao_line_detection|Line detection]]
*[[Getting-real-world-coordinates-from-image-frame|Getting real world coordinates from the image]]
*[[Nao_localization|Localization]]

== How To ==
Developing software for Nao is full of challanges. Here we have some tutorials/instructions that might help you get started with the basic things. 

*[[Setup_Nao | How to start with Nao?]]

*[http://www.aldebaran-robotics.com/documentation/nao/upgrade.html Upgrade Nao - link to Aldebaran guide.]

*[[Start_learning_robotics | How to start learning robotics?]]

*[[Nao_Module | How to build a module?]]

*[[Nao_Keyboard_Control_Python | How to use keyboard to walk Nao around?]]

*[[Coping_with_Linux | How to cope with linux?]]

*[[Set_up_Opencv | How to set up OpenCV?]]

==What problems have we faced?==
===Beginnings "Big Unknown"===
As you are reading this you are on the right direction:) We found also useful to read other users' tutorials especially wiki from [http://robotica.unileon.es/mediawiki/index.php/Nao_tutorial_1:_First_steps Robotics Group of the University of León].

===Version differences===
We started with Naos having version 1.12 everything and started learning from the default reference of SDK 1.14. We noticed it but didn't think that it would make a lot of difference... Wrong! Aldebaran is very actively advancing it's software. We had many problems with modules not existing on Nao etc and overall have spent a lot of time on it. So update and work with the newest and the coolest modules, or go to [https://developer.aldebaran-robotics.com/doc/1-12/ref/index.html API 1.12 reference] to avoid some frustration. A good tutorial is [http://www.aldebaran-robotics.com/documentation/nao/upgrade.html here]. 

It is also useful to know that Aldebaran has made a list of function which have been changed [http://www.aldebaran-robotics.com/documentation/dev/cpp/tutos/porting_to_1.12.html]. 

N.B 
When updating Naos, don't forget updating the SDK, cross-toolchain, qibuild etc. If you do, you might encounter some errors that are really hard to interpret. One of such errors occoured when a module was built using the old version of Atom-cross-toolchain. When Naoqi tried loading the module, it gave: 
<code>libboost_signals-mt-1_45.so.1.45.0: cannot open shared object file: No such file or directory</code> 

==="Hidden" files in users.aldebaran-robotics.com===
Getting state of art software for nao you can visit [http://users.aldebaran-robotics.com/ http://users.aldebaran-robotics.com/]. Login. Go Software > Download > All Downloads > Navigate there (all software isn't on the front page!).

Getting-real-world-coordinates-from-image-frame

2013-05-23T10:31:39Z

Moment92: /* How to map 2D points to 3D? */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D points to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices. 
TODO: A picture of previous formula altered.

In the end, it turned out that it was really easy to solve the equation. 

When we assume that Z=0 and (a)3,4 is the camera matrix multiplied by the rotation-translation matrix then we actually get the following system of equations.
 
[[File:Inv_eq.png]]
 
We don't know the value of s, X and Y. As we have three unknowns and thee equations, we can just solve the equation system and obtain our coordinates u and v. We implemented it using the [http://en.wikipedia.org/wiki/Cramers_rule Cramer's rule].

== Changing plane rotation while robot is walking ==
We noticed some quite heavy tilting happening on the image while Nao was moving, in figures it was approximately +5...-5 degrees over the orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually, it seemed a lot like this - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

The angle that Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make the camera's pose dependent on the robot's rotation. After some thinking we found that we had to modify the camera's rotation matrix according to Nao's torso rotation in a bit more complex way, as they don't have a linear correlation.

== Performance ==
TODO:

Getting-real-world-coordinates-from-image-frame

2013-05-23T10:25:48Z

Moment92:

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D points to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices. 
TODO: A picture of previous formula altered.

In the end, it turned out that it was really easy to solve the equation. 

When we assume that Z=0 then we actually get the following system of equations.
 
[[File:Inv_eq.png]]
 
We don't know the value of s, X and Y. As we have three unknowns and thee equations, we can just solve the equation system and obtain our coordinates u and v. We implemented it using the [http://en.wikipedia.org/wiki/Cramers_rule Cramer's rule].

== Changing plane rotation while robot is walking ==
We noticed some quite heavy tilting happening on the image while Nao was moving, in figures it was approximately +5...-5 degrees over the orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually, it seemed a lot like this - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

The angle that Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make the camera's pose dependent on the robot's rotation. After some thinking we found that we had to modify the camera's rotation matrix according to Nao's torso rotation in a bit more complex way, as they don't have a linear correlation.

== Performance ==
TODO:

File:Inv eq.png

2013-05-23T10:24:05Z

Moment92:

Getting-real-world-coordinates-from-image-frame

2013-05-23T10:23:00Z

Moment92: /* How to map 2D point to 3D? */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D points to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices. 
TODO: A picture of previous formula altered.

In the end, it turned out that it was really easy to solve the equation. 

When we assume that Z=0 then we actually get the following system of equations. 

We don't know the value of s, X and Y. As we have three unknowns and thee equations, we can just solve the equation system and obtain our coordinates u and v. We implemented it using the [http://en.wikipedia.org/wiki/Cramers_rule Cramer's rule].

== Changing plane rotation while robot is walking ==
We noticed some quite heavy tilting happening on the image while Nao was moving, in figures it was approximately +5...-5 degrees over the orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually, it seemed a lot like this - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

The angle that Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make the camera's pose dependent on the robot's rotation. After some thinking we found that we had to modify the camera's rotation matrix according to Nao's torso rotation in a bit more complex way, as they don't have a linear correlation.

== Performance ==
TODO:

Getting-real-world-coordinates-from-image-frame

2013-05-23T10:04:43Z

Moment92: /* Changing plane rotation while robot is on the move */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D point to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices. 
TODO: A picture of previous formula altered.

In the end, it turned our that it was really easy to solve the equation.

== Changing plane rotation while robot is walking ==
We noticed some quite heavy tilting happening on the image while Nao was moving, in figures it was approximately +5...-5 degrees over the orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually, it seemed a lot like this - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

The angle that Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make the camera's pose dependent on the robot's rotation. After some thinking we found that we had to modify the camera's rotation matrix according to Nao's torso rotation in a bit more complex way, as they don't have a linear correlation.

== Performance ==
TODO:

Getting-real-world-coordinates-from-image-frame

2013-05-23T09:59:23Z

Moment92: /* How to map 2D point to 3D? */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D point to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices. 
TODO: A picture of previous formula altered.

In the end, it turned our that it was really easy to solve the equation.

== Changing plane rotation while robot is on the move ==
We noticed quite aggressive tilting happening one the image while Nao was moving, in figures it was approximately +5...-5 degrees over orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually it seemed a lot - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

Angle what Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make camera pose dependent on the robot rotation. After some thinking and drawing we found that we had to update camera rotation matrix according to Nao's torso rotation.

== Performance ==
TODO:

Getting-real-world-coordinates-from-image-frame

2013-05-23T09:59:07Z

Moment92: /* How to map 2D point to 3D? */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D point to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this equation. We had problems with inverting matrices. We found that the OpenCV Mat::inv() didn’t give good enaugh results for most matrices – maybe because these matrices weren’t invertible. The other explanation is that the matrix had really small numbers (10^-15) and really big numbers (10^15) and it might be that the calculations became too inaccurate due to it.

TODO: Dig deeper, what was the problem with not being able to invert those matrices.
TODO: A picture of previous formula altered.

In the end, it turned our that it was really easy to solve the equation.

== Changing plane rotation while robot is on the move ==
We noticed quite aggressive tilting happening one the image while Nao was moving, in figures it was approximately +5...-5 degrees over orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually it seemed a lot - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

Angle what Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make camera pose dependent on the robot rotation. After some thinking and drawing we found that we had to update camera rotation matrix according to Nao's torso rotation.

== Performance ==
TODO:

Getting-real-world-coordinates-from-image-frame

2013-05-23T09:53:37Z

Moment92: /* How to map coordinates from 3D to 2D? */

== Why do we need it? ==
Our goal was to convert coordinates on the image to real world coordinates e.g. we wanted to know object's position relative to robot's position. This is necessary to make a robot understand, where the objects are relative to his position. More generally speaking, this is also needed to figure out the absolute coordinates on the football field.

== Pinhole camera model ==
We used the pinhole camera model.
I am not going to describe all the theory for that, here is the main formula that describes how the objects are projected onto the screen.
 
[[File:Pinhole_Camera_Model.png]]
 
To learn about this model, there are good enough resources available (start with [http://en.wikipedia.org/wiki/Pinhole_camera_model wiki] and [https://www.youtube.com/watch?v=uhP3jrxraMk udacity]), but focus more on the overall idea, problems we had and tools we used.

== How to map coordinates from 3D to 2D? ==
To be able to convert between two coordinate systems, we need to know the intrinsic parameters and extrinsic parameters of the camera. The former describes how any real world object is projected to camera’s light sensors. It consists of parameters such as camera’s focal length, principal point and skew of the image axis. The latter gives information about camera’s pose in the observed environment (3 rotations and 3 translations as we live in a 3-dimensional world). To convert a 2-dimensional screenpoint into a 3-dimensional world point, we also need to make an extra assumption that all the objects that interest us are on a specific plane (e.g. on the floor where Z=0).

== How did we find out the camera parameters and the pose? ==
We based our calibration system on the OpenCV implementation of an algorithm that gathers tries to match screen coordinates with known real-world coordinates and hence tries to estimate the required parameters by using many different observations . Because we only need to find those parameters once, we did not see a need for developing anything more complex for that. We found useful OpenCV functions specially designed for finding the camera matrix (intrinsic parameters) and rotation-translation matrix (extrinsic parameters) (see [http://docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html documentation] for calibrateCamera(), findChessboardCorners(), drawChessboardCorners() and projectPoints() also [http://docs.opencv.org/doc/tutorials/calib3d/camera_calibration/camera_calibration.html tutorial] might be useful). If you are interested in the inner workings of these algorithms, you can go and read the documentation or the source code.
After collecting all results in terms of these parameters, we were able to convert real world 3D-points onto the image plane. But as this wasn’t our goal (we wanted 2D -> 3D) we had to reverse this equation.

== How to map 2D point to 3D? ==
There was a bit of chaos and many “wasted” days in terms of reversing this operation. We had problems with inverting matrices. We took the model Opencv Mat::inv() didn’t give right results and some matrix pseudo inverse seemed to be not working – probably these matrices weren’t invertible.

TODO: Dig deeper, what was the problem with not being able to invert those matrices.
TODO: A picture of previous formula altered.

In the end we solved equations with [http://en.wikipedia.org/wiki/Cramer's_rule Cramer’s rule].

== Changing plane rotation while robot is on the move ==
We noticed quite aggressive tilting happening one the image while Nao was moving, in figures it was approximately +5...-5 degrees over orthogonal axis relative to the image frame (TODO: something more convincing needed as a figure). Visually it seemed a lot - have a [https://www.youtube.com/watch?v=9PlHgYVYTgQ look].

Angle what Nao's torso is moving can be easily measured with [http://www.aldebaran-robotics.com/documentation/naoqi/core/almemory-api.html?highlight=memoryproxy#ALMemoryProxy::getData__ssCR AL::ALValue ALMemoryProxy::getData("device")]. Next we made an assumption that Nao's torso is rotating the same amount as it's head and made connection fixed between torso and head.
Having obtained the angle we had to make camera pose dependent on the robot rotation. After some thinking and drawing we found that we had to update camera rotation matrix according to Nao's torso rotation.

== Performance ==
TODO: