December 01, 2013

Number recognition with neural network in LabVIEW

Introduction

I have a class about neural nets at the university, so I tried to program one for number recognition like this but in LabView. Then I realized: I didn't have enought knowledge for this, so I google neural networks and found this awesome course: https://www.coursera.org/course/neuralnets

So how it's working?

It has a recognition and a learning mode. As input the network get xij, an 8x8 boolean array with the pixels of the image. And there is wkij, the weight; a 3D integer array. It contains for all the 10 numbers (this is the first dimension) for all the pixels (2nd and 3rd dimension) the probability of this pixel is in the image of the number. If the pixel is always in the image, then it's a big positive number. If it's never there, it's big negative number.
In recognition mode it makes a sum of product for each numbers so: sumi=0->7( sumj=0->7( Xij*wkij)) (Xij is +1 if xij true, and -1 if xij false) and the result is a 1 D integer array yk. Then it looks for the maximum of the array y, and the index of the maximum is the result.

In learning mode it needs the image of a number, the number itself and from these information it calculates and changes the weight. It goes through all the numbers and for each number every pixel. If the number is equal the given number AND the pixel is true, then +10 to the weight, if pixel is false -10 to the weight for this case. If the number is NOT equal with the given one, AND the pixel is true, then -1 to the weight and +1 if the pixel is false. Train the system at least 2-3 image for each number, but the more image you train, the better result you get. But you have to use the same amount of image for each numbers, otherwise it will give back an incorrect result.

Result

It was also surprising for me, how good it works:
The code is available here

Best regards:

Mark

November 13, 2013

TDK - Gépi látás - képfeldolgozás és mélységi kép kettő webkamera segítségével

For the english version click here.
[Frissítés] Azóta lezajlott az eredményhirdetés is, ahol 2. díjat kaptam a biomechatronika szekcióban :P
----------------
A BME-n a 2013-as TDK konferencián ezzel a dolgozattal indultam. Dolgozatom a gépi látás megvalósításáról szól LabVIEW környezetben. Változatos képfeldolgozási technológiákat mutatok be folyamatosan szem előtt tartva, hogy a módszerek szűkebb költségvetésű egyetemi és hallgatói projektekben is használhatóak legyenek.
Először áttekintem a LabVIEW programozás alapjait, az adatfolyam programozás mibenlétét, előnyeit, hátrányait. Csak olyan mélységig, hogy a későbbi fejezetek azok számára is érhetőek legyenek, akik még nem programoztak LabVIEW környezetben.
Ezután rátérek az egyszerű képfeldolgozási módszerekre. Itt először a színalapú mintázatkeresést mutatom be. Ennek keretében a megadott színbeli mintázatot keresi a program a képen. Majd a fekete-fehér képen történő forma alapú mintázatkeresést írom le. Itt a különböző módszerekkel monokrómmá tett képen keresi az előre definiált alakzatot.
Majd folytatom a magasabb szintű képfeldolgozási technológiákkal. Ennek keretében a kétdimenziós vonalkód (QR kód és Data Matrix) olvasásáról írok, melyek egyre elterjedtebbek mind az iparban, mind a mindennapi életben.
Ezt követően bemutatom azt, hogy két egymás melletti, csupán horizontálisan eltolt webkamera képét felhasználva hogyan lehet - az emberi látáshoz hasonlóan - mélységi képet előállítani. Itt fontos a megfelelő beállítás, ami a kameráknak több szögből megmutatott fekete-fehér ráccsal végezhető. Ezután a beállítás természetesen elmenthető, így ezt elegendő kamera-beállításonként egyszer elvégezni. 
Ezt követően a program képes a webkamerák képét élőben (real time) mélységi képpé alakítani és azt egy színes grafikonon megjeleníteni. Az összes ponthoz rögtön rendel egy mélységi koordinátát is, amit egyrészt a színekkel érzékeltet, másrészt ha az egeret az adott pont fölé visszük, akkor külön is megjeleníti. 
Természetesen ennek a technológiának is megvannak a korlátai, nem várható el, hogy két középkategóriás webkamera képéből tökéletes háromdimenziós képet kapjunk, emellett a módszer mérési tartománya nagyban függ a kamerák elrendezésétől. Egymáshoz nagyon közel elhelyezett kamerák esetén elsősorban közelebbi célpontok esetén fog pontosabban működni, míg távolabb elhelyezettek elsősorban távolabbi célpontoknál adnak használhatóbb eredményt.
Az általam készített vi-ok letölthetők innen:  https://www.dropbox.com/sh/3k30qofbyssvj01/SzJ0mVAZZ- míg a dolgozatom itt érhető el: https://www.dropbox.com/s/26ui2qnpywgg3yh/TDK.pdf
Kiemelt köszönet illeti konzulensemet, Dr Aradi Petra tanárnőt (BME-MOGI) és Kl3m3n-t az ni.com oldalról.

Üdv:
Márk

Machine Vision - Image processing and depth image with two webcams

[update] yaaaay, I got 2nd place on the biomechatronics sections :P
----------
It was my university project, and now I share it with you. Unfortunately it's only in Hungarian language, but I hope, at least the vi-s can help also the non-hungarian readers.
My paper is about machine vision in LabVIEW environment. Various image processing methods are presented, considering that it must be able to used in non-industrial environment like low-budget university and student projects.
First I review the basics of programming in LabVIEW, the dataflow programming method. I write about it only so quick to make the following chapters clear for those people who have no previous experience with LabVIEW programming.
Then I start with the easier image processing methods. First the color matching: here the program looks for a predefined color pattern and gives back the coordinates of it. Then the pattern matching algorithm on a black-white image: first we make the picture monochrome with one of the various options and then the program searches for the predefined shape or geometry pattern.
Thereafter the more sophisticated image processing methods come. I write about the two dimensional bar code (QR Code and Data Matrix) scanning, which are more and more frequently used in the industry and all day life.

Next I present how to use two horizontally offset webcam to create a depth image, like human vision. The correct settings are very important. It can be done with a black-white grid showed in different angel to the camera. Of course the configuration can be saved so it has to be done only once per camera configuration. For more info, see this video:

After that the program is ready to process the images real time to a depth image. This depth image is presented on a colored graph, where each color represent a depth, and the actual depth value can be read by hover the mouse over the point. 

Of course this technology has its own limitations. It can’t be expected to get a perfect 3D image from the image of two midrange webcam, but it can help us to choose the closer so the more important part of the image, where the previous image processing technologies should be used. It saves us resources and combining with a moving robot, the robot will be able to turn its head towards the closest activity. The measurement range highly depend on the configurations of the cameras. Very close cameras give better results in case of closer targets and cameras with higher distance can be used in farer targets.

All of my vi-s and my paper are available here: https://1drv.ms/f/s!An5KBEiOStfLgvAf6yFFrktZR4LmqQ

Special thanks to:
Best regards:
Mark

September 16, 2013

Bluetooth remote control car

Introduction

Last semester my roommate brought an RC-car to the dormitory, but we didn't have the batteries to use it. It was the beginning of the semester (=we had free time) so we decided to make it driven with an arduino via bluetooth. It took us approx. a week to make it ready. In the end it could be driven with an android smartphone or with a computer.


Hardware

First of all we removed the cover and the original IC from the car. We only kept the undercarriage and the back-motor (a simple small DC-motor, similar to this). We wanted to keep the wheel-motor, but it accidentally fell apart, so we replaced it with a servo motor (model: Tower Pro SG90).
We wanted to drive the DC motor in both direction, so we used a H-bridge IC. I don't remember the exact product number, but I think the cheapest one should be enough.
We used a cheap bluetooth modul (JY-MCU) for the connection with the remote controller (android or pc).
The wiring was not so complicated. We connected the H-bridge forward pin to pin 11, backward pin to pin 8, servo's pin to pin 3 and the BT modul to pin 0 and 1 so, that BT's RX pin was connected to arduino TX pin (pin 1) and BT's TX to arduino's RX (pin 0). But be careful! If you connect the arduino TX and RX pin it won't get any further data from the USB cable, so connect those pins only after upload your code! Or use arduino mega, which can handle more communication parallel.

Software part1: arduino

I hope it doesn't need too much comment. It was hard to find the right angle for the steering-servo, so we implemented an easy way to change the scale with sending a number 0-9. Here is the code.

Software part2.1: android

None of us could program android, so we tried to find a program written by someone else. Fortunately we found one, available on Google Play for free: https://play.google.com/store/apps/details?id=braulio.calle.bluetoothRCcontroller Actually we first found this program and after that wrote the code with the command (which letter for which direction) given by the program. The program has two interfaces, it can be used with buttons on the touchscreen but you can also use the accelerometer in your phone and just tilt your phone. Overall it's a great program and saved a lot work for us.

Software part2.2: computer

Arduino uses Bluetooth as a simple serial input (but it should be clear after the arduino code). So we wrote a program in LabVIEW to send the drive-commands (represented by characters) to the arduino. We wrote it so that you could use your arrow-keys to drive the car. Here is the vi.
The block diagram:
And the front panel in two situations (with no button pressed and with front and right keys pressed):

Video

My roommate used this project as a homework and recorded a video. Unfortunately  it's only in Hungarian language, but here it is:



Further development

Later we updated it with 2 ultrasonic distance-sensor to avoid crash. Here is the updated arduino code.

Best regards, feel free to comment:

Mark

August 03, 2013

Magnetic inverse pendulum concept

So I'm thinking about my university research and have a concept. Basically it's an inverse pendulum without wheels, levitated by permanent magnets.

Without the electronics the upper magnet would turn around and stick to the base. But when it starts turning the accelerometer detects it and starts the DC-motor. Then the propeller pushes the magnet back to the vertical position. It's fixed with plexi sheets from to direction so it can move only in 2D.

So basically this is my concept. If you have any opinion or remark, feel free to comment here, or on my facebook site. 

Best regards:

Mark

June 09, 2013

Playing Tetris in engineer style

Introduction

Month ago my girlfriend showed me a 9gag post and asked, if I could build something like that. So I ordered 2 led matrices and buttons from ebay, and built it, so here it is:
Sorry for the quality, I recorded it with my cellphone yesterday evening, but I hope you see the point of it.
It was a complicated project, so writing a step-by-step tutorial would take too many time, so I just summarize the main steps of the project and some advises.

First step: wiring of the led-matrix and functional for driving it

I checked the ebay site where I ordered the matrices and found this:
It's very similar in the inner working as the 7 segment display, see below. I wrote a custom functional for driving it, see at the code.

Second step: connect the buttons

I used very simple buttons from dx:
If you press it, it connects the 2 wire. It was the first time I've ever used a button with arduino, so I was a little bit confused about it, but Google helped me out again.

Third step: writing the tetris program

As I said in the title, and as you can see in the video, this version is only beta. There are bugs in the code, sometimes it makes weird things, and I can't figure out why. I'm going to fix it sometimes, but until then, here is the code:
(if link's broken, here: http://codepad.org/gY3fLXXx)
But during the programming I've learned something useful. There is a a 2D int array, called fix[16][8], which store the fix dots, the fix pixels. It has a 5 in every pixel, where led lights and a 0 where led's dark. So I wanted to check if there is a full line, so I wrote this code:
//full line
for(int i=0; i<16; i++){
  int ch=1;
  for(int j=0; j<8; j++) ch=ch*fix[i][j];
  if(ch>0){
    for(int k=i; k<15; k++)
      {for(int l=0; l<8; l++)
        fix[k][l]=fix[k+1][l];
      }
     for(int k=0; k<8; k++) fix[15][k]=0;
    }
     
}
It multiply every number in a line and if it's bigger than 0 at the end, ie there was no 0 in the line, so it's a full line, the program deletes it. And it didn't work. Why? After a long time I figured it out: 5^8=390625. And it's too big for an int, so it overflows and gives back a negative number! So finally I changed ch to long int and so it works now. I could change if(ch>0) to if(ch!=0)or ch=ch*fix[i][j]; to ch=ch*fix[i][j]/5; but long int was my first tough and it works so I leave it so.

Summary

It was, and it's still a big project, I hope, someday I'll finish it. Maybe I will add a demultiplexer to use less output of the arduino, and use better buttons, maybe even a use a keypad. But first I have to make the program better.

Best regards, hope you enjoy:

Mark

March 29, 2013

4-digit-7-segment display

Introducion


In this tutorial I'm going to show you the basics of the working of a 7 segment display, and how to use, when you have 4 in a row. So lets take look:

So that"s it. I ordered it from dx.com half years ago, because it was only $2, and I was interested in it. But in the beginning it was too hard for me, I just can't make it work. It has 6+6 so 12 pins in the back, and has no sign about them, as here:
So I took it away, and didn't even touch it till this month. We learned about 7 segment displays, and I was wondering, what if I could make my work. So I looked up on the internet, and find, that this display has its own library, here: https://github.com/sparkfun/SevSeg First I just downloaded and tried it, and it worked. So I looked into the code, and started to understand the inner working. So basically this display has 4*8 leds ( because each the number has 7, and there are 4 dots also). How is it possible then to control them only with 12 pins? Don't we need 4*8=32 pins for that? Yes, theoretically we would need 32 pins. But then, how does it works? The trick is, that we don't need to light up all lightning led constantly. If they blink very fast, we will see as if they're lighting. So there is 4 pin for specifying the number, and then whit the rest 8 pins we can control the 8 leds of the number. And if we change between the numbers very fast, nobody will be able to see the blinking. And changing it very fast is not a problem, when we are using arduino.

Inner working

Okay, you can say, but how is it possible? I mean, how does it work? It's quiet simple, using the special attribution of the led, that it's a diod, so the current can flow only in one direction. So here is  how to show a 5 in the 3. place:
(Red lines are HIGH, blues are LOW, because my display is common anode type. If yours is common cathode, just change every HIGH to LOW and )So as you can see 8 of the 12 pins is connected to each led, but in every number. The other 4 pins are connected to the other side of the leds but one by every number. The led will show up, if it get a red line (HIGH) from the top and a blue line (LOW) from the button. Otherwise it stays dark. Try to understand this part, because many-many things works in this principle like led cubes, and matrix displays. 

Using it

The pins are numbered this way:
And connect these pins of the display to those pins of the arduino:

   //Declare what pins are connected to the digits
   int digit1 = 2; //Pin 12 on my 4 digit display
   int digit2 = 3; //Pin 9 on my 4 digit display
   int digit3 = 4; //Pin 8 on my 4 digit display
   int digit4 = 5; //Pin 6 on my 4 digit display
   
   //Declare what pins are connected to the segments
   int segA = 6; //Pin 11 on my 4 digit display
   int segB = 7; //Pin 7 on my 4 digit display
   int segC = 8; //Pin 4 on my 4 digit display
   int segD = 9; //Pin 2 on my 4 digit display
   int segE = 10; //Pin 1 on my 4 digit display
   int segF = 11; //Pin 10 on my 4 digit display
   int segG = 12; //Pin 5 on my 4 digit display
   int segH= 13; //Pin 3 on my 4 digit display

After that we can control the display from the arduino. For this here is my code:
(NOTE: I've used Arduino MEGA for this tutorial and connected the display to pins 22-33, so if you use an UNO, change it as shown under)

boolean show[4][8];

void setup()
{
  //actually we don't need this part, but it's always good to know the wiring 
   //Declare what pins are connected to the digits
   int digit1 = 22; //Pin 12 on my 4 digit display
   int digit2 = 23; //Pin 9 on my 4 digit display
   int digit3 = 24; //Pin 8 on my 4 digit display
   int digit4 = 25; //Pin 6 on my 4 digit display
   
   //Declare what pins are connected to the segments
   int segA = 26; //Pin 11 on my 4 digit display
   int segB = 27; //Pin 7 on my 4 digit display
   int segC = 28; //Pin 4 on my 4 digit display
   int segD = 29; //Pin 2 on my 4 digit display
   int segE = 30; //Pin 1 on my 4 digit display
   int segF = 31; //Pin 10 on my 4 digit display
   int segG = 32; //Pin 5 on my 4 digit display
   int segDP= 33; //Pin 3 on my 4 digit display


   for(int i=22; i < 34; i++) pinMode(i, OUTPUT);
   
   //adding the text or numbers
   for(int i=0; i < 4; i++){for(int j=0; j < 8; j++) show[i][j]=true;} //if true: the led is dark, if false the led shows up
   //A
   show[0][0]=false;
   show[0][1]=false;
   show[0][2]=false;
   show[0][4]=false;
   show[0][5]=false;
   show[0][6]=false;
   //n
   show[1][2]=false;
   show[1][4]=false;
   show[1][6]=false;
   //n
   show[2][2]=false;
   show[2][4]=false;
   show[2][6]=false;
   //a
   show[3][2]=false;
   show[3][3]=false;
   show[3][4]=false;
   show[3][6]=false;
   show[3][7]=false;

}



void loop()
{
for(int i=0; i < 4; i++) //run on the digits
{
 for(int j=0; j < 8; j++) digitalWrite(j+26, true); //turn off every segment
digitalWrite(22+i, true); //turn the actual number on
digitalWrite(22+(i+1)%4, false); //and 
digitalWrite(22+(i+2)%4, false); //fade
digitalWrite(22+(i+3)%4, false); //the rest

for(int j=0; j < 8; j++)
{
digitalWrite(j+26, show[i][j]); //turn the specified segments on as defined in the array show
}
delay(1);
}

}

And the same code for arduino UNO:

boolean show[4][8];

void setup()
{
   //I've deleted the unnecessary part from here
   for(int i=2; i < 14; i++) pinMode(i, OUTPUT);
   
   //adding the text or numbers
   for(int i=0; i < 4; i++){for(int j=0; j < 8; j++) show[i][j]=true;} //if true: the led is dark, if false the led shows up
   //A
   show[0][0]=false;
   show[0][1]=false;
   show[0][2]=false;
   show[0][4]=false;
   show[0][5]=false;
   show[0][6]=false;
   //n
   show[1][2]=false;
   show[1][4]=false;
   show[1][6]=false;
   //n
   show[2][2]=false;
   show[2][4]=false;
   show[2][6]=false;
   //a
   show[3][2]=false;
   show[3][3]=false;
   show[3][4]=false;
   show[3][6]=false;
   show[3][7]=false;

}



void loop()
{
for(int i=0; i < 4; i++) //run on the digits
{
 for(int j=0; j < 8; j++) digitalWrite(j+26, true); //turn off every segment
digitalWrite(2+i, true); //turn the actual number on
digitalWrite(2+(i+1)%4, false); //and 
digitalWrite(2+(i+2)%4, false); //fade
digitalWrite(2+(i+3)%4, false); //the rest

for(int j=0; j < 8; j++)
{
digitalWrite(j+6, show[i][j]); //turn the specified segments on as defined in the array show
}
delay(1);
}

}

If it runs perfectly you should see "Anna" on the screen, like shown here:

You can control the text with changing the array show. For example for the first character "A" we need 6 segment from the first display: a,b,c,e,f,g. So because it's the first number, we change show[0][*] and because a=0, b=1, c=2, e=4, f=5, g=6, so

   //A
   show[0][0]=false;
   show[0][1]=false;
   show[0][2]=false;
   show[0][4]=false;
   show[0][5]=false;
   show[0][6]=false;
I think it's not so complicated :)

Deeply into the case

As you see, it use 12 pins, which actually all of the digital pin of a normal arduino UNO. So what can we do? If you think about it, there is a redundant information in the system, because every time only one of the 4 digit-pin is HIGH, so it can be controlled with a demultiplexer (aka demux). It works so:
So I0 is connected to Vcc, and F0-F3 to the 4 digit pin of the display. So now if you want a HIGH on F0, you only have to give LOW and LOW on S1 and S0, for F1 LOW and HIGH, for F2 HIGH and LOW and for F3 HIGH and HIGH. So in the end you save 2 pins, which are not the word, but sometimes can be very helpful.

I hope this tutorial was helpful, if you have any question, please feel free to write or comment

Best regards:

Mark