A few days ago I wrote part 1 of this tutorial on building a robotic arm that you can control with your hand movements in air. That tutorial dealt with the hardware side, so now I’ll talk about the software and how it all works together. In case you missed the first part, here’s the robot arm again:
Software
I used node.js with the leapjs and johnny-five packages to get data from the Leap Motion device and send it to the Arduino. The code itself is on GitHub here:
https://github.com/ytham/robotarm
To run it, we’ll start by including the leapjs and johnny-five packages. You should install the packages first into the directory that you’ve unzipped everything to (npm install johnny-five && npm install leapjs). Next, in your Arduino software, you will need to go to hook up your board and then go to Files > Examples > Firmata > StandardFirmata (this is required for johnny-five). Then hit Upload.
To start the software:
node robotarm.js
The javascript library for Leap, called leapjs, works by waiting for a Leap ‘frame’ to be emitted to the listener, which then calls an anonymous function. We use this function to set our variables for later use with johnny-five. Here’s the basic form of it:
var controller = new Leap.Controller();
controller.on('frame', function(frame) {
<get our Leap Motion data here>
});
The johnny-five package basically requires you to initialize a board and whatever else you are going to use inside that.
board = new five.Board();
board.on('ready', function() {
servoBase = new five.Servo(3);
servoShoulder = new five.Servo(9);
servoElbow = new five.Servo(10);
servoClaw = new five.Servo(6);
...
There are four servos in the arm, and three separate main controls. The base rotation is controlled by the x position of the hand. The end effector is controlled by the distance between two fingers. Finally, the shoulder and elbow joint angles are calculated by the inverse kinematics equation with inputs of the y and z coordinates.
function calculateBaseAngle(x) {
var n = 100*normalize;
x = 1.5+2*x/n;
var angle = 90+Math.cos(x)*90;
return angle;
}
While the Leap Motion device measures coordinates linearly, the base rotation is circular. I wanted to feed the x value through a cosine function so that it would not be too sensitive in midranged values. In order to calculate the base angle, I needed to get its range of values that would jibe well with a cosine function, and since the Leap Motion’s vales are in the range of -300 to 300 in Leap Space, I divided by 100 times the normalize value, which I will go into more detail in later paragraphs.
Since the servo rotation values are between 0 and 180, I wanted 90 to be the value when the arm was pointing forward, so modified the values so that they will range from 0 to 180 with 90 being in the center.
For the shoulder and elbow joint angles, the servos are placed in such a way as to allow for a large range of motion and “negative” values for the elbow. See the diagram below; the triangles show the pivot points of the servos.
In order to calculate the servo angles, we use an inverse kinematics equation. In physics, forward kinematics is used to calculate the position of an end effector, given its joint angles. However, with the Leap Motion device, we want to translate our hand’s position information into joint angles, thus we have to use inverse kinematics:
function calculateInverseKinematics(x,y,z) {
z = -z;
var t1 = Math.acos((square(z)+square(y)-square(l1)-square(l2))/(2*l1*l2));
var t2 = Math.asin(((l1+l2*Math.cos(t1))*y-l2*Math.sin(t1)*z)/(square(l1)+square(l2)+2*l1*l2*Math.cos(t1)));
return {
theta1: t1,
theta2: t2
}
}
I invert the z value so that reaching towards the Leap Motion device in front of me will make the arm go forward as intended. The other theta1 and theta2 equations are simply the inverse kinematics equations from the following website:
http://cnx.org/content/m11613/latest/
The one thing to note is that we have to calculate the length (variables l1 and l2) based on the distance in terms of the Leap Motion device coordinates. The easiest way to find the length is to output your hand’s y-position to the console.
Place your arm section over the Leap Motion device and place your hand at the bottom of the section. Make note of the y-coordinate. Move your hand to the top of the arm section and make a note of the y-coordinate. Subtract the two numbers and the absolute value is the length in Leap space, which is what you will use for your length in the inverse kinematics equation. Do the same for the second arm section. These values will obviously be different for different arm lengths, so be sure to change this:
var l1 = 40*normalize; var l2 = 40*normalize;
The normalize value is set to 3 in my app, but you can tweak it to change the sensitivity of the system as well as the range of motion. Normalize is also used in the input for the inverse kinematics function:
if(frame.hands.length > 0) {
handPosition = frame.hands[0].palmPosition;
angles = calculateInverseKinematics(0,-10+handPosition[1]/normalize,handPosition[2]/normalize);
moveBase = 180-calculateBaseAngle(handPosition[0]/2);
moveShoulder = toDegrees(angles.theta1);
moveElbow = 45+toDegrees(angles.theta2);
}
Since we are operating the shoulder and elbow joints on only the yz plane, we feed a zero into the x position (the x axis is calculated separately by the base rotation angle). I decreased the y input by 10 so that my hand would not have to be as high above the Leap Motion device, but you can play with this value and your mileage may vary. Another thing to note is that the elbow has an offset of 45 degrees. This is intentional so as to allow for “negative” elbow values.
The last part that needed to be done was to calculate the angle of the end effector, which would basically open and close the end effector (gripper). This was simply done in the Leap loop while calculating the finger distance:
if(frame.pointables.length > 1) {
f1 = frame.pointables[0];
f2 = frame.pointables[1];
fingerDistance = distance(f1.tipPosition[0],f1.tipPosition[1],f1.tipPosition[2],f2.tipPosition[0],f2.tipPosition[1],f2.tipPosition[2]);
moveClaw = (fingerDistance/1.5) - minimumClawDistance;
}
What I’m doing here is basically getting the first two fingers (and ignoring the rest) and finding the distance in free space between them using a distance function which I have defined (which would be the same as any other distance function). The Leap motion does have one annoying bug, which is that it will no longer detect your two fingers if you put them together. Thus, I wanted to make the claw close without having to have my two fingers touch, which is the minimumClawDistance.
All of the global variables are read by the Johnny-Five board loop and the output is sent to the servos here:
this.loop(40, function() {
if(!isNaN(moveShoulder) && !isNaN(moveElbow)) {
servoBase.move(moveBase);
servoShoulder.move(moveShoulder);
servoElbow.move(moveElbow);
}
if(moveClaw >= 0 && moveClaw <= 100) {
servoClaw.move(moveClaw);
}
console.log("Base: " + Math.floor(moveBase) + "\tShoulder: " + Math.floor(moveShoulder) + "\tElbow: " + Math.floor(moveElbow) + "\tClaw: " + Math.floor(moveClaw));
});
The main things to note are the if statements; they basically protect the system from some unintended values (and will protect your arm from crashing down into the table if it his a NaN value). The loop reads values every 40ms, but you can play around with this value if you’d like.
Well, that’s all for now. I hope you enjoyed this post! Let me know if you have any questions.
September 26, 2013 at 4:50 am
Hello, i have given a project to do this leap motion to control a robotic arm. I have no knowledge about Java but just C++ and adrduino. I have look through and try many ways on the way you have done about the programming. Your help will be really useful to help me solve my problem on connecting the arduino with the leap motion. Thank You.
October 1, 2013 at 11:25 pm
It’s written in JavaScript, which is much different than Java. Either way, the logic is the same; we’re just putting the leap motion data through an inverse kinematics function to control the robotic arm. It can be done in any language, but you’d need to have the libraries for leap motion and arduino.
October 2, 2013 at 3:03 pm
Hello, i try the coding using Visual studio to edit, den use command prompt for node robotarm.js to run, but it seems there are errors.. is there any step dat i have miss out?? Thank you.
October 4, 2013 at 4:29 am
I’ve updated the code on github; I think it should work now.
November 12, 2016 at 8:46 am
Please, could u show me how to wire the ardunio to motor
October 2, 2013 at 4:03 pm
Hello, I have an error in executing robotarm.js. When I type robotarm.js, there is TypeError, saying that “Cannot read property ‘Controller’ of undefined at Object., at Module._compile, at Module.load, at Function.Module._load, at Function.Module.runMain”. Could you help me to solve this problem?
December 12, 2013 at 5:19 pm
Ensure that you have leapjs installed (npm install leapjs). In addition to that, it looks like they’ve changed their API so you should change the line “var Leap = require(‘leapjs’).Leap;” to “var Leap = require(‘leapjs’);”. Good luck!
October 4, 2013 at 2:18 am
Hello, I have an error in executing robotarm.js. When I type robotarm.js, there is TypeError, saying that “Cannot read property ‘Controller’ of undefined at Object., at Module._compile, at Module.load, at Function.Module._load, at Function.Module.runMain”. Could you help me to solve this problem?
December 15, 2013 at 11:14 pm
Hello, and wow this is a truly amazing project! I am trying to build an arm with a distance sensor and i was wondering what version of the arduino did you use and which pins did you connect the servos to? Thank you.
March 21, 2014 at 5:47 pm
I used the Arduino Uno. For the johnny-five library, you have to hook up the servos to the PWM pins.
July 15, 2015 at 11:09 am
can u explian how built with ardunio uno
February 6, 2014 at 8:57 pm
Nice work bro! But how did you wire it? You have a diagram how to wire this thing to the bread board?
March 21, 2014 at 5:50 pm
The wiring is simple in this case (although not recommended as it may sometimes brown out):
Wire all of the servos to the same 5V and GND node as the arduino. Wire all of the control pins to the arduino. Plug in USB.
The correct and recommended way to do this would be to connect the grounds as before, but power the servos with a separate 6V DC power source and connect that to the common ground as well.
March 21, 2014 at 5:26 pm
Hey,
I just had one question, is it possible to code this using Java? If its possible, please guide me how to approach it. Thanks!
March 21, 2014 at 5:52 pm
I’m sure there is a way to do it, but I have no experience with Java, so you’ll probably have the scour the internet. The first thing you would want to check is if Leap exposes a Java interface. Then look and see if there’s any Java library from which you can write code to control an Arduino. If you can do those two things, you should be able to re-create this.
August 18, 2014 at 4:14 am
Hello how did you know how to get the data from the leap motion? i mean, what did you do for making the servos moves while the hand moves, as well?
October 19, 2014 at 4:08 pm
Hey,
I don’t get one thing: How on earth do I install leapjs and from where?? Please help me with my problem because all I see on leap’s site is a link which does not open and a message talking about CDN. How do I install run johnnyfive??
Thanks
October 30, 2014 at 3:00 pm
hi, i’m have this error:
module.js:340
March 8, 2015 at 1:34 pm
Trying 😛
alert(0);
March 8, 2015 at 1:34 pm
abc
March 8, 2015 at 1:37 pm
Another Try Sory:
Text
July 1, 2015 at 1:40 pm
For people trying to do this in Visual Studio, using NTVS makes everything much easier to use.
https://nodejstools.codeplex.com/
August 24, 2015 at 3:31 am
hello sir….
how to install node.js and jhony five in windwos 7 32bit .
i have problem, i have all installed in my lappy ..,. but some problems!!
and sir ,how to write java script in windows ?. which software you used for java scripting.??
thanku 🙂
August 27, 2015 at 3:37 am
Usually I don’t read article on blogs, but I wish to say that this write-up very pressured me to try and do so!
Your writing taste has been surprised me. Thank you,
very nice article.
October 15, 2015 at 9:39 pm
Hey bud,
Great great work on the arm. I was planning on doing something very similar, but didn’t know where to start about, was looking into kinematics and the whole array of equations.
I’m new to arduino, so would you by any change have any info on the circuitry of this system and how you connected everything?
Thanks alot in advance
V
October 29, 2015 at 6:45 am
Hi!
I’m trying to implement your code with my own built bot, I’ve tested the bot and it works. However, some of your code gives me problems, especially when it calculates the inverse kinematics
When calculating b with following code
var b = Math.acos((square(LENGTH1)+square(hypotenuse)-square(LENGTH2))/(2*LENGTH1*hypotenuse));
it returns a number above 1, but below 2, which causes Math.acos to return NaN
Can you see what’s wrong?
March 31, 2016 at 4:38 pm
Hi, great job with the arm, however when when I built it and was testing it the code gave me this error, I posted it here, maybe you can help? Thank you and I would really appreciate it 🙂 http://stackoverflow.com/questions/36344989/leap-motion-node-js-arduino-giving-me-serialport-not-open-error
July 22, 2016 at 4:23 am
Please can i get a code for arduino with five servo motors
September 7, 2016 at 5:05 am
Hii UJI,
I want to do this project in my college…kindly send me the circuit diagram of this project…..because if I do this project they will ask circuit diagram,with out circuit diagram they don’t give the permission to do this project..please send me as soon as possible
March 8, 2017 at 8:39 pm
Hi, this is very helpful to me. Thanks for publishing. if our robot arm has 6 DOF , how to handle the wrist servos? please help me.
July 19, 2017 at 2:22 am
I am given a project to make robotic finger sensor.Anyone can u help me with the circuit diagram for it