One of the most ambitious challenges in robotics is to develop an intelligent, autonomous machine that is capable of using objects found in the environment to perform some task. For example, humans and chimpanzees can use a stone to “hit” or a stick to “poke”. Even more ambitious is developing a robot that can operate tools or other machines, such as using a screwdriver, drilling a hole, or operating a cutting saw.

One approach is to design tools specifically for use by a robot. Currently industrial robots can swap between multiple tools or end-effectors using a custom-designed tool changer mechanism. A more interesting challenge is enabling robots to use tools designed for humans. This means a human and robot could share the same workspace and tools, to collaboratively complete a task. Or if a robot has to enter an existing environment designed for humans in an emergency situation, it could use whatever tools were laying around.

In the DARPA Robotics Challenge the goal was to develop a robot that could enter a hazardous environment and use any vehicles or tools that were available on-site. We developed a semi-autonomous system that enables the robot to use a cutting tool to cut a hole in a drywall panel (Fig. 1). This report describes the development of software algorithms for perception and dexterous manipulation using the robot CHIMP (CMU Highly Intelligent Mobile Platform) [1].

NASA wants to use robots to perform maintenance tasks in space and has done some research towards enabling robots to grasp, manipulate and use equipment designed for astronauts. In 1996 the Johnson Space Center built the (DART) Dexterous Anthropomorphic Robotic Testbed which had dual 6-DOF arms and touch-sensitive 3-fingered hands. This lead to the development of the anthropomorphic robot Robonaut 1 (R1A and R1B) in 1998, in partnership with DARPA and General Motors. The redesigned version Robonaut 2 (R2) was announced in 2010, as well as an improved 5-fingered hand with increased dexterity [2]. These robots are physically capable of complex manipulation tasks and tool usage, such as tightening a nut using a torque tool, however they are predominantly tele-operated. The human operator uses their visual perception and prior experience to make the robot perform some task.

DARPA funded the ARM (Autonomous Robotic Manipulation) research program in 2010 to build a dual-arm robot and develop new capabilities in object manipulation [3]. The goal was that a human would only provide high-level instructions and the robot would autonomously perform tasks such as inspecting the inside of a gym bag or disarming an IED. Similar to Robonaut, the robot was physically capable of performing these tasks but only when remotely controlled by a human. Ultimately the ARM-S robot demonstrated autonomous use of tools, including turning on a flashlight, drilling a hole with a cordless drill, and installing a small tire and lug nuts.

The task was to use a standard power tool to cut a hole 8″ (20 cm) diameter in a sheet of drywall, also known as gypsum, sheetrock or plasterboard. The minimum hole size was marked by a black circle on a drywall panel (Fig. 2).

Originally the dream was to make the robot break concrete or smash through a wall. However, supporting and manipulating a rock hammer or reciprocating saw (“Sawzall”) requires using two hands, and a combination of strength and dexterity that is challenging for a robot. In contrast, a cordless cut-out tool can be held with one hand and easily cuts through drywall.

For a mobile manipulator to complete this task, the following steps are required:

1. Identify the tool and wall.
2. Determine optimal location for the robot’s base (to pick-up the tool).
3. Move robot.
4. Detect pose of the tool, relative to the robot base.
5. Grasp the tool.
6. Move robot backwards from shelf.
7. Check if it’s possible to activate the tool.
   1. Squeeze trigger and release.
   2. Check motor is turning or light is on.
   3. If can’t actuate trigger, operator chooses next action.
8. Determine optimal location for the robot’s base (to cut hole).
9. Move robot.
10. Compute motion plan to cut hole in wall.
11. Squeeze the tool trigger.
12. Cut hole in wall.
    1. Monitor that cutting bit is following the path.
    2. If not following path, operator chooses next action.
13. Check that cut was successful.
    1. If not, operator chooses next action.
14. Move robot backwards from wall.
15. Discard the tool.

The robot’s software system uses the concept of supervised [4] or shared autonomy [5, 6], whereby control authority is swapped back-and-forth between a human operator and the robot. A sequence of sub-tasks is encoded into an autonomous Agent, implemented using a state machine. The robot will autonomously perform one or more actions, and then stop and wait for input from the operator when required. After the operator makes a decision, the robot continues in autonomous mode.

CHIMP has two stereo pairs (4 cameras) on its head, with one camera used to take a close-up look at the tool being grasped. A panomorphic (anamorphic + fisheye) camera and a 3D spinning LiDAR sensor provide immersive 360° perception. The LiDAR is mainly used for SLAM (Simultaneous Localization and Mapping) and obstacle-detection when the robot moves it arms or operates a vehicle.

The robot has two 7-DOF position-controlled arms with good repeatability sufficient for grasping. However, each joint has a torque-tube and clutch on the output shaft that can affect the arm’s calibration.

Various grippers and robot hands were considered:

• 3-finger Barrett Hand (used in the DARPA ARM-S program).
• 3-finger Robotiq Adaptive Gripper.

as well as prototypes developed during the DARPA ARM-H program:

• 3-finger hand by iRobot, Harvard and Yale.
• 4-finger hand by SRI, Stanford and MEKA Robotics.
• 4-finger hand by Sandia National Lab.

The Robotiq 3-Finger Adaptive Gripper (Fig. 3) was chosen because it is the most robust multi-finger hand that was available. It is made of aluminum and each finger is actuated using internal gears. The gripper was modified for increased durability, including tougher gears and re-routing the wiring through the wrist-center. One limitation of the 3-finger design is that the gripper can’t perform a 2-finger “precision” grasp or a 4-finger “power-cylindrical” grasp, which are useful for grasping objects like a hose or plug.

An anthromorphic hand has more dexterity for manipulation tasks like squeezing a drill trigger, however the mechanical design is less-robust. Typically each finger is actuated by a string or wire cable, with a second cable or spring for the opposing motion. During experiments we found these finger cables would often break and it is tedious to replace and re-tension them.

The Robotiq gripper is attached to each of CHIMP’s arms via a Barrett 6-axis Force/Torque Sensor. The F/T sensor uses a proprietary CAN Bus protocol that is different to the CANOpen protocol used by the robot’s motor controllers, however both protocols use a standard 11-bit CAN message ID and therefore worked correctly on the same 2-wire bus.

Two types of tool were considered for cutting the drywall board: a battery-powered cut-off tool and a power drill.
To make it easier to solve the perception and manipulation problem, a specific product model for each type of tool was selected (Table 1).
When using a tool, you need to turn it on and use some method to verify that its operating (Table 2).

Tool type	Model	Cutting bit	Speed	Trigger	Notes
Demolition hammer	Milwaukee 5337-21 Bosch 11255VSR	Chisel			● Robot could not manipulate tool.
Sawzall / Reciprocating saw	Milwaukee 6538-21 DeWalt DW310	Saw blade			● Robot could not manipulate tool.
Cut-out tool / Rotary spiral saw	DeWalt DC550 KA Photo	DeWalt DW6603 1/8″ drywall bit	26000 RPM	Press button on or off	● Used at DRC Trials in 2013.
Cut-out tool / Rotary spiral saw	DeWalt DCS551 D2 Photo	DeWalt DW6603 1/8″ drywall bit	26000 RPM	Press button on or off	● Used at DARPA Testbed and DRC Finals in 2015.
Driver / Power Drill	DeWalt DCD980 L2 Photo	Morris 13042 5/16″ saw drill bit	Variable 3-speed, max 2000 RPM	Squeeze trigger fully	● Used at DRC Trials in 2013.
Driver / Power Drill	DeWalt DCD771 Photo	Morris 13042 5/16″ saw drill bit	Variable 2-speed, max 1500 RPM	Squeeze trigger fully	● Used at DARPA Testbed in 2015.
Driver / Power Drill	DeWalt DCD995 M2 Photo	Morris 13042 5/16″ saw drill bit	Variable 3-speed, max 2000 RPM	Squeeze trigger fully	● Used at DRC Finals in 2015. ● Motor runs for 5 minutes after squeezing trigger.

The cut-out tool is specifically designed for cutting drywall panels, so it seems like the obvious choice. It has a higher RPM than the power drill and can cut holes in thick drywall using less force at the cutting tip. However, the on/off button is smaller and stiffer than a trigger mechadnism. This makes it more difficult to activate, normally requiring two-handed manipulation. And the cutting bit is very small diameter, meaning it’s prone to fracturing (snapping) if not inserted precisely perpendicular to the drywall panel.

The cordless drill can be operated with only one hand, assuming one has sufficient dexterity to hold the handle and independently squeeze the trigger. The drill has a white LED that illuminates when the trigger is partially or completely pressed. This means the robot would not have to listen for sound or see that the drill chuck is rotating. However the drill has a timer and the motor only runs for 5 minutes before the trigger must be released and pressed again.

The newer model DeWalt DCD995 cordless drill is slightly more bulky than the DCD995. Also the width from the front of the trigger to the back of the handle is 7mm larger. This means that the DCD995 is more difficult to manipulate using the Robotiq gripper.

	Sensor	Method
1	Microphone in palm or wrist	Detect sound of motor
2	Camera	Use a high-resolution camera to check if cutting bit is rotating.
3	Camera	Some power tools have an LED that illuminates when trigger is pressed.
4	Force/Torque sensor in wrist	Push tool bit against wall, check if penetrates the wall.

To verify that a power tool is operating correctly, humans use their ears to check for a familiar sound based on their prior experience. We can tell if the tool chuck is spinning and the battery has sufficient power. We also do a quick visual check to see that the cutting bit is not broken or clogged.

CHIMP wasn’t designed with any “ears” (microphone), which precluded that option. Placing a microphone on the robot’s limbs would require an analogue-to-digital convertor to send a signal back along the data bus, similar to the force-torque sensors located on the wrists.

To capture the spinning motion of a tool chuck requires a high-resolution camera. At low resolutions, a spinning chuck is indistinguisable from a stationary chuck. CHIMP’s vision system was designed to perform heavy hardware-based compression of the camera images so they could be transmitted over a wireless link. There was no capability to request a “one shot” high-resolution image, which precluded that option.

The robot does have the capability to measure forces exerted on the wrist, to check if the tool is working. However, restarting the entire process to re-squeeze the trigger and re-check the tool operation would be very time consuming.

For each type of tool, various grasping strategies were considered (Table 3). Each “grasp” is a method for a specific gripper to pick-up that tool. Some grasps only allow simple pick-and-place behaviour. However, if a gripper has sufficient dexterity then more complex manipulation tasks can be achieved. Alternatively it may be possible to use a re-grasping policy to achieve complex manipulation, but that is not covered here.

	Tool type	No. of hands	Grasp type	Description	Example(s)
1	Rotary Hammer / Power Drill	2	Power – palm, Power – wrap	Two hands: Support tool with one hand, squeeze trigger using forefinger of second hand. (like a human)
2	Cut-out tool	2	Power – wrap, Precision – poke	Two hands: Grasp tool with one hand, press on/off button using finger of second hand. (like a human)
3	Cut-out tool	2	Power – wrap	Two hands: Grasp tool with one hand, poke the on/off button using a stick attached to the second hand.
4	Cut-out tool	1	Power – wrap	Grasp handle, poke the on/off button by pushing it against a protrusion located on robot’s torso.
5	Cut-out tool	1	Power – wrap	Grasp handle using enclosing-power grasp, pressing the on/off button at the same time using a protrusion in palm of gripper.
6	Power Drill	1	Power – medium wrap + finger hook	Grasp handle, independently squeeze the trigger with one finger. (like a human)
7	Power Drill	1	Power – wrap / Cylindrical	Grasp handle using power grasp, squeezing trigger at same time. (there is not sufficient dexterity to independently press the trigger)
8	Power Drill	1	Tripod + finger abduction	Grasp the drill body, squeeze trigger by abduction of 2 fingers in “scissor” motion.

The Robotiq gripper was capable of grasping the power drill using grasp types #7 or #8. With grasping strategy #7 the fingers are closed around the handle using a “power-cylindrical” grasp whilst simultaneously pressing the trigger. Unfortunately the drill shifts in the gripper while the fingers are closing, so it’s difficult to determine the location of the cutting bit without re-localizing the drill pose. Also when the tool stops working after 5 minutes, it’s not possible to re-press the trigger without placing the drill back on a flat surface.

We wanted the ability to independently trigger the drill because it took a long time to compute the motion plan and cut the wall. Therefore we used grasp type #8, whereby the drill is cradled using two spread fingers (see Fig. 7 below), similar to how a human might hold a wine glass. The trigger is pressed by spreading the two fingers sideways in a “scissor” motion (abduction and adduction). This “scissor” action works best if the drill is held upside-down using the 3rd (middle) finger.

In conclusion, it was decided to use the power drill with grasp type #8, and check the tool’s operation by looking for the LED light.

The power drill was selected for the following reasons:

• It has an LED light that indicates when the tool is operating.
• The robot could grasp the drill and actuate the trigger using only one hand. This meant the robot could still accomplish the task if one arm or gripper was broken.
• The cutting bit in the power drill is larger diameter, meaning the strong robot was less likely to snap it.

In order to grasp the power drill or cut the wall, the robot must be positioned such that the arm has sufficient kinematic redundancy to perform each task. The CHIMP robot does not have a waist yaw joint and can’t perform these 2 tasks from the same position. Also the friction between the base tracks and the ground means that the robot can’t rotate in-place, it must turn using a serious of arc motions.

We developed a planning module that computes the optimal 2D pose of the robot’s base relative to an object. For each task, like grasping the drill, the object’s graspability [7] is pre-computed offline and stored in a discrete cost-map. The navigation system them moves the robot to the position with maximum graspability.

However, we found that an experienced operator could tele-operate the robot’s base faster than the autonomous planning and navigation system. The operator is guided by task-specific, graphical guide markers that are overlaid on the robot’s view of the environment.

The object detection system leverages the perception capabilities of the human operator. The operator selects an object 3D model from a list of known objects e.g. “DeWalt Drill DCD980” and uses the mouse to click on a point in the camera image or LiDAR point cloud (Fig. 4a). This knowledge is used to initialize the object pose detector.

This module detects the 3D pose of an object model relative to the position of the robot’s base. The 2D point selected by the human operator is projected onto the LiDAR point cloud using raycasting. Then the system determines the pose of a known 3D model, such as the power drill, within the larger LiDAR point cloud. Each known object is modelled offline as a 3D polygon mesh, which is converted to a 3D point cloud. These are stored in a database. The algorithm (shown below) uses ICP [8] to register the small point cloud of the model within the larger point cloud. Various heuristics were developed [1] to speed-up the model fitting process.

The robot must be positioned such that the LiDAR sensor scans both the length and one end of the power drill, so that the cutting bit is visible in the shape silhouette. One issue was that the orientation and spin-rate of the LiDAR sensor was optimized for SLAM and collision-avoidance. The fast rotation rate of the LiDAR is important to avoid hitting objects when the robot is moving quickly, however it is less accurate for object pose detection. When capturing a point cloud of an object like a tool or door handle, the “lapping” pattern of the LiDAR creates increased distance error and a lower-density pointcloud. In contrast to this, other teams in the DARPA Robotics Challenge used a LiDAR with a “spinning” pattern or a slow “knodding” scan.

We also experimented with storing object pose information in the world frame of the robot’s global pose frame, to build a map of known objects in the environment. However, the robot could not return and re-grasp an object due to error build-up from the localization system.

The power drill is grasped using grasp type #8 (see Table 3 above), as shown in Figs. 5-6.
The grasping policy is a multi-step process, fusing gripper state and force measurements from the wrist to autonomously grasp the drill. The approach was inspired by [9], using the detected forces to “feel” the location of the drill in the presence of object pose uncertainty and arm compliance.
In the initial phase, a “caging” grasp [10] uses gravity to cradle the drill in 2 fingers (see Fig. 6d below), similar to how you might hold a wine glass. The final phase uses a force-closure grasp (see Fig. 6g below), where the drill can’t slip within the gripper fingers.

The grasp policy is implemented in code using a state machine:

The robot squeezes the drill trigger twice; once to check that it can activate the drill, and the second time when ready to cut the wall. The power drill only operates for 5 minutes, after which the trigger has to be pressed again. If grasped poorly, the gripper fingers may have gotten stuck on the trigger, or stuck in a position that they could not press the trigger. In this situation the operator can command the robot to re-grasp the same tool, or try to grasp a different one.

The robot rotates the drill upside-down and releases 2 of the 3 fingers. In this caging grasp [10], the body of the drill is held using the middle finger and the force of gravity, while the other 2 fingers “cage” the side of the handle. These 2 fingers both support the grasp and make the “scissor” motion required to squeeze the trigger.

The sequence of actions to actuate the drill trigger is implemented using a state machine:

After pressing the trigger a white LED light indicates that the tool is operating. The system presents an image to the operator showing the light “before” and “after” squeezing the trigger. It is possible to check the light status automatically using computer vision, however the light is barely perceptible to the human eye when operating the robot outdoors in the sun. An alternative approach was to capture a photo of the tool chuck rotating, however the operator can only see a low-resolution image.

The operator identifies the cut location on the camera image, using the mouse to select the planar region of the wall and the center of the hole. These 2D points are projected onto the LiDAR point cloud using raycasting and any 3D points outside the selected region are discarded. The dominant plane is detected using RANSAC. The operator can manually specify parameters such as the cutting path shape (circle, square, triangle), size, approach direction and cutting angle.

The environment around the robot is modelled using a Voxel Grid generated from the spinning LiDARs. The grid has a coarse resolution with a high update-rate and is used by the motion planner to ensure the arm does not collide with any obstacles. When the perception system detects a known object, the ROI (Region of Interest) is replaced by a high-resolution voxel grid and the object is replaced by a cuboid shaped bounding box. The robot’s geometry is modelled using either spheres or cuboids.

For generating the arm motions we used a probabilistic motion planning approach based on RRTs (Rapidly Expanding Random Trees) [11, 12]. This generates a trajectory of joint-space configurations where each waypoint W_j = {t (sec), joint_angle_i (rad)}. Two variants of the RRT algorithm are used, each with different capabilities and performance (Table 4).

A free-space motion is where the arm needs to move between two predefined joint configurations, or to move the gripper to a specific configuration such as an object’s “pre-grasp” pose. Cartesian motions are where the gripper needs to move in a straight line, such as approaching to grasp an object or lifting the object. Path following motions are used to make the gripper trace out shapes like a circle or square. The pose constraints for cartesian motion and path-following are represented using TSRs (Task Space Regions) [12].

During motion planning, the planner checks for collisions between the robot, the world and any known objects. To manipulate or grasp an object, collision-checking between the gripper and object is briefly disabled. This allows the planner to put the gripper fingers around the drill handle or to move the tool through the wall. When the robot grasps an object like the drill, the object’s bounding box is deleted from the world model and attached to the robot’s geometry tree. The motion planner is aware that the tool can penetrate the cutting zone but must not collide with the surrounding wall, the shelf or the ground.

The planner outputs a feasible joint-space trajectory, however the number of waypoints and the cartesian velocity can vary greatly due to the nature of probabilistic planning. Another issue is that the arm can make big motions that have little effect on the tool point position, such as when the arm needs to pass-through the null-space or “flip” between redundant arm configurations. These issues could be solved by using a constraint-aware post-processing algorithm or an optimization-based motion planner. Currently we use a trajectory re-timing algorithm, whereby we keep the same number of waypoints but scale the velocity and acceleration to maximize trajectory-wide constraints. The overall planning process (constrained RRT + retimer) is executed multiple times in parallel and the best trajectory is kept, based upon minimum number of waypoints and maximum total path velocity.

After cutting the wall, the operator reverses the robot away from the area. The robot plans a collision-free trajectory to place the drill to be within a preset zone above the closest foot. A second motion moves the tool down closer to the ground and the gripper is opened. The bounding box for the drill is deleted from the robot’s collision geometry tree and added to the environment model.

The robot successfully demonstrated autonomous grasping, dexterous manipulation and tool usage. The system was tested both indoors and outdoors, and the robot can reliably cut a hole in the wall, of variable diameter or position, using either arm.

The photos below (Figs. 12-19) show CHIMP cutting a hole in a wall at the DARPA Robotics Challenge, in an outdoor mock-up environment.

For the DARPA Robotics Challenge we chose to cut a square hole because our software implementation of CBiRRT2 [12] had some problems with circular holes, due to how rotation constraints are represented. Currently the planner is constraining the rotation of each arm joint and the tool tip to be +/- 180°. However, the robot’s arm joints are capable of continuous rotation and obviously a rotary tool should be free to rotate around the axis of the cutting bit. We found during experiments that the robot could perform square cuts (straight lines) at higher speed and with lower joint torques.

The one-handed, ambidextrous approach for manipulation proved to be useful in the DRC Finals. On day 1 of the competition the robot fell on its right arm. This caused clutch slippage in the arm joints which meant the arm lost calibration of the “home” configuration. After initially trying the right arm, the operator identified the problem and commanded the robot to use its left arm to successfully perform the task.

Unfortunately we had some other problems at the competition. We had developed and tuned a grasping strategy to manipulate a specific model of power drill. The competition organiser decided to use a different power drill which had different physical dimensions. The DeWalt company seems to release a new model of power drill every year, with an updated design that is different to the old model. The shape of the handle was thicker, and the width from the front of the trigger to the back of the handle was longer, meaning the gripper could not physically wrap around it as well as before. In contrast, the design of the cut-out tool did not change much during the same time period.

One day 1 of the competition, while halfway through cutting the wall, the drill’s spinning chuck started pushing too hard against the wall and started to overload one of the arm joint motors. If a motor overloaded it would cause the joint to “relax”, making it more difficult to resume the task. Therefore the operator intervened and commanded the robot to stop moving the arm, then adjusted the cutting depth, and then resumed the cutting action.

On day 2, our attempt at the overall challenge was less successful. There was a communications problem between the robot and the operators’ control room. There were multiple human operators who had trained for specific tasks, each with a laptop computer. On this day, additional laptops had been connected to the TCP/IP network to watch the action, more than had ever been connected before. The “master” computer needs to send N number of messages to each computer. The finally-tuned network was pushed over its limit, causing status and control messages to arrive “late” or disappear. The robot grasped the drill and then asked the operator for instruction, but then erroneously put the drill down on the ground.

In the DRC Finals, our team (Tartan Rescue) was one of only 2 teams to use the power drill for the wall cutting task. Most teams opted to use the specialised cut-out tool for cutting the hole. The winning team KAIST used a two-handed approach to grasp and activate the cut-out tool. Instead of detecting the exact location of the on switch, the robot tried to press its finger in multiple places until the motor’s noise was detected by a microphone. This two-handed strategy does have a risk, for example the MIT team was unable to use the tool after the robot fell over and broke one arm. In contrast, the second-place IHMC team used a novel one-handed approach, grasping the cut-out tool using a Robotiq gripper that automatically pressed the on switch using a piece of plastic attached to the palm.

Future work will be to:

• Add position encoders to the output shaft of each joint.
• Improve the motion planner to maximize the force at the cutting tip, whilst ensuring the arm joint torques are balanced throughout the arm.
• Develop a method to more accurately set the depth of the cutting bit.
• Implement a hybrid position-force controller to execute the cutting trajectory.
• Experiment with using the cut-out tool.