What happened since ...

It has been some time since posting the last time here on the blog. But a lot stuff happened. First we taught TIAGo to bring us beer for the NVIDIA Jetson Developer Challenge. The video we posted got quite some attention on Youtube (for our standard :)) and some news reports followed on popular pages like (CNN Chile, IEEE Spectrum, Sputnik, RTV, mail.ru …). Sadly we didn’t make it to the finals, but still we had a lot of fun during the implementation.

Best you see the resulting video where TIAGo autonomously opens the fridge and takes a beer to deliver it (click to play):

Bring me a beer from the fridge

Brining beer sounds easy for us to do. But when you start translating it in order to teach it to a robot it’s so much more. First the robot needs to learn the different types of beer and localize it. Same for the handle. The robot needs to be aware of where it actually is located right now and it also needs to be aware of it’s current state. That means it has to now it’s current joint angles of each joint at a high frequency in order to execute arm path to grasp the handle and then the beer. And and and …

Opening the fridge
Opening the fridge.
Getting the beer
Getting the beer.
Closing the fridge
Closing the fridge.

For localizing the beer we developed a semantic segmentation network, trained with quite some images of the different types of beer and another network trained for detecting the handle of the Fridge. Depending on the current state of the robot we execute a different model, therefore are able to set our focus of search depending on what we are looking for. Based on these estimates we programmed motions for gripping the beer and opening as well as closing the fridge.

Prediction result
Prediction result.

Then next, as a follow up to our attendance in the European Robotics League in Edinburgh we were invited to a studio talk and one day later Campus TV came by to report about our team and filmed the beer delivery.

Currently Raphael is in Tampere at the European Robotics Forum and presented the highlights from the last European Robotics League. This evening the award ceremony of the European Robotics League will be held.

European Robotics League Highlights from homer

Impressing the press

Teams at the tournament
Teams at the tournament.

First up in the morning was the General Purpose Service Robot task, meaning that Lisa basically has to understand a sequence of commands and execute them afterwards. She did quite fine. There were 5 commands given each containing 3 subcommands. First she had to bring the salt from the kitchen table to Mauro in the hall. Then she had to find a person close the bedside table and tell what time it is. She told the time in the Central European Timezone, therefore she was one our off. Afterwards she should bring the soup from the bookcase to a person in the hall. This is where she failed, cause she couldn’t reach the sour on top of the bookcase. Next up was again finding a person in the hall and answer a question. She found a person but in the kitchen but answered the question immediately. And done, GPSR was done quite fast.

Raphael gets interviewed by the press
Raphael gets interviewed by the press.

Perfect, because the press was coming meaning that a photographer from the university and a tv-station came by to catch some nice photos and a report. They were filming some coverage from all teams. So they showed the Tiagos from HEARTS Bristol and IRI from Barcelona and Lisa also got the chance to show what she is able to do. So she decided to go to the Couch and bring a person sitting there a drink and reading glasses (the latter one had to be handed over because it’s very hard to grip them). Afterwards they did interviews and Raphael was also interviewed. In the end we made some nice group picture with all teams and robots doing some posing together.

Handover an object
Handover an object.

This time we got a Whiskey handed over by the organizers. A 10 year old Whiskey branded by the Heriot University, what a great present, we will come again if possible also because of the Whiskey, but in general it was a nice tournament. All teams did very well for almost all tasks, and we got the chance to improve the abilities of Lisa here and there.

The Homer ERL team from Edinburgh at photoshooting
The Homer ERL team from Edinburgh at photoshooting.

Today was a good day

Today we met a bit earlier in the morning to keep up with all the tasks, everything except the General Purpose Task was on the list.

Table for counting the runs done
Table for counting the runs done.

First up was Catering For Granny. Lisa understood all commands but failed to grip the cardboard box, because it was to big for her grippers. Nevertheless Lisa’s happy owner could control the smart home environment through Lisa.

Then the Welcoming Visitors task took place, where Lisa recognized in three rounds all visitors correctly and also handles them properly.

Now we have a robot that is capable of welcoming visitors, controlling a smart home and bringing stuff on demand. By the way in the navigation task Lisa also showed her following capabilities and of course demonstrated how she can navigate in the apartment where also dynamic changes could be trained by a person. An overall useful robot isn’t it? For this tasks maybe, but there are things missing. May in the future she starts adapting behaviour that she observed at persons in order to be able to learn more special tasks like preparing a meal, unloading the dishwasher and …. We are working on it.

Lisa starts gripping and everybody took out his smartphone to film it
Lisa starts gripping and everybody took out his smartphone to film it.

Tomorrow there will be some press around reporting about the event. We are looking forward to this. Lisa is already stretching her arm in order to grip something tomorrow with all the cameras around. Usually she gets very nervous when that many people are around but we’ll see tomorrow.

Beside the press coming by there is just the General Purpose Task taking place, where Lisa has to show all capabilities that she is able to do in any order and there are some more testing slots where teams can decide whatever the want to get tested in.

That’s it for today. We are hungry and are looking forward to the social event taking place on this very special Scottish day, which is also called Burns Day named after the scot poet Robert Burns.

Competition Day 2 - Getting To Know My Home and Welcoming Visitors

Today we arrived in time for Getting To Know My Home in the morning and Welcoming Visitors in the afternoon.

Lisa in the testbed
Lisa in the testbed.

The first round of Getting to Know My Home was just medium, there were was a lamp and a plant added that during our preparation was not mentioned and have been recently added.

We soon did a second and third round. Those went as expected. Lisa understood everything and learned changes in the environment during the runs. Some objects where moved and a semantic map annotating the changes was created. In the second part Lisa got a command to actually move the objects moved to furniture that also was moved. There was just one little problem. Lisa tried to go very close to the kitchen chair but couldnt reach the goal.

Shot from the camera at the door of the testbed
Shot from the camera at the door of the testbed.

Next up was the Welcoming Visitors task where Lisa awaited random people after ringing the door bell. People could be a doctor, a delivery man, a postman and a thief. Of course Lisa should not let in the thief but handle the other people in a way that they can do their task. For instance the doctor should be guided to the owner. The delivery man should place the parcel somewhere and the delivery man should bring the food to the kitchen.


Now we are preparing for tomorrow.