Customer Interaction of the future

Convenience store of the future will be for sure the place, where we will meet a lot of robo-helpers. Today we had a chance to demonstrate, how the robots and new services like a SmartBag can make shopping more pleasant for customers.

Here is a video of today’s presentation:

Customer Interaction

So, back to the competition. For us the competition started today with Customer Interaction Task. The aim of this challenge is to show the ideas *of customer interaction in a convenience store in the near future using robot technology. *

The telegram application used for sending a map, handle the payment, get statistics and provide further services strongly coupled with the robot:

In our demo we demonstrated a human robot interaction, detection of calling customer as well as integration of SmartBag and Telegram app in a convenience store.

This is by the way the main entrance of the competition location. Bigger than it could fit in the picture:

In the very beginning TIAGo was looking for a call from customers. This feature is very helpful in noisy and crowded environments like a store. After that he went to the calling person and recognized Ivanna. For this task we trained Ivanna’s face beforehand. Based on customer’s profile of Ivanna, TIAGo guided her to the section ‘Health and Beauty’. Ivanna could not find any shopping bag and triggered help function by pressing a button in the Telegram App. TIAGo reacted immediately: he went to counter and called Lukas, who was a shopping manager in our demo-store. Lukas brought Ivanna a SmartBag, where you can scan items using installed camera and get an invoice using again the Telegram App. Meanwhile Tobias, who was called James, like Bond, James Bond :), entered the store with his suitcase and asked TIAGo to take his trolley bag using a Telegram App. Unfortunately at this moment TIAGo could not localize himself and did not know exactly, where to bring the bag. But we did not panic (ok, Ivanna panicked, OK, Raphael panicked too :D) and restarted the demo after some short time.

After restart TIAGo went to Tobias again and brought his suitcase to designated location. Moreover, he detected Ivanna’s waving and helped Ivanna to carry her very heavy shopping basket. The basket was so heavy, that TIAGo barely could carry it, but he did. He carried the basket to exit and waved Ivanna at the end.

After this Raphael, who moderated our demo, asked volunteers from the audience to enter the store and to call for help by waving. Two girls went inside of the arena and waved. Tiago successfully approached one of them and recognized that she was a new customer in our store. He recommended her to go to the section ‘Health and Beauty’ based on gender recognition. And than the time went out.

Unfortunately, we could not show all of the features we had prepared, like bringing a coffee from counter to the customer. But we could demonstrate our overall idea of the customer interaction in the future convenience store and we hope jury liked it.

So guys, stay tuned, there are two more challenges we are participating in. And keep your fingers crossed for TIAGo.

May I store your suitcase? ARD Tagesschau tomorrow.

For the Customer Interaction task we taught TIAGo to take care of a customers belongings, that means he greets a customer at the entrance of the convenience store and takes over the suitcase to store it in the storage space of the convenience store. After the customer can enjoy shopping without taking care about his belongings. This is just a teaser of what we are doing during the 20 minute demonstration tomorrow. We will present scenarios for how a robot can improve the customers experience in the future.

We began this morning in one of those old fashioned convenience stores and got some breakfast. There are fun ways to enrich this experience in Japan. Here we see a bun:

And this is how the bun looks like after you take a bite:

After we arrived at the competition place, this time just 4 minutes in advance such that the waiting time was quite short, the testings slots where distributed. In those slots our top goal is to test our tasks completely and notice errors and fix them in between the slots.

In advance of taking the following picture we exposed such a bug and are fixing it in joint work. Whats better than pair programming -> Trio programming.

After breakfast and the initial test the ARD Tokio studio came by for a report about the World Robot Summit. We were accompanied for some time and Ivanna and Raphael were interviewed shortly. Later they filmed our test run of the Customer Interaction test.

The report will be broadcasted tomorrow at 15:00 german time in an midday episode of the ARD Tagesthemen or Tagesschau somewhere in between the Bavarian election reports.

Lukas tested the storing and disposal task. As we don’t have the shelf available in between the testing slots TIAGo is handing over the object in this test case. The test is also getting better and better each time.

This is how a competition team looks like. They already participated in last years future convenience challenge and are quite experienced. The robot is looking well and has some advantages e.g. a rotating joint in the wrist and two arms. It was a nice show.

Learning Japanese

Today was a very exciting day. In the morning we were jumping over the fences to save some time but in the end we could enter just by nine. We love the precision of Japanese people, they are even stricter than Germans in many cases. The previous sentence contains compliments for both cultures :).

Ivanna has a little bit of cold and decided to fight it with a vitamin inception through 2 liters of orange juice. That must be healthy. All in all it was a quite good day. TIAGo passed the security checks and cleaned the toilette and served in a future convenience store. We are exited about tomorrow and will go to our capsules early tonight. Later that day there was a surprise welcome reception with a lot of food, beer and sake like drinks involved. During the day we were provided with all of the needed setup information which turned out to be fully in fluent Japanese. After only a few hours of communication and questioning of a lot of people we finally figured out most of it. This was continued by team leader meetings held in Japanese. Luckily we got instant translations by our Happy Mini friends and visual input from a screen. We are coming along quite good and are now motivated to take some Japanese lessons.

TIAGo also starts preparing for the Storing and Disposal task. Here this is our custom prepared container. Note there are markers attached. That’s something we usually don’t do but it eases up things in this task a lot.

Today we are in the mood to post some slightly more technical pictures and we like to present our “recipe” for extracting toilet seats to clean.

  • Step 1) Find planar surface in front of the toilet

  • Step 2) Do some morphology to get a closed form of the toilet seat

  • Step 3) And reduce it to a line

Those lines then serve as input for sampled trajectory points

  • Step 4) Attack!

When you code all day and need to fix urgent things you get creative about the places and poses you code.

To be honest, a domestic environment like RoboCup@Home and the European Robotics League for Service Robots has it’s benefits in that case.

We have a lot of fun here and TIAGo improves every day in big steps. Let’s face some new challenges tomorrow.

Sadly Lukas doesn’t improve like TIAGo and faces new bug all time. This time posing on the official competition toilet.

Tome to go asleep.

World Robot Summit first preparation day

wrs arrival

This morning we arrived at the Big Sight exhibition center in Tokyo. The event is very nice organized. The registration went quite fast and everything was already set. Soon we got our TIAGo and could start testing quite fast. Beside some time synchronization errors everything went smooth this far. Seems like that not all teams are participating in all challenges like we do and many use heavily customized robots for each task. Our approach is to use one mobile robot for all of the tasks. There are some also big companies as well participating. For now we have to leave the competition place and will face new challenges tomorrow. The actual challenges start on Thursday.

preparation After we fixed the synchronization issues we were ready to test.

toilet_cleaning Tobias was testing the toilet cleaning task. TIAGo managed to find the toilet seat and executed a trajectory to clean the seat. Further we managed to find the trash on the ground. Up to now it is a bit tricky because we are using a sponge as a gripper for the prior cleaning task.

toilet_cleaning2 This is TIAGo in action during the cleaning of the toilette.