Scores from yesterday

We got the scores of the challenges yesterday very late. So just a short blog containing summing up the scores:

FBM3 aka Speech Recognition:

Team            FCR     AC      WER     SR Acc. SPQReL          70%     91%     0.06    62% homer           60%     89%     0.05    68% b-it-bots       37%     86%     0.06    62%

Now all teams are using google speech recognition. We did some preprocessing to improve the results a bit. This gains us a slightly higher accuracy of 68% in comparison to the 62% from the raw results. It seems like we have some frame errors yielding lower Full Command Recognition (FCR) results and the Action Classification (AC). WER determines the Word Error Rate.

TBM5 aka General Purpose Service Robot:

  • run 1  3A/0P  3:20
  • run 2  3A/0P  3:30
  • run 3  2A/0P  2:15
Lisa following Luca out of the apartment
Lisa following Luca out of the apartment.

TBM4 aka Navigation:

homer

  • run 1: 5A/0P  7:40
  • run 2: 5A/0P  9:00
  • run 3: 7A/0P  8:50

SPQReL

  • run 1:  3A/0P  5:30
  • run 2:  3A/0P  6:40
  • run 3:  1A/2P  6:30

TBM1 aka Getting to know my home:

homer

  • run1: 6A/2P 10:00
  • run2: 6A/1P 10:00
  • run3: 3A/1P  4:30

SPQReL

  • run1: 3A/3P  5:30
  • run2: 2A/2P  6:40
  • run3: 3A/3P  5:50
Today the Object Perception functionality benchmark is up. Lukas is preparing it
Today the Object Perception functionality benchmark is up. Lukas is preparing it.