The ongoing challenge has been updated with additional tasks and performance scores.
Click here for more info and results.

Description

A major challenge for human-robot cooperation in household chores is enabling robots to predict the properties of previously unseen containers with different fillings. Examples of containers are cups, glasses, mugs, bottles and food boxes, whose varying physical properties such as material, stiffness, texture, transparency, and shape must be inferred on-the-fly prior to a pick-up or a handover.

The challenge focuses on the estimation of the capacity and mass of containers, as well as the type, mass and percentage of the content, if any. Participants will determine these physical properties of a container while it is manipulated by a human. Containers vary in their physical properties (shape, material, texture, transparency, and deformability). Containers and fillings are not known to the robot: the only prior information available is a set of object categories (glasses, cups, food boxes) and a set of filling types (water, pasta, rice). Previous and related challenges/benchmarks, such as the Amazon Picking Challenge or the Yale-CMU-Berkeley (YCB) benchmark, focus on tasks where robots interact with objects on a table and without the presence of a human, for example grasping objects, table setting, stacking cups, or assembling/disassembling objects.

Advancements in this research field will help the integration of smart robots into people's lives to perform daily activities involving objects and handovers. For example, this is a step towards supporting people with disabilities in performing everyday activities.

The CORSMAL Containers Manipulation dataset

CORSMAL distributes an audio-visual-inertial dataset of people interacting with containers, for example while pouring a filling into a glass or shaking a food box. The dataset is collected with four multi-sensor devices (one on a robotic arm, one on the human chest and two third-person views) and a circular microphone array. Each device is equipped with an RGB camera, a stereo infrared camera and an inertial measurement unit. In addition to RGB and infrared images, each device provides synchronised depth images that are spatially aligned with the RGB images. All signals are synchronised, and the calibration information for all devices, as well as the inertial measurements of the body-worn device, is also provided.

Camera 1	Camera 2	Camera 3	Camera 4	Audio

The dataset consists of 1140 audio-visual-inertial recordings of people interacting with (15) containers, using 4 cameras (RGB, depth, and infrared) and a 8-element circular microphone array. Containers are either empty (0%) or filled at 2 different levels (50%, 90%) with 3 different types of content (water, pasta, rice). For example, people can pour a liquid in a glass/cup or shake a food box.

The dataset is split into training set (9 containers), public testing set (3 containers), and private testing set (3 containers). We provide to the participants with the annotations of the capacity of the container, filling type, filling level, the mass of the container, and the mass of the filling for the training set. No annotations will be provided for public testing set while private testing set will not be released to the participants. The containers for each set are evenly distributed among the three container types.

The CORSMAL Challenge includes three scenarios with an increasing level of difficulty, caused by occlusions or subject motions:

• Scenario 1. The subject sits in front of the robot, while a container is on a table. The subject pours the filling into the container, while trying not to touch the container, or shakes an already filled food box, and then initiates the handover of the container to the robot.

• Scenario 2. The subject sits in front of the robot, while holding a container. The subject pours the filling from a jar into a glass/cup or shakes an already filled food box, and then initiates the handover of the container to the robot.

• Scenario 3. A container is held by the subject while standing to the side of the robot, potentially visible from one third-person view camera only. The subject pours the filling from a jar into a glass/cup or shakes an already filled food box, takes a few steps to reach the front of the robot and then initiates the handover of the container to the robot.

Webpage to access and download the dataset.

Note that participants will receive the password to access the public testing set after completing the registration form and sending the request to the organisers.

Tasks

Prior to a human-to-robot handover, the robot must determine on-the-fly the weight of a container (e.g. the amount and type of content). This estimation will enable the robot to apply the correct force when grasping it, avoiding slippage, crashing the container or spilling its content.

We define the weight of the object handled by the human as the sum of the mass of an unseen container and the mass of the unknown filling within the container. However, we focus the challenge only towards the estimation of the mass of the filling (overall task). To estimate the mass of the filling, we expect a perception system to reason on the capacity of the container and to determine the type and amount of filling present in the container.

Therefore, we identify three tasks for the participants:

• Task 1 (T1) Filling level classification. Containers can be either empty or filled with an unknown content at 50% or 90% of the whole capacity of the container. For each recording related to a container, the goal is to classify the filling level. There are three classes: 0 (empty), 50, (half full), and 90 (full).
• Task 2 (T2) Filling type classification. Containers can be either empty or filled with an unknown content. For each recording related to a container, the goal is to classify the type of filling, if any. There are four filling type classes: 0 (empty), 1 (pasta), 2 (rice), 3 (water).
• Task 3 (T3) Container capacity estimation. Containers vary in shape and size. For each recording related to a container, the goal is to estimate the capacity of the container.

For each recording related to a container, we then compute the filling mass estimation using the estimations of filling level from T1, filling type from T2, and container capacity from T3, and using the prior density of each filling type per container. The density of pasta and rice is computed from the annotation of the filling mass, capacity of the container, and filling level for each container. Density of the water is 1 g/mL. The formula selects the annotated density for a container based on the estimated filling type.

Starting kit and documentation

Evaluation toolkit + script to pre-process the dataset
[code]

Vision baseline for CORSMAL Benchmark: a vision-based algorithm, part of a larger system, proposed for localising, tracking and estimating the dimensions of a container with a stereo camera.
[paper][code][webpage]

LoDE: a method that jointly localises container-like objects and estimates their dimensions with a generative 3D sampling model and a multi-view 3D-2D iterative shape fitting, using two wide-baseline, calibrated RGB cameras.
[paper][code][webpage]

Mask R-CNN
[paper][code]

SiamMask
[paper][code]

ResNet-18 (network available in PyTorch)
[paper][code]

Mask R-CNN + ResNet-18: Vision baseline for filling properties estimation. Independent classification of filling level and filling type using a re-trained ResNet-18 and a single RGB image crop extracted from the most confident instance estimated by Mask R-CNN in the last frame of a video. The baseline works only with glasses and cups, and fails with non-transparent containers (extra class opaque). We refer to this baseline as Mask R-CNN+RN18 in the leadeboard (run for each camera view independently).
[code] (COMING SOON)

Additional references
[document]

Leaderboard

The evaluation is based on the results of both testing sets and participants will be ranked across all configurations. To calculate the estimation of the filling mass (overall task), the organisers use results from the random case if one (or two) of the tasks is (are) not submitted by a participant team.

Note: The score for the overall task is not a linear combination of the scores outputted for Task 1, Task 2, and Task 3, but it takes into consideration the formula for computing the filling mass based on the estimations of each task for each configuration. This means that a method with lower Task 1, Task 2 and Task 3 scores can obtain a higher Overall score compared to other methods because the performance on each run is more accurate in general.
More details in the technical document.

The organisers will declare the winner of the challenge based on the score on the overall task. The best-performing entries will be presented at the conference venue. Selected participants will be invited to co-author a paper to discuss and analyse the research outcomes of the challenge.

Overall task: Filling mass estimation

Task 1: Filling level classification

Task 2: Filling type classification

Task 3: Container capacity estimation

Legend:
- View 1: view from the fixed camera on the left side of the manipulator
- View 2: view from the fixed camera on the right side of the manipulator
- View 3: view from the fixed camera mounted on the manipulator (robot)
- View 4: view from the moving camera worn by the demonstrator (human)
- A: audio modality
- RGB: colour data
- D: depth data
- IR: infrared data from narrow-baseline stereo camera
- ZCR: Zero-crossing rate
- MFCC: Mel-frequency cepstrum coefficients
- ZCR: Zero-crossing rate
- A5F: Audio 5 features (MFCC, chromogram, mel-scaled spectrogram, spectral contrast, tonal centroid)
- kNN: k-Nearest Neighbour classifier
- SVM: Support Vector Machine classifier
- RF: Random Forest classifier
- PCA: Principal component analysis

Submissions

Team name:
Team members:
Task 1:
Task 2:
Task 3:
Summary:
Paper:
Code:

Team name:	Concatenation
Team members:	Qi Liu	Chuanlin Lan	Fan Feng	Rosa Chan
Task 1:	Audio and RGB from all views. Integrate the audio feature learning and the knowledge of container categories via the object detection pre-trained model.
Task 2:	Audio and RGB from all views. Integrate the audio feature learning and the knowledge of container categories via the object detection pre-trained model.
Task 3:	RGB from all views. Sample from the shape distribution based on the prior of container categories
Summary:	The solution is divided into three folds to help the agent shape a rich understanding of the pouring procedure. First, the agent obtains the prior of container categories (cup, glass, box) through the object detection framework. Second, audio features are integrated with the prior to make the agent learn a multi-modal feature space. Finally, the agent infers the distribution of both the container capacity and fluid properties.
Paper:	VA2Mass: Towards the Fluid Filling Mass Estimation via Integration of Vision & Audio
Code:	N/A

Team name:	HVRL
Team members:	Reina Ishikawa	Yuichi Nagao	Ryo Hachiuma	Hideo Saito
Task 1:	Audio. From the prediction model for Task2, intermediate features are extracted and pass through LSTM models.
Task 2:	Audio. Raw audio waveform converted into a log-Mel spectrogram that is cropped into a fixed-size and provided as input to convolutional neural network model with a VGG backbone.
Task 3:	RGB + Depth from view 1: fixed camera on the left side of the manipulator (robot). Mask-RCNN detects the target object (silhouette) and a point cloud is obtained from a selected frame in the video. The volume of the container is then computed by approximating the object shape as a cuboid from the point cloud.
Summary:	N/A
Paper:	Audio-Visual Hybrid Approach for Filling Mass Estimation
Code:	https://github.com/YuichiNAGAO/ICPRchallenge2020

Team name:	Because It's Tactile
Team members:	Vladimir Iashin	Francesca Palermo	Gokhan Solak	Claudio Coppola
Task 1:	Audio + RGB from all views. GRU(VGGish) + GRU(R(2+1)d [RGB-only]) for each view, and RandomForest(classical audio features)
Task 2:	Audio. GRU(VGGish) and and RandomForest
Task 3:	RGB + IR + Depth (left-side view). LoDE on detector's predictions; if no object was detected, use of the train set's average
Summary:	We sum up logits from all four views obtained from GRU on top of R(2+1)d features to form one prediction for each event, which are, then, averaged with the GRU output on top of VGGish features, and RandomForest predictions on top of 30+ classical audio features (eg mfccs, chromagram, energy, spread). LoDE with Mask R-CNN for object detection (glass, bottle, or book for boxes), in frame 1 and 20 of the videos (view 1, RGB-D-IR) to estimate container capacity. Average of the train set is used if no detection.
Paper:	Top-1 CORSMAL Challenge 2020 submission: Filling mass estimation using multi-modal observations of human-robot handovers
Code:	https://github.com/v-iashin/CORSMAL

Team name:	NTNU-ERC
Team members:	Guilherme Christmann	Jyun-Ting Song
Task 1:	N/A
Task 2:	Audio
Task 3:	Depth from view 3: the fixed camera mounted on the manipulator (robot)
Summary:	Extraction of 40 normalized MFCC features in a window size of 20 ms at 22 kHz, with a maximum length of 30 s, and zero-padding to preserve the same duration across audio data. Filling type classification with a neural network consisting of 2 convolutional layers and 1 linear layer. Regression of the container capacity by extracting a region of interest (ROI) around the object localised in the depth data (view 3) and providing the ROI and its size to a neural network (4 convolutional-batchnorm followed by 3 linear layers). The size of the ROI is concatenated to the feature between the 2nd and 3rd linear layer. Only detections/ROIs up to 700 mm far from the camera, while processing the video backwards, are considered (prior knowledge that the person will extend the arm towards the robot). The closest contour is selected, if multiple detections in a frame.
Paper:	NTNU-ERC Report
Code:	https://github.com/guichristmann/CORSMAL-Challenge-2020-Submission

Team name:	Challengers
Team members:	Neeharika	Krishna	Bakhtawar	Dinesh
Task 1:	Audio
Task 2:	Audio
Task 3:	N/A
Summary:	Sound-based classification of filling type and level: After suppressing the noise in each audio signal via spectral gating, the absolute value of the Short-Time Fourier Transform (STFT) is extracted as input feature for a classifier based on a 5-layer fully connected neural network, trained with Adam optimizer and dropout on the last layer to reduce overfitting.
Paper:	N/A
Code:	N/A

Team name:	Organisers
Team members:	Santiago Donaher	Alessio Xompero	Andrea Cavallaro
Task 1:	Audio
Task 2:	Audio
Task 3:	N/A
Summary:	Sound-based model that first identifies the action performed by a person with a container and then determines the amount and type of content using an action-specific classifier. The models consists of three independent CNN classifiers and combines content types and levels into a set of seven feasible classes. Task 1 and Task 2 are jointly performed by the model.
Paper:	https://arxiv.org/pdf/2103.15999.pdf
Code:	https://github.com/CORSMAL/ACC/