CORSMAL: Collaborative object recognition, shared manipulation and learning

The CORSMAL challenge:
Audio-visual object classification
for human-robot collaboration

PRIZE: £1,800

Winner: Hengyi Wang, Chaoran Zhu, Ziyin Ma, Changjae Oh (Squids team)
Improving generalization of deep networks for
estimating physical properties of containers and fillings

Sponsored by

Programme (23 June 2022, starts at 12pm BST)
Free registration: https://www.eventbrite.co.uk/347863578327

On the day of the workshop, the event will be streamed live via the CIS YouTube channel, and we will provide a link to Slido for Q&A.

11:45 am UTC+1 / 19:45 pm UTC+9 Virtual windows opening
12:00 pm UTC+1 / 20:00 pm UTC+9 Welcome and introduction: Collaborative Object Recognition, Shared Manipulation and Learning Andrea Cavallaro Queen Mary University of London & Alan Turing Institute
12:15 pm UTC+1 / 20:15 pm UTC+9 Audio-Visual Object Classification For Human-Robot Collaboration Alessio Xompero, Yik Lung Pang, Timothy Patten, Ahalya Prabhakar, Berk Calli, Andrea Cavallaro Queen Mary University of London, U.K.; École polytechnique fédérale de Lausanne, Switzerland; University of Technology Sydney, Australia; Worcester Polytechnic Institute, USA [paper] [arxiv] [video] [poster]
12:30 pm UTC+1 / 20:30 pm UTC+9 Container localisation and mass estimation with an RGB-D camera Tommaso Apicella, Giulia Slavic, Edoardo Ragusa, Paolo Gastaldo, Lucio Marcenaro (Visual team) University of Genoa, Italy [paper] [arxiv] [code]
12:45 pm UTC+1 / 20:45 pm UTC+9 Shared Transformer Encoder with Mask-based Volume Estimation for Filling Mass Estimation Tomoya Matsubara, Seitaro Otsuki, Yuiga Wada, Haruka Matsuo, Takumi Komatsu, Yui Iioka, Komei Sugiura, Hideo Saito (KEIO-ICS team) Keio University, Japan [paper] [code]
13:00 pm UTC+1 / 21:00 pm UTC+9 Improving generalization of deep networks for estimating physical properties of containers and fillings Hengyi Wang, Chaoran Zhu, Ziyin Ma, Changjae Oh (Squids team) University of Birminghan; University College of London; Queen Mary University of London, U.K. [paper] [arxiv] [code]
13:15 pm UTC+1 / 21:15 pm UTC+9 Q&A with the speakers and discussion
13:45 pm UTC+1 / 21:45 pm UTC+9 Challenge leaderboard and award
13:55 pm UTC+1 / 21:55 pm UTC+9 Closing

Leaderboards

Legend
Tasks
Performance scores

Combined CCM test sets

CCM public test set

CCM private test set

Filling level classification

Filling type classification

Container capacity estimation

Container mass estimation

Joint filling type and level classification

Container dimensions and capacity estimation

Filling mass estimation

Solutions of the teams

Select a team to see the details

Squids
KEIO-ICS
Visual
Because It's Tactile
HVRL
Concatenation
NTNU-ERC
Challengers

Team name:	Squids
Team members:	Hengyi Wang	Chaoran Zhu	Ziyin Ma	Changjae Oh
Task 1:	Training of an efficient Convolutional Neural Network, e.g. Mobilenet-v2 with Coordinate Attention, to compute deep features from both the sliding windows of the audio signals converted in log-Mel features and the synchronized RGB frames of the video from the fixed, frontal view. Deep audio features are re-used from the fine-tuning of the Mobilenet-v2 for filling type classification, and concatenated with the deep visual features. Concatenated features are fed into an LSTM unit prior to the final prediction with fully connected layers.
Task 2:	Training of an efficient Convolutional Neural Network, e.g. Mobilenet-v2 with coordinate attention, to compute deep audio features from the sliding windows of the audio signals converted in log-Mel features. Deep audio features are fed into an LSTM unit prior to the final prediction via majority voting.
Task 3:	Fine-tuning of an efficient Convolutional Neural Network, e.g. Mobilenet-v2 with coordinate attention, using i) data augmentations, and ii) variance-consistency evaluation. The model is pre-trained on Task 5 (container dimensions estimation) and takes as input RGB-D image crops of the containers extracted with a YOLO-v5 detector from the fixed, frontal view. Data augmentation: image crops and corresponding capacity labels are randomly and proportionally resized by a scaling factor (drawn by a truncated uniform distribution and keeping the distance value fixed) to account for the intrinsic 2D-3D geometric relationship and the limited training data (labels). The variance-consistency evaluation aims at selecting the model that achieves the highest accuracy and the lowest total variance across the samples from the validation set, based on the observation that predictions of the capacity for the same container should be consistent.
Task 4:	Fine-tuning of an efficient Convolutional Neural Network, e.g. Mobilenet-v2 with coordinate attention, using i) data augmentations, and ii) variance-consistency evaluation. The model is pre-trained on Task 5 (container dimensions estimation) and takes as input RGB-D image crops of the containers extracted with a YOLO-v5 detector from the fixed, frontal view.
Task 5:	Training of an efficient Convolutional Neural Network, e.g. Mobilenet-v2 with coordinate attention, using i) data augmentations, and ii) variance-consistency evaluation. The model is pre-trained on Task 5 (container dimensions estimation) and takes as input RGB-D image crops of the containers extracted with a YOLO-v5 detector from the fixed, frontal view.
Keywords:	Transfer learning, Mobilenet-v2, Coordinate Attention, LSTM, Data augmentation, Variance-consistency evaluation
Paper:	Improving generalization of deep networks for estimating physical properties of containers and fillings
Code:	https://github.com/Noone65536/CORSMAL-Challenge-2022-Squids
Models:	Challenge & only audio, new (Squids-2)

Organisers

Alessio Xompero, Queen Mary University of London (UK)
Andrea Cavallaro, Queen Mary University of London (UK)
Pascal Frossard, École polytechnique fédérale de Lausanne (Switzerland)
Véronique Perdereau, Sorbonne Université (France)
Berk Calli, Worcester Polytechnic Institute (USA)
Timothy Patten, University of Technology Sydney (Australia)
Ahalya Prabhakar, École polytechnique fédérale de Lausanne (Switzerland)
Dounia Kitouni, Sorbonne Université (France)
Yik Lung Pang, Queen Mary University of London (UK)

Team name:	Because It's Tactile
Team members:	Vladimir Iashin	Francesca Palermo	Gokhan Solak	Claudio Coppola
Task 1:	Audio + RGB from all views. GRU(VGGish) + GRU(R(2+1)d [RGB-only]) for each view, and RandomForest(classical audio features)
Task 2:	Audio. GRU(VGGish) and and RandomForest
Task 3:	RGB + IR + Depth (left-side view). LoDE on detector's predictions; if no object was detected, use of the training set's average
Summary:	We sum up logits from all four views obtained from GRU on top of R(2+1)d features to form one prediction for each event, which are, then, averaged with the GRU output on top of VGGish features, and RandomForest predictions on top of 30+ classical audio features (eg mfccs, chromagram, energy, spread). LoDE with Mask R-CNN for object detection (glass, bottle, or book for boxes), in frame 1 and 20 of the videos (view 1, RGB-D-IR) to estimate container capacity. Average of the training set is used if no detection.
Paper:	Top-1 CORSMAL Challenge 2020 submission: Filling mass estimation using multi-modal observations of human-robot handovers
Code:	https://github.com/v-iashin/CORSMAL

Team name:	HVRL
Team members:	Reina Ishikawa	Yuichi Nagao	Ryo Hachiuma	Hideo Saito
Task 1:	Audio. From the prediction model for Task2, intermediate features are extracted and pass through LSTM models.
Task 2:	Audio. Raw audio waveform converted into a log-Mel spectrogram that is cropped into a fixed-size and provided as input to convolutional neural network model with a VGG backbone.
Task 3:	RGB + Depth from view 1: fixed camera on the left side of the manipulator (robot). Mask-RCNN detects the target object (silhouette) and a point cloud is obtained from a selected frame in the video. The volume of the container is then computed by approximating the object shape as a cuboid from the point cloud.
Summary:	N/A
Paper:	Audio-Visual Hybrid Approach for Filling Mass Estimation
Code:	https://github.com/YuichiNAGAO/ICPRchallenge2020

Team name:	Concatenation
Team members:	Qi Liu	Chuanlin Lan	Fan Feng	Rosa Chan
Task 1:	Audio and RGB from all views. Integrate the audio feature learning and the knowledge of container categories via the object detection pre-trained model.
Task 2:	Audio and RGB from all views. Integrate the audio feature learning and the knowledge of container categories via the object detection pre-trained model.
Task 3:	RGB from all views. Sample from the shape distribution based on the prior of container categories
Summary:	The solution is divided into three folds to help the agent shape a rich understanding of the pouring procedure. First, the agent obtains the prior of container categories (cup, glass, box) through the object detection framework. Second, audio features are integrated with the prior to make the agent learn a multi-modal feature space. Finally, the agent infers the distribution of both the container capacity and fluid properties.
Paper:	VA2Mass: Towards the Fluid Filling Mass Estimation via Integration of Vision & Audio
Code:	N/A

Team name:	Organisers
Team members:	Santiago Donaher	Alessio Xompero	Andrea Cavallaro
Task 1:	Audio
Task 2:	Audio
Task 3:	N/A
Summary:	Sound-based model that first identifies the action performed by a person with a container and then determines the amount and type of content using an action-specific classifier. The models consists of three independent CNN classifiers and combines content types and levels into a set of seven feasible classes. Task 1 and Task 2 are jointly performed by the model.
Paper:	https://arxiv.org/pdf/2103.15999.pdf
Code:	https://github.com/CORSMAL/ACC/

Team name:	KEIO-ICS
Team members:	Tomoya Matsubara	Seitaro Otsuki	Yuiga Wada	Haruka Matsuo
	Takumi Komatsu	Yui Iioka	Komei Sugiura	Hideo Saito
Task 1:	(Jointly with Task 2) A model composed of a shared 2-layer Convolutional Neural Network (depthwise convolutional layer and a pointwise convolutional layer), a shared Transformer (3 encoder blocks, each with a 4-head self-attention layer, 2 Layer-Norm layers, 2 Fully Connected layers) , and 2 task-specific Multi-Layer Perceptron heads (2 fully connected layers). The model takes as input audio signals pre-processed and transformed into mel-spectrograms in logarithmic scale, discarding portions of the signal at the beginning and ending with a randomly amount (e.g., 0-40% and 60-100%, respectively).
Task 2:	See Task 1.
Task 3:	The method builts on LoDE, a multi-view iterative fitting approach that estimates the 3D shape (point cloud) of rotationally symmetric objects as a hypothetical cylinder constraint to the binary object masks in two fixed, wide baseline camera views. The method modifies how the radii are reduced across iterations to account for the hand occlusions and smooth the 3D model. For the binary object masks, the method defines a formula to determine the frame and pair of views among the three fixed views (frontal, and 2 sides) where the container is most visible to overcome partial occlusions, low detection confidences, and containers detected too close at the image borders. To estimate the capacity, the volume of the 3D shape is approximated to the Riemman sum of the partial volumes (slicing method) as a canonical frustum.
Task 4:	Regression approach of the container mass with a custom CNN that takes as input a resized image-crop of the container extracted from the best selected frame (see Task 3) and the object mask of the left side, fixed view. The architecture of the CNN has four 3x3 convolutional layers with a padding of size 1 and 3 fully connected (FC) layers. Each convolutional layer is followed by the ReLU activation function, batch normalization, and 2x2 max-pooling layers, and each FC layer by the ReLU activation function and batch normalization layer. The output of the second FC is concatenated with the vector containing the container dimensions (height, width at the bottom, width at the top). To overcome the problem of chipped object masks due to the hand occlusions, mask restoration is addressed by exploiting the symmetrical property of the containers and hence pixel replacement is perfomed after determining the symmetric axis on the image plane.
Task 5:	See Task 3. The container dimensions are estimated as a by-product of the estimated 3D shape.
Keywords:	Convolutional Neural Network, Transformers, Multi-Layer Percpetron, Log-Mel Spectrograms, Best frame selection, LoDE, slicing method, mass regression, object mask restoration.
Paper:	Shared Transformer Encoder with Mask-based Volume Estimation for Filling Mass Estimation
Code:	https://github.com/YuigaWada/CORSMAL2021
Models:	link

Team name:	NTNU-ERC
Team members:	Guilherme Christmann	Jyun-Ting Song
Task 1:	N/A
Task 2:	Audio
Task 3:	Depth from view 3: the fixed camera mounted on the manipulator (robot)
Summary:	Extraction of 40 normalized MFCC features in a window size of 20 ms at 22 kHz, with a maximum length of 30 s, and zero-padding to preserve the same duration across audio data. Filling type classification with a neural network consisting of 2 convolutional layers and 1 linear layer. Regression of the container capacity by extracting a region of interest (ROI) around the object localised in the depth data (view 3) and providing the ROI and its size to a neural network (4 convolutional-batchnorm followed by 3 linear layers). The size of the ROI is concatenated to the feature between the 2nd and 3rd linear layer. Only detections/ROIs up to 700 mm far from the camera, while processing the video backwards, are considered (prior knowledge that the person will extend the arm towards the robot). The closest contour is selected, if multiple detections in a frame.
Paper:	NTNU-ERC Report
Code:	https://github.com/guichristmann/CORSMAL-Challenge-2020-Submission

The CORSMAL challenge:Audio-visual object classificationfor human-robot collaboration

The CORSMAL challenge:
Audio-visual object classification
for human-robot collaboration