To Author's Page
Right after the Pitch Session of Cooperation Project at the beginning of this semester, where companies introduced their challenges to be solved by us, I was already quite certain that Drooms, a leading IT Service company, was the company I wished to work with on their challenging data science problem in the field of object detection on contract documents. The Semantic Team at Drooms has well defined their challenge for us, in which signature, date, and ideally stamps and missing signature need to be detected on a contract document, and the bounding box of detected objects also needs to be returned.
The project started officially after a warm welcome, an office tour and a kick-off session with the Drooms internal project team where we defined the problem, discussed data sources and IT infrastructure, possible directions of solution as well as coordination styles in our methods.
From the very beginning of the project, we benefited from a well maintained IT infrastructure and technical support from colleagues in Drooms’ team, which enabled us to get hand on directly to the case quite fast. More to our surprise, the in-house data, on which we need to detect through object detection model has been well maintained and labelled from Drooms as well, which saved us a lot of time to clean the data and only required some small extra efforts regarding a pre-processing phase for thousands of in-house contracts containing signatures, dates and stamps.
Although we were equipped with general knowledge of image classification through our deep learning course, object detection is still a fresh field for us, which is mostly utilised in autonomous driving area. We started by doing research for current algorithm development statuses in deep learning fields where classification and localisation of detected objects in an image are returned by the model at the same time, and learnt that algorithms that lead today’s object detection solutions are R-CNN, Fast-CNN, Faster-RCNN and Yolo. Due to faster training time, we decided to give Faster RCNN, the current state-of-art algorithm of real-time object detection a try on our problem to detect signature, and date on contract document (in jpg format) and return bounding box for each detected object.
It was challenging yet exciting experience to truly understand the mechanism of Faster-RCNN, especially how regions in one image are sent to classification layers and how further the bounding box is generated. As the essence of the model, Faster RCNN reuses feature map extracted from convolutional layers following the input of the image, and uses “Regional Proposal Network” consisting of both regressor and classifier to propose regions, which will be later sent to classifier layers for final-class classification (in our case signature and date) and a regressor for bounding box return as input data. With help of Tensorflow API’s implementation of Faster-RCNN and open source wrapper into keras, we managed to train a Faster RCNN Model with keras in python on around 2000 of the objects on Drooms in-house contract data, fine–tuned neural network layers of our model, and have achieved quite satisfying results on signature and date class.
We organized presentations with Drooms to analyse possible use cases on this problem and our model as well as analysing together the false detected and miss-classified objects in testing data and suggested ways of further optimization. We are also excited to learn that Drooms will consider add our solution to their internal IT service.
I am very grateful for ever having this chance to work on this cooperation-project with Drooms and its sincere and professional project team which largely enriched my knowledge in object detection and working on real business cases in general. In addition to be involved in a well-structured project and received lots of supports and guidance from colleagues in Drooms’ team, I also quite enjoy the atmosphere working there and it will for sure be an unforgettable experience in my master study journey. I highly recommend participating in the company-cooperation project.
Frankfurt School gGmbH
Adickesallee 32-34
60322 Frankfurt am Main
