An entity can be a word or series of words that consistently refer to the same thing. Every detected entity is classified into a prelabelled category. For example, a NER model might detect the word "London" in a text and classify it as a 'Geography'.
But we need to find the entities from the images. So for this purpose, we need to extract text from the images, so for extracting text we are a technique called OCR.
What is Optimal character recognition(OCR)?
OCR stands for Optical Character Recognition. It is widespread technology to recognize text inside images, such as scanned documents and photos. OCR is used to convert any kind of image containing text like(typed handwritten or printed) into machine-readable text format.
For extracting the text we are using open-source software called tesseract which can be implemented using the Pytesseract package.
Techniques used
- Open CV
- Spacy
- Nltk
- OCR
- Regex
- Pandas
How does Pytesseract work?
Pytesseract detects the images in five different stages where we can collect complete text step by step
Step: 1 -> detect complete page
Step: 2 -> detect individual blocks of the image
Step : 3 -> detect paragraphs
Step: 4 -> detect Line
Step: 5 -> detect words
For detecting entities, we collected individual words from images using step 5 and created a rectangle box on the top of each word using geometric transformations
Same procedure we applied for entire data and collected individual words and saved it in a CSV file.
For detecting the entities we need class labels for each word, so for creating custom entity recognition on images we used a technique called BIO, where B - Token begins an entity, I - Token is inside an entity, O - Token is outside an entity. Using five different labels we made unstructured data into a structured format. Class labels we used to train the models are,
Now for training the custom entity recognition, we selected a spacy pre-trained model, so we convert the data into a spacy format like complete image data and its corresponding words and labels into dictionary type, This process, we applied for entire data and divide the data into the training part and the testing part. For training purposes, we used 50 epochs where at the end of the training we got 94% accuracy for the model, 91% precision, and 90.6% recall. We test around 20 images using our trained model and check the results,
But we got predictions based on the labels we gave for each word while training using the BOI concept, so we find a solution to collect the index position, left and right positions which we get from Pytesseract, for the predicted word and if any word matched BOI format with the same name we can add those detected labels.
Finally, after adding their index positions it works well to detect entities from the images.
Few Predicted Images :
