Summary extraction of an article using experimental NLP techniques

---

Summary Extraction is the technique for generating meaningful and complete information of the text while focusing on the sections that convey useful information, and without losing the overall meaning of the text. Summary Extraction aims to transform lengthy Text into shortened versions, which makes it easy to understand.

Techniques and Libraries used

• NumPy
• nltk
• Spacy
• newspaper3k
• Regular Expression

Summary Extracting newspaper3k Library.

Using the newspaper Library we can collect the complete text of the Article just by using a predefined function called Article. There are some advantages and disadvantages to using the newspaper3k library.

    >>> from newspaper import Article

    >>> url = “Enter the url”>>> 

    article = Article(url)


      

The complete text of an article was in a variable called article.

Advantages of using newspaper3k

• Predefined functions
• Easy to access the complete text of an Article
• Easy to collect the Author names and Publish dates of an article with authors and publish_date functions.
• Using the newspaper3k library it is very easy to collect the summary of an article using a variable. summary function.

Disadvantages of using newspaper3k

• Cannot download the complete URLs of the website if we run the code n number of times
• The summary collected using newspaper3k was not as expected
• Using a newspaper library becomes difficult during the time of deployment on Heroku. Because Heroku has no corpora keywords as the default installed, we need to provide corpora keywords in a text file while deployment.

How Summary Extracted using newspaper3k Library

Punkt can pick its first 5 tokenized sentences and string them together to form a complete summary. Punkt helps us to divide the complete text into a list of sentences.

To overcome the disadvantages of newspapers 3k and to collect meaningful summaries. Now we are using a Library called spacy.

Before using the techniques of the spacy library we just took the help of the nltk library to convert the complete text into sentences. For this purpose, we are using a function called tokenizer.

• Sent_tokenizer
• Word_tokenizer

Using the spacy Library we are removing the complete string punctuation and unwanted words. which completely harms the data.

.	Full stop or Period	()	Round Brackets
,	Comma	[]	Square Brackets
;	Semi-colon	""	Double Inverted Quamma's
?	Quotation mark	...	Ellipsis marks
!	Exclamation mark	/	Slash
'	Apostrophe	_	Underscore
	Underline	-	Hyphen
:	Colon	@	At sign

Instead of collecting top 5 sentence data as a summary. Using spacy will just give the importance for each word called text normalization and will divide them with the complete sentence length of the data. The highly important data will be collected.

We can collect the summary depending on probability values from the result. 0.3 probability amount of text gives a complete explanation of the summary. After collecting the summary it's better to do the cleaning part for better understanding. Used regular expression and special sequences and metacharacters techniques for data cleaning.

Metacharacters

[]	returns a match if contains patterns/characters specified in []
^	the string starts with given patterns
$	ends with
.	any character except newline
*	Zero or more occurrences
+	one or more occurrences
{}	a specified number of occurrences

Special sequences

\d	If a given string has digits (0-9)
\D	If the given string does not have strings
\w	If a given string has word characters(a-z, A-Z,0-9)
\W	If a given string does not have word characters
\s	If the given string has spaces
\S	If the given string has no spaces

Techniques used for cleaning the data

• Removing unwanted Punctuation for the text
• Removing URLs from the text
• Removing hashtags
• Removing Extra white spaces
• Removing the own mentions

Few Examples

---

---

Back To Blogs

---