Get BBC News Search Results

Let’s get the BBC News search results.


You can view the news search results page for a search term by linking it later, as shown above.


Extract desired data

As shown in the previous example, if you look at the source through the Developer Tools (Ctrl+Shift+I or F12), you will find the news title part.

You can see that the text in the ‘.search-results’ class > li tag > article tag > div tag > h1 tag > a tag in order.


Example

import requests
from bs4 import BeautifulSoup


def print_news_result(sw):
    url = 'https://www.bbc.co.uk/search?q=' + sw
    r = requests.get(url)
    html = r.content
    soup = BeautifulSoup(html, 'html.parser')
    titles_html = soup.select('.search-results > li > article > div > h1 > a')

    for i in range(len(titles_html)):
        title = titles_html[i].text
        link = titles_html[i].get('href')
        print(str(i + 1) + '. ' + title + ' (' + link + ')')


print_news_result('python')

The title and link of the bbc news search results for the search term ‘python’ will be printed.


Description

def print_news_result(sw):
   url = 'https://www.bbc.co.uk/search?q=' + sw
   r = requests.get(url)
   html = r.content
   soup = BeautifulSoup(html, 'html.parser')
   titles_html = soup.select('.search-results > li > article > div > h1 > a')

When you enter a search term in a function, the search term is linked to the url address, and on that url page, the html part for the news title is imported in the form of the list.


for i in range(len(titles_html)):
    title = titles_html[i].text
    link = titles_html[i].get('href')
    print(str(i + 1) + '. ' + title + ' (' + link + ')')

The title and link address of the news are output based on the number of news.


Results

The output results are shown below.

1. Monty Python's Flying Circus (https://www.bbc.co.uk/programmes/p00fp03l)
2. Monty Python's Flying Circus: Series 1 (https://www.bbc.co.uk/programmes/p00fz14r)
3. Python at 50: Silly Talks and Holy Grails (https://www.bbc.co.uk/programmes/m0008bzf)
4. Monty Python's Flying Circus: Series 1: Whither Canada (https://www.bbc.co.uk/programmes/p00gz89w)
5. Monty Python: Almost The Truth - The BBC Lawyer's Cut (https://www.bbc.co.uk/programmes/b00n7sf3)
6. Six-foot python abandoned in Burnley Tesco car park (https://www.bbc.co.uk/news/uk-england-lancashire-49908576)
7. Python Lee Jackson (https://www.bbc.co.uk/music/artists/3c86a819-d7b3-415d-86b8-7c7a880c288e)
8. Comedy Club Extra: Monty Python at 18,262 Days (https://www.bbc.co.uk/programmes/m00093mg)
9. Emma Britton: Bristol Bus Boycott, Monty Python & toilets (https://www.bbc.co.uk/programmes/p074ljch)
10. Monty Python (https://www.bbc.co.uk/music/artists/4a5c8526-f8ec-43f1-97af-49722ad88394)

Print out the titles and links of the news in order that contains the search term ‘python’.

Prev