'nonetype' Object Has No Attribute 'text' In Beautifulsoup

I am trying to scrape Google results when I search 'What is 2+2', but the following code is returning 'NoneType' object has no attribute 'text'. Please help me in achieving the req

Solution 1:

A big gotcha when parsing websites is that the source code can look very different in your browser when compared to what requests sees. The difference is javascript, which can hugely modify the DOM in a javascript capable browser.

I'd suggest 3 options:

1. use requests to get the page, and then examine it closely - does that tag exist when the page is retrieved by a non-js enabled agent?
2. use https://www.seleniumhq.org/ as your agent - it's essentially a fully featured browser that you can control programatically, inc w/ python.
3. use google's search API instead of trying to scrape the html

Solution 2:

Next time use the query string exactly as it is.

import requests
from bs4 import BeautifulSoup
search="2%2B2"link="https://www.google.com/search?q="+search
headers={'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'}
source=requests.get(link,headers=headers).text
soup=BeautifulSoup(source,"html.parser")
answer=soup.find('span',id="cwos")
print(answer.text)

Output:

 4  

Visit these urls - they do not return the same result

https://www.google.com/search?q=What+is+2+2

https://www.google.com/search?q=2%2B2

https://www.google.com/search?q=2+2

Solution 3:

When you run the code, you might encounter an AttributeError:

shell: AttributeError: 'NoneType' object has no attribute 'text'

If that’s the case, then take a step back and inspect your previous results. Were there any items with a value of None? You might have noticed that the structure of the page is not entirely uniform. There could be an advertisement in there that displays in a different way than the normal job postings, which may return different results

Reference:https://realpython.com/beautiful-soup-web-scraper-python/#extract-text-from-html-elements.