Scrapy Error: TypeError: __init__() Got An Unexpected Keyword Argument 'deny'

I've put together a spider and it was running as intended until I've added the keyword deny into the rules. This is my spider : from scrapy.linkextractors import LinkExtractor fro

Solution 1:

deny is supposed to be passed as an argument to LinkExtractor, but you put it outside those parentheses and passed it to Rule. Move it inside, so you have:

rules = (Rule(LinkExtractor(allow=[r'/*'], deny=('blogs/*', 'videos/*', )),
                  callback='parse_html'), )

__init__ is the method that is called when you pass arguments when instantiating a class, as you did here with the Rule and LinkExtractor classes.

Solution 2:

Er, wouldn't it be the function you actually passed deny to? That would be Rule. The __init__ is the internal method of the Rule class that is called when you construct the object.

The examples I found online with this pass deny= to the LinkExtractor, not the Rule. So you want:

rules = (Rule(LinkExtractor(allow=('/*',),
                            deny=('blogs/*', 'videos/*', )),
              callback='parse_html'), )