Using Phantomjs For Dynamic Content With Scrapy And Selenium Possible Race Condition

First off, this is a follow up question from here: Change number of running spiders scrapyd I'm used phantomjs and selenium to create a downloader middleware for my scrapy project.

Solution 1:

try using webdriver to interface with phantomjs https://github.com/brandicted/scrapy-webdriver