Python Selenium - Adjust Pause_time To Scroll Down In Infinite Page

I'm trying to scrape all the links available in an infinite page, scrolling down and getting the new links available. However, time.sleep() does not allow to pause the driver for a

Solution 1:

If you do not want to guess each time how long does it take to load the page and set some random seconds to sleep, you can use Explicit Waits. Example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
    element = WebDriverWait(browser, 10).until(
                                    EC.presence_of_element_located((By.ID, "myDynamicElement"))
                                )
except common.exceptions.TimeoutException:
    print('TimeoutException')
finally:
    driver.quit()

# do what you want after necessary elements are loaded

This will solve the problem when time.sleep() becomes too low compared to the refreshing speed of the webpage.