Having trouble scraping data from website with Selenium

Question

I've tried a couple different routes to get this to work but I've had no luck.

1)

driver= webdriver.PhantomJS(executable_path=r'C:\Users\x\Downloads\phantomjs-2.1.1-windows\phantomjs-2.1.1-windows\bin\phantomjs.exe')
driver.get('https://www.drf.com/race-results')

elements = driver.find_elements_by_xpath("//div[@class='handicappingBottom']")

for item in elements: print(item)

*Returns an empty list

I then read up a bit and feel like the webpage isn't fully loading those items so I tried multiple variants of this:

elements = ui.WebDriverWait(driver, 45).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "handicappingBottom")))

Ideally I'm trying to scrape the current horse race locations into a list in a program I'm mocking up. Any help at all would be greatly appreciated. I'm new-ish to python and even newer than that to webscraping. Thanks a ton in advance!

Answer 1

Hi I am not sure why you are printing that

try the below code if you want to get the displayed text from those elements:

value= driver.find_elements_by_xpath("//div[@class='handicappingBottom']").get_attribute('textContent');

print(value);

if this doesn't work, then try in your code:

for item in elements: print(item.get_attribute('textContent'))

I made it working using protractor:

If you just need any tool for achieving this, then you could use protractor:

Steps to use protractor:

1. Install nodejs first, it will install npm along with it which is like pip for nodejs : https://nodejs.org/en/download/
2. now install protractor:

npm install -g protractor

4. Now create a file called conf.js and add the below code in it and save

 exports.config = {

specs: ['test.js'],

capabilities: {
  'browserName': 'chrome',
},

onPrepare: function(){
    browser.waitForAngularEnabled(false);
    browser.driver.manage().window().maximize();
},

  jasmineNodeOpts: {
    showColors: true, // Use colors in the command line report.
    defaultTimeoutInterval: 10000000,

},
SELENIUM_PROMISE_MANAGER: false,
}

5. Now create a file test.js and copy-paste below code in it :

describe('Protractor Demo App', function() {
it('should have a title @Regression @Smoke @test', async   function() {
    await browser.get('https://www.drf.com/race-results');
    let a=await element.all(by.xpath('//div[@class="handicappingBottom"]')).getAttribute('textContent');
    console.log(a);
  });
  });

5. Keep both conf.js and test.js in the same folder and run below command in cmd:

protractor conf.js