0

I've tried a couple different routes to get this to work but I've had no luck.

1)

driver= webdriver.PhantomJS(executable_path=r'C:\Users\x\Downloads\phantomjs-2.1.1-windows\phantomjs-2.1.1-windows\bin\phantomjs.exe')
driver.get('https://www.drf.com/race-results')

elements = driver.find_elements_by_xpath("//div[@class='handicappingBottom']")

for item in elements: print(item)

*Returns an empty list

I then read up a bit and feel like the webpage isn't fully loading those items so I tried multiple variants of this:

elements = ui.WebDriverWait(driver, 45).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "handicappingBottom")))

Example of code snippet from webpage:

Ideally I'm trying to scrape the current horse race locations into a list in a program I'm mocking up. Any help at all would be greatly appreciated. I'm new-ish to python and even newer than that to webscraping. Thanks a ton in advance!

  • Welcome. Well, Selenium is one option, but the tool is not designed for web scrapping. I'd probably go for sth. like Scrapy. It might be faster as well as designed for this very purpose, therefore offering better ways for web scrapping. – pavelsaman Dec 20 '19 at 19:39
  • 1
    Will look into this - thank you for the recommendation! – Dane W. Dec 20 '19 at 20:08
1

Hi I am not sure why you are printing that

try the below code if you want to get the displayed text from those elements:

value= driver.find_elements_by_xpath("//div[@class='handicappingBottom']").get_attribute('textContent');

print(value);

if this doesn't work, then try in your code:

for item in elements: print(item.get_attribute('textContent'))

I made it working using protractor:

enter image description here

If you just need any tool for achieving this, then you could use protractor:

Steps to use protractor:

  1. Install nodejs first, it will install npm along with it which is like pip for nodejs : https://nodejs.org/en/download/
  2. now install protractor:

npm install -g protractor

  1. Now create a file called conf.js and add the below code in it and save
 exports.config = {

specs: ['test.js'],

capabilities: {
  'browserName': 'chrome',
},

onPrepare: function(){
    browser.waitForAngularEnabled(false);
    browser.driver.manage().window().maximize();
},

  jasmineNodeOpts: {
    showColors: true, // Use colors in the command line report.
    defaultTimeoutInterval: 10000000,

},
SELENIUM_PROMISE_MANAGER: false,
}
  1. Now create a file test.js and copy-paste below code in it :
describe('Protractor Demo App', function() {
it('should have a title @Regression @Smoke @test', async   function() {
    await browser.get('https://www.drf.com/race-results');
    let a=await element.all(by.xpath('//div[@class="handicappingBottom"]')).getAttribute('textContent');
    console.log(a);
  });
  });
  1. Keep both conf.js and test.js in the same folder and run below command in cmd:

protractor conf.js

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.