0

I'm working on a project with www.NJMLS.com. I'm trying to extract email of agent name "Edward An":

import dryscrape as d
d.start_xvfb()
br = d.Session()
br.visit('http://www.njmls.com/members/index.cfm?action=dsp.results&city=&county=&nametype=firstname&name=Edward&x=70&y=33')
#open url and try to extract the email of Edward An

How to get email of "Edward An"? If I try:

for x in br.xpath("//*[@class = 'realtor-info']/*")
# selecting all elements in <div class="realtor-info">....</div>
 if "Edward An" in x:
  print x['herf']
#Not responding

I am not getting any response.

What's wrong with my script?

if "Edward An" in x.text()
 print x

This is also not working. Still there is no error and no response.

1
  • I think the issue is if "Edward An" in x. Looks like x is not a text but a web element. Try to change to if "Edward An" in x.text:. Let me know if it works so that I add this answer since currently I am not sure if it is correct.
    – Alexey R.
    Jan 28, 2019 at 11:30

1 Answer 1

0

Have a look at the example here - https://dryscrape.readthedocs.io/en/latest/usage.html

Looks like you need to start the session object by giving it a base url

br = d.Session('http://www.njmls.com')

and then when you call visit give in the URI

br.visit('/index.cfm?action=dsp.results&city=&county=&nametype=firstname&name=Edward&x=70&y=33')

your if statement seems unlikely to work as its implying that X is is a string when its likely to be an element object but it seems there might be more issues.

  1. Does the page require a login ?
  2. Does the server let you visit that page ?
  3. Does the server accept city and county being empty ?

You should be able to add some error handling or use a debugger (pycharm has awesome python debugger integration) or logging statements to see what else is happening.

My guess is that you are not getting the page you expect or it is not loading.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.