I have recently been learning Python and am dipping my hand into building a web-scraper.  It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.  Most of the issues are solvabl...

I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searchi...

I am trying to scrape links from a page that generates content dynamically as the user scroll down to the bottom (infinite scrolling). I have tried doing different things with Phantomjs but not able to gather links beyond first page. Let say the el...

I have HTML webpages that I am crawling using xpath. The etree.tostring of a certain node gives me this string:  <script> <!-- function escramble_758(){   var a,b,c   a='+1 '   b='84-'   a+='425-'   b+='7450'...

I am navigating a site using python's mechanize module and having trouble clicking on a javascript link for next page.  I did a bit of reading and people suggested I need python-spidermonkey and DOMforms.  I managed to get them installed by I am...

There is a website I am trying to pull information from in Perl, however the section of the page I need is being generated using javascript so all you see in the source is:  <div id="results"></div>   I need to somehow pull out th...

I am scraping JSON data from a url. The time is military time and I was wondering if there is a way once I retrieve on the client side to convert it to standard time.  Here is the JSON:  [   {     SaturdayClose: "21:00",     SaturdayOpen: ...

I am trying to scrape data from a website using beautiful soup. By default, this webpage shows 18 items and after clicking on a javascript button "showAlldevices" all 41 items are visible. Beautiful soup scrapes data only for items visible by...

So I'm doing some screen scraping on a site that is very JS heavy. It uses a client side templating engine that renders all the content. I tried using jQuery and that worked in the console, but not on the server (Nodejs), obviously.   I looked at...

I'm currently trying to scrape Google Keyword Tools with CasperJS and PhantomJS (both excellent tools, thanks n1k0 and Ariya), but I can't get it to work.  Here is my current process:   Log in with my Google Account (to avoid captchas in the...