I have recently been learning Python and am dipping my hand into building a web-scraper.  It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.  Most of the issues are solvabl...

I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searchi...

I am trying to scrape links from a page that generates content dynamically as the user scroll down to the bottom (infinite scrolling). I have tried doing different things with Phantomjs but not able to gather links beyond first page. Let say the el...

I have HTML webpages that I am crawling using xpath. The etree.tostring of a certain node gives me this string:  <script> <!-- function escramble_758(){   var a,b,c   a='+1 '   b='84-'   a+='425-'   b+='7450'...

I am scraping JSON data from a url. The time is military time and I was wondering if there is a way once I retrieve on the client side to convert it to standard time.  Here is the JSON:  [   {     SaturdayClose: "21:00",     SaturdayOpen: ...

So I'm doing some screen scraping on a site that is very JS heavy. It uses a client side templating engine that renders all the content. I tried using jQuery and that worked in the console, but not on the server (Nodejs), obviously.   I looked at...

I'm currently trying to scrape Google Keyword Tools with CasperJS and PhantomJS (both excellent tools, thanks n1k0 and Ariya), but I can't get it to work.  Here is my current process:   Log in with my Google Account (to avoid captchas in the...

I need to do some web scraping. After playing around with different web testing framework, of which most where either too slow (Selenium) or too buggy for my needs (env.js), I decided that zombie.js looks most promising, as it uses a solid set of lib...

I cannot, for the life of me, rig HtmlUnit up to grab this site:  http://www.bing.com/travel/flight/flightSearch?form=FORMTRVLGENERIC&q=flights+from+SLC+to+BKK+leave+07%2F30%2F2010+return+08%2F11%2F2010+adults%3A1+class%3ACOACH&stoc=0&vo1...

This is part of a project I am working on for work.  I want to automate a Sharepoint site, specifically to pull data out of a database that I and my coworkers only have front-end access to.  I FINALLY managed to get mechanize (in python) to accomplis...