I have recently been learning Python and am dipping my hand into building a web-scraper.  It's nothing fancy at all; its only purpose is to get the data off of a betting website and have this data put into Excel.  Most of the issues are solvabl...

I am in the process of learning and simultaneously building a web spider using scrapy. I need help with extracting some information from the following javascript code:  <script language="JavaScript" type="text/javascript+gk-onload"...

I am working on a scrapy app to scrapte some data on a web page  But there is some data loaded by ajax, and thus python just cannot execute that to get the data.  Is there any lib that simulate the behavior of a browser?...

I am trying to crawl this URL. It changes cost from 130 ot 154.99 via Javascript. When you see view source of Price DOM it looks like:  <span id="item_price_6516">&pound;130.00</span>   And it's definitely setting via Java...

I am trying to scrape used items in my area from https://us.letgo.com/en for a personal project.  I found this video helpful https://youtu.be/EelmnSzykyI. However, there are some subtle differences that the video doesn't help with.    The info I...

I'm new to Scrapy-splash and I'm trying to scrape a lazy datatable which is a table with AJAX pagination.  So I need to load the website, wait until JS is executed, get html of the table and then click on the "Next" button on paginati...

I'm trying to use scrapy with selenium to be able to interact with javascript and still have the powerful scraping framework that scrapy offers. I've written a script that visits http://www.iens.nl, enters "Amsterdam" in the search ba...

Trying to automatically grab the search results from a public search, but running into some trouble. The URL is of the form   http://www.website.com/search.aspx?keyword=#&&page=1&sort=Sorting   As I click through the pages, after visiting...

I've been trying to extract data from script tag using Scrapy(xpath). My main issue is with identifying the correct div and script tags. I'm new to using xpath and would appreciate any help!  <script>     (function(e, a) {             v...

I would like to know if there is something like Scrapy for nodejs ?. if not what do you think of using the simple page download and parsing it using cheerio ? is there a better way....