Scraping dynamically generated html inside Android app

I am currently writing an Android app that, among other things, uses text information from websites which I do not own. In addition, some of the pages require authentification.

For some pages I have been able to log in and retrieve the html code using BasicNameValuePairs and an HTTPClient with its associated objects.

Unfortunately, these methods retrieve the webpage source without running any javascript functions that a browser (Android Webview even) would normally run. I need the text that some of these scripts are retrieving.

I've done my research, but everything I've found is guesswork & extremely confusing. I'm okay with ignoring pages that require login for now. Also, I am willing to post any code that may be useful for constructing a solution; It is an independent project.

Any concrete solutions for scraping the html result from javascript calls? An example would be absolutely top-notch.

Answers:

Answer

The aforementioned solutions are very slow and restrict you to 1 url (well, not really, but I dare you to scrape 10 urls with Rhino while your user is impatiently waiting for results).

An alternative is to use a cloud scraping solution. You get the benefit of not wasting phone bandwidth on downloading content you won't use.

Try this solution: Bobik Java SDK

It gives you the ability to scrape up to hundreds of sites in a matter of seconds

Answer

Final Success:

Other Things I Tried:

  • HttpClient provided by Android
    • Cannot run javascript
  • HtmlUnit
    • 4 hours, no success. Also huge, added 12 mb to my apk.
  • SL4A
    • Finally compiled. Used THIS guide to set-up. Abandoned as overkill for a simple rhino jar.

Things That Might Work:

  • Selenium

Further results will be posted. Others results will be added if posted.

Note: many of the options listed above reference each other. I think rhino is included in both sl4a and htmlunit. Also, I think htmlunit contains selenium.

Tags

Recent Questions

Top Questions

Home Tags Terms of Service Privacy Policy DMCA Contact Us

©2020 All rights reserved.