Web Scraping using JSoup: Getting Weekly Top Songs Project

In this article we’ll build a simple web scraper with the jsoup library.

What is web scraping?

Web scraping extracts data from websites that don’t offer it any other way. Many companies expose some data to developers through an API: the Twitter and Instagram APIs, for example, give access to posts on a topic or from a location. Others, such as Amazon or eBay, don’t offer an API for things like tracking a product’s price. In those cases you can scrape their pages.

Is scraping legal?

Companies don’t usually mind a one-off scrape, but hitting their site every minute is a big no-no (only Google’s crawlers get away with that). Keep the number of requests to a minimum and scrape responsibly. Check the site’s terms of service and robots.txt too.

Project: weekly top songs

We’ll scrape JioSaavn for its list of weekly top songs, at https://www.jiosaavn.com/featured/weekly-top-songs.

Inspecting the page shows the elements we need. Trimmed down, each song in the list looks like this:

<ol class="page-group track-list">
  <li class="song-wrap" data-songid="imAwtsuz">
    <div class="index">1</div>
    <span class="time">2:48</span>
    <!-- … artwork, title, menu … -->
    <div class="hide song-json">
      {"title":"Genda Phool","album":"Genda Phool","language":"hindi",
       "year":"2020","duration":"168","singers":"Badshah, Payal Dev", …}
    </div>
  </li>
  <li class="song-wrap" data-songid="jZjE_NfL">…</li>
</ol>

So we’re looking for the ordered list with the class track-list. Each item in it has a hidden div with the class song-json, which holds all the data for that song as JSON. Let’s write the code to parse it.

public class AppMain {

    public static void main(String[] args) {
        try {
            Document doc = Jsoup.connect("https://www.jiosaavn.com/featured/weekly-top-songs").get();
            Elements elements = doc.select("div.song-json");
            elements.forEach((element) -> {
                JsonElement json = JsonParser.parseString(element.text());
                System.out.println(json.getAsJsonObject().get("title"));
            });
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

We use jsoup to fetch and parse the page, and Gson to read the JSON. Line by line:

Document doc = Jsoup.connect("https://www.jiosaavn.com/featured/weekly-top-songs").get();

jsoup connects to the site and parses the page’s HTML into a Document object.

Elements elements = doc.select("div.song-json");

This selects every div with the class song-json, which gives us the data for all the songs.

elements.forEach((element) -> {
    JsonElement json = JsonParser.parseString(element.text());
    System.out.println(json.getAsJsonObject().get("title"));
});

Finally, we take each element’s text, parse it with Gson’s JsonParser, and read any field we want as a JSON property. Here we print the title of every song, and that’s it.

Conclusion

jsoup parses static HTML. It doesn’t run JavaScript and it isn’t a substitute for a browser. For pages that build their content with JavaScript, look at a browser automation tool such as Selenium WebDriver or Playwright. And once again: scrape responsibly.

The project is on GitHub: shahulbasha/WebScraping.