Creating an RSS JSON API - a Google Feed API alternative

2017-01-11

Last month (December 2016) the Google Feed API was discontinued. The CUGOS website (cugos.org) used the Google Feed API to list out the recent posts in our Google Group, which has a less-than-stellar user interface. The JavaScript console on cugos.org looked like this:

A quick search on the internet and glance at the Google Feed API website showed the API had been officially deprecated in December… blast! Not only was it discontinued, no alternative was presented. Bummer.

After doing some research, it appeared there wasn’t a common FOSS alternative to the API provided by Google. Research showed a paid service called Superfeedr, but this doesn’t fall within the $0 budget for CUGOS, nor the open source mission.

My next thought was to try consuming XML through an Ajax request. I tried setting up the following on the website:

$.ajax({
  url: 'https://groups.google.com/forum/feed/cugos/topics/rss.xml?num=15',
  dataType: 'xml',
  success: function(data) {
    // ...
  }
});

The plan was to get an XML response, then convert it to JSON using some sort of XML -> JSON library. Unfortunately, this solution was cut short:

This error means other domains (origins) aren’t granted access to this information via XMLHttpRequests. Google disallows it, and we can’t change that. This means the jQuery, on-the-fly conversion option was a no-go.

The last option was to create our own service that did this for us on a server, rather than in the browser. This means we’d have to spin up an HTTP API that can download the XML at the URL above, convert it to JSON and respond to a GET request made by a browser.

I mocked up a quick example using ExpressJS and Heroku, called rss2js. Upon deploying, it worked! After a bit more research, I found this exact solution had already been created at feed2js, but unfortunately it has been shut down due to high costs and lack of maintenance interest. I took note, and realized that creating a drop-in replacement for all feeds would lead to more cost than I desired. I quickly changed the approach, forked my repository into the CUGOS GitHub organization, and built the same Heroku server that ONLY returned information from the CUGOS Google Group, which, presumably will receive less traffic.

The source for the CUGOS Feed API is on GitHub, and you can see a JSON response at the following URL: https://cugosfeedapi.herokuapp.com/?count=15.

One down side to using Heroku, if the app hasn’t been used in over an hour, it goes to “sleep”. This results in a general wake up period that can lead to some poor user experience. A downside that doesn’t outweigh the alternative: no feed on the website.

The CUGOS Feed API is live at cugos.org!