Twitter REST API from Scala and Java using signpost

If you’ve read some other articles on this blog you might know that I like creating visualizations of various datasets. I’ve just started a small project where I want to visualize some data from Twitter. For this I want to retrieve information about followers and profile information directly from twitter. I actually started looking for a set of all twitter accounts, but could only find one that was two years old. So, only option left, directly access the twitter API and get the data myself.

There are a couple of open source libraries out that we can use directly from Scala (or java) but as far as I could see they use the old v1 API and not the v1.1 API. The old API has a very strict data rate limit, which is a bit lighter in the new API. And besides that I’m more interested in the raw data and parsing the returning JSON isn’t that hard with Scala (or Java for that matter).

Register an application at twitter

First thing to do, and easiest way to get started is registing a new application for your twitter account. Go to https://dev.twitter.com/apps/new and create a new application. Don’t worry about the urls, since we won’t be using the OAuth callback mechanism:

Depending on what you want to do with the API you need to give additional permissions to this app. Default is ‘read-only’, if you want to allow your new application to post or access your direct messages you need to update the permissions. This is done from the settings page of your application:

Once you’ve created the application and setup the correct permissions you can generate an access token. Doing this will avoid having to go through the complete OAuth dance. You do this by going to your new app details and at the bottom select the ‘create my access token’ option.

Now you’ll have a set of tokens (see the details part of your applications):

We’ll use these tokens to authenticate the requests we’ll make to twitter.

Using a OAuth library

The OAuth protocol is a pretty good documented protocol, but implementing it yourself is a lot of work and very error-prone. Luckily there are many OAuth libraries that can help you. I’ve tried a couple and the one that’s most easy to use (at least for me) was signpost. The examples below show how to do this from Scala, but you can follow the same approach for Java.

First the dependencies. I’ve used sbt and from signpost I use the client with support for HTTP commons. In sbt add the following:

..
libraryDependencies ++= Seq(
 'oauth.signpost' % 'signpost-core' % '1.2',
 'oauth.signpost' % 'signpost-commonshttp4' % '1.2', 
 'org.apache.httpcomponents' % 'httpclient' % '4.2',
         ...
)

For maven you can use the same libraries. Next we can write a simple test to see if everything is working. In java it looks like this:

import oauth.signpost.OAuthConsumer;
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer;
 
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
 
 
public class Tw {
 
   static String AccessToken = 'access token for your app';
   static String AccessSecret = 'access secret for your app';
   static String ConsumerKey = 'consumer key for your app';
   static String ConsumerSecret = 'consumer secret for your app';
 
 /**
  * @param args
  */
 public static void main(String[] args) throws Exception {
  OAuthConsumer consumer = new CommonsHttpOAuthConsumer(
                ConsumerKey,
                ConsumerSecret);
 
        consumer.setTokenWithSecret(AccessToken, AccessSecret);
        HttpGet request = new HttpGet('http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen');
        consumer.sign(request);
 
        HttpClient client = new DefaultHttpClient();
        HttpResponse response = client.execute(request);
 
        int statusCode = response.getStatusLine().getStatusCode();
        System.out.println(statusCode + ':' + response.getStatusLine().getReasonPhrase());
        System.out.println(IOUtils.toString(response.getEntity().getContent()));
 }
}

And in Scala it looks pretty much the same:

import org.apache.http.client.HttpClient
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpGet
import oauth.signpost.commonshttp.CommonsHttpOAuthConsumer
import org.apache.commons.io.IOUtils
 
 
object TwitterPull {
 
   val AccessToken = 'access token for your app';
   val AccessSecret = 'access secret for your app';
   val ConsumerKey = 'consumer key for your app';
   val ConsumerSecret = 'consumer secret for your app';
 
 
  def main(args: Array[String]) {
 
  val consumer = new CommonsHttpOAuthConsumer(ConsumerKey,ConsumerSecret);
  consumer.setTokenWithSecret(AccessToken, AccessSecret);
 
     val request = new HttpGet('http://api.twitter.com/1.1/followers/ids.json?cursor=-1&screen_name=josdirksen');
     consumer.sign(request);
     val client = new DefaultHttpClient();
     val response = client.execute(request);
 
     println(response.getStatusLine().getStatusCode());
     println(IOUtils.toString(response.getEntity().getContent()));
  }
}

When you run this the output will look something like this:

200
{"previous_cursor_str":"0","next_cursor":0,"ids":
[48342167,21011010,824959303,97242821,16953163,218083367,20869799,5234221,13604142,80
4783128,271050984,405121284,26470609,50201837,1723451,374494377,120867838,14311946,25
3114713,39554511,7375412,42507395,112806109,92787154,218238023,110443797,76922155,198
798790,294104985,305625416,217698029,21803482,14927822,15453445,15715866,15657036,186
956616,36028164,70380613,326158542,573546312,14401332,521488579,9108612,576970378,293
236313,16398366,16220300,15234937,32000283,439444353,14300622,67204409,155850135,1419
8255,32264673,15852981,313248158,20123099,608942046,234930032,36896958,18466675,45496
942,330899833,18980755,88253383,461023805,31175627,11044952,142780445,63175189,107991
607,94830953,600993241,6195002,115391430,550080945,381418927,168603682,142388604,8258
462,218411138,30450578,77728346,2521381,182867524,494119147,29426983,572417260,943448
49,325413275,389354525,501438275,164346498,22730282,8293302,21085554,341645357,569788
53,180507788,10074002,22536424,14247654,581293627,15259428,483317230,462826270,477464
1,15366832,96850673,278486993,22273826,17716679,14566626,158473088,20461042,161242434
,43756629,40163100,141165981,5325152,7620782,266749648,524476136,557713614,39602637,1
8843154,1623,565954426,39639621,166672305,18683074,233118689,44876099,235258223,21931
0062,10699922,12660502,218030046,91552210,19361980,206645598,35346200,58440021,470388
557,26495649,59066453,40292255,543375441,33242290,6015852,317150447,22935775,23230034
6,476045917,90913482,249088920,67658976,614873,522722520,186766721,285517705,71683175
,131444964,166501605,477920664,38154550,18738205,8861832,15594932,18536741,7595202,46
5378842,11838952,14848133,431696576,14358671,414520167,222578501,67058139,28976735,95
601387,426582611,24874129,418762594,128157235,106030956,31352215,18733178,260196778,1
53179029,91842580,229494512,83414433,285579699,19957600,54295155,14929418,51516573,20
0076011,18758733,17776895,59397841,216802709,149834999,327507356,8200322,174345369,10
8636400,27504001,326877592,139919716,49949338,215035403,118421144,49410665,149550914,
18446431,25662335,261725134,267634174,57737391,146506056,126964949,71055234,20870640,
210196418,222806923,13290742,72247756,180410163,14784480,36684216,25611502,95614691,5
4629161,112967594,181656257,17994312,72918901,140082918,149087212,137272324,99534020,
121755576,93964779,35848342,43059008,34704029,87672717,113137792,17863333,90407665,90
591814,54297023,57924897,87551006,28300354,48990752,26188013],"previous_cursor":0,
"next_cursor_str":"0"

If you get a 403 check whether the tokens match.

Happy coding and don’t forget to share!

Reference: Access the Twitter REST API (v1.1) from Scala and Java using signpost from our JCG partner Jos Dirksen at the Smart Java blog.

Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

Leave a Reply


two × = 12



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books