AWS SimpleDB – what’s it good for?

Actually – it’s a nice and handy part of your cloud toolbox is what.

First of all – it’s a non-relational data store.  Which means you are not going to have highly normalized table structures and you are not (for the here and now) going to be able to join two tables in a SQL like query.

BUT!  It does work faster and more efficiently that trying to get access to a XML config file; it provides persistence that an in-memory data structure can’t; you can query it with SQL type statements; you can either insert or update data using a decent API; plus you can add data fields on the fly in your code.  Best of all – it’s very easy to use, and it’s cheap.  Far cheaper than either the MySQL or Oracle offerings – you get charged for every hour those guys are up and running.  For SimpleDBthe pricing structure is a lot more attractive.

Some things to square away first.  Forget the idea of tables – each unique data structure in SimpleDB is called a domain.  It’s a non-relational/quasi no SQL data structure that does allow SQL queries to get data.  Think tree and you have it – the primary node for each data row is called an itemName – the rest of the data fields are elements of the primary node.

So for my very simple Java servlet – I needed a user data structure that had to be persistent.  With SimpleDB the domain (think table) name became MyUsers – and the primary node for each row – the itemName in SimpleDB syntax – was the unique user ID.  I had one other data element – the child node – it was a URL for the user.

I have covered the making the connection to AWS in an earlier post so I’m not going over that again.

A great resource is the sample code that Amazon provides – available for more than just Java.

Next couple of posts I’m going to cover creating a domain, putting data into it, and then getting the data back out.

Amazon Web Services and SimpleDB Part Two

Next item: adding the AWS jar file libraries to the ant build.xml file.

Actually – this gets a bit round about.  Amazon supplies the SDK files – and if I had more experience with Eclipse this probably would not have been an issue.  In any event – I downloaded the AWS SDK for Java files into a directory on my C drive – C:\AWS_HOME to be exact.  I added the files to my path and class path – didn’t matter for building a servlet as the code wouldn’t compile.

Next step – I added a folder in my project, copied the AWS jar files into that file and set the path in my ant build.xml file.  Okay – got it to compile but Jetty would bomb.

Onto step number three – I added the following files to my Jetty ..\lib\ext folder:

From \aws-java-sdk-1.2.10\lib – aws-java-sdk-1.2.10.jar, aws-java-sdk-1.2.10-javadoc.jar and aws-java-sdk-1.2.10-sources.jar.

From \aws-java-sdk-1.2.10\third-party\commons-codec-1.3\commons-codec-1.3.jar.

From \aws-java-sdk-1.2.10\third-party\commons-logging-1.1.1\commons-logging-1.1.1.jar.

And lastly from \aws-java-sdk-1.2.10\third-party\httpcomponents-client-4.1.1\httpclient-4.1.1.jar and httpcore-4.1.jar.

My code would now compile – but then I got a weird error attempting to connect to SimpleDB.  I’m working with a 64 bit version of Windows 7, and Amazon supplies a 32 bit encrypt/decrypt codec from Apache.  I downloaded the commons-codec-1.5 jar file from Apache and added that to my Jetty directory – now it all worked fine.

Issue four – I naively thought that the Tomcat environment I was using on Amazon would have the library files included to connect to SimpleDB.  I was wrong – I had to include the files in my build file.  I created a WEB-INF\lib directory and placed copies of the above files into it.  My ant build file now included:

<path id=”compile.classpath”>

<fileset dir=”${env.JETTY_HOME}/lib”>

<include name=”**/*.jar”/>

fileset>

<fileset dir=”${env.AWS_HOME}/third-party” includes=”**/*.jar”/>

<fileset dir=”${env.AWS_HOME}/lib” includes=”**/*.jar”/>

<pathelement location=”.”/>

<pathelement location=”../lib/aws-java-sdk-1.2.10.jar”/>

<pathelement location=”../lib/aws-java-sdk-1.2.10-javadoc.jar”/>

<pathelement location=”../lib/aws-java-sdk-1.2.10-sources.jar”/>

<pathelement location=”../lib/commons-codec-1.5.jar”/>

<pathelement location=”../lib/commons-codec-1.5-javadoc.jar”/>

<pathelement location=”../lib/commons-codec-1.5-sources.jar”/>

<pathelement location=”../lib/commons-logging-1.1.1.jar”/>

<pathelement location=”../lib/httpclient-4.1.1.jar”/>

<pathelement location=”../lib/httpcore-4.1.jar”/>

</path>

Success!  Now not only could I connect to SimpleDB on my client using Jetty, but also on my Amazon environment.

Next post – some more on using SimpleDB.

Amazon Web Services and SimpleDB

Recently finished a simple Java Servlet using Amazon Web Services which required a simple data persistence layer.  The natural choice was SimpleDB.  And I found some gotchas…

First of all – a thumbnail sketch of SimpleDB.  It’s a schemaless data structure consisting of hash tables with sets of key value pairs.  A nice write-up is here.  The AWS SimpleDB site is here.

My typical dev process was build and test local, then move up to AWS.  I’m using Java version jdk1.6; Apache Ant 1.8.1; Jetty 8.0.1x.

Start by downloading the AWS Java SDK .  Amazon provides a good resource of Java programs providing examples of using their various cloud products – the two I used as reference for this project was in the samples\AmazonSimpleDB and samples\AwsConsoleApp directories.

There were four issues I ran into before being able to successfully use the product:

1.  Connect to SimpleDB thru the supplied AWSCredentials API.

2. Adding the AWS jar file libraries to the ant build.xml file.

3. Add the AWS jar files to Jetty for development and testing.

4. Make sure the AWS instance had access to the jar files.

First issue, connection using a credential set.  You create your credentials when you set up your Amazon Web Services account.  And a blank credential file is provided in the SDK demo programs.  Since this is a servlet program, the code resides as child directories of a WEB-INF folder.  I placed my credentials file into the WEB-INF\resources folder.

Next thing is to bring the credential file into the various methods that need access to the SimpleDB.  First I defined a String variable containing the path to the credential file:

String filename = “/WEB-INF/resources/AwsCredentials.properties”;

The file is brought in as a stream:

InputStream is = context.getResourceAsStream(filename);

An AWS credentials object is set using the data from the stream:

AWSCredentials credentials = new PropertiesCredentials(is);

And finally an AWS SimpleDB object is created, using the AWS credentials.  This SimpleDB object is what you use to connect to your SimpleDB domain:

AmazonSimpleDB sdb = new AmazonSimpleDBClient(credentials);

I’ll cover the other issues in future posts.

Cookies, Stacks and Lists

I can’t get into too much detail on this assignment however in general – a user would access a simple Java web server.  The user would make a request of the server and some text would be displayed in their browser.

There was to be a collection of text items which would be sent back to the user in random order – and no duplicates until the entire collection had been run thru.

So – a list of strings, some way of randomizing the list and then a way to keep track of what was being sent to the user were parts of the core requirements.

My solution was to start with a Java LinkedHashMap – using integers for the Keys, and then the text would be the Value part of the pair.   And no, I would not be doing some kind of random loading of the list – the providing random text comes from another piece.  An array would work here as well – both have  O(1) look-up, there isn’t any iteration nor adding/deleting from the list.

Oh, and one other requirement – the user needed to add a nickname so as to personalize the return.  So my method looked something like this:

/**
*  Takes a String name, places it into the line of text and then return.
*
*  @param name String containing a nickname supplied by a user.
*  @return LinkedHashMap containing collection of text.
*
*/

static  LinkedHashMap  getrt(String name){

LinkedHashMap lhm = new LinkedHashMap();

String rt = “A. ” + name + “, what goes 99 click? A centipede with a wooden leg.”;
lhm.put(1, rt);
rt = “B. ” +  name + “, how many seconds are in a year? There are 12 seconds in a year. January 2nd, February 2nd, March 2nd…”;
lhm.put(2, rt);
rt = “C. ” +  name + “, here is one for you.  Little four-year-old Julie was looking at her new baby brother for the first time. He was fast asleep. After staring at her tiny, motionless baby brother for a few minutes, Julie looked up at her mother and asked why didn’t he come with batteries?”;
lhm.put(3, rt);
rt = “D. ” +  name + “, you can sure pity the dad with three kids in college. He tells his wife that they are getting poorer by degrees.”;
lhm.put(4, rt);
rt = “E. You know ” + name + “, when I die I want to go peacefully — like my grandfather did — in his sleep. Not screaming like the passengers in his car.”;
lhm.put(5, rt);
return lhm;
}

Next part was create a random Queue – this gets passed back to the user as part of their cookie, and gets used to deliver the text from the LinkedHashMap:

/**
*  Takes an int count of a list and returns a randomized queue
*
*  @param listSize int value containing size of a list.
*  @return Queue containing a randomized List of ints.
*
*/
static Queue outputQueue(int listSize){
Queue queue = new LinkedList();
Random rnd = new Random(System.currentTimeMillis());

for(int cntr = 0; cntr < listSize; cntr++){

int i = Math.abs(rnd.nextInt() % 75);
// Want a random list of unique integers no greater than the size of the text list.
while(i <= 0 || i > listSize || queue.contains(i) ){
i = Math.abs(rnd.nextInt() % 75);
}
queue.add(i);
}
return queue;
}

And no, the user isn’t getting the queue passed back to them – the value of the queue gets added to an empty string and that gets returned to the user:

// No more values – need to requeue
out.println(name + “, I told you everything – restarting the list.”);
TextServer.textQ = outputQueue(lhm.size());
s = TextServer.textQ.poll().toString();
int qSize = TextServer.textQ.size();
for(int i = 1; i <= qSize; i++){
s += TextServer.textQ.poll().toString();
}
jIndex = s;

The first value in the string gets ‘popped’ off, as what would happen in a queue – the string value gets used to find a corresponding Key in the LinkedHashMap – and the user gets served a line of text.  The string is returned to the user as part of their cookie.  Once all the values from the string are ‘popped’ off – in other words the string is empty – another call is placed to the outputQueue method and a fresh string is sent back to the user.  Here is the code for that method:

/**
*  Gets a nickname from client and returns text using the name to output stream.
*
*  @param lhm LinkedHashMap containing text.
*  @param name String containing a nickname supplied by a user.
*  @param jokeIndex String holding the list of text yet to be told.
*  @param out PrintStream sent to client containing text with the supplied nickname.
*  @exception IllegalArgumentException Thrown to indicate that a method has been passed an illegal or inappropriate argument.
*
*/

static String printText(LinkedHashMap lhm, String name, String textIndex, PrintStream out){
String jIndex = textIndex;

try{
int qnum = 0;
// If starting from scratch – get a random queue and set the values into the textIndex string
if(Integer.parseInt(jIndex) <= 0){
out.println(name + “, I’m starting the text list.”);
out.flush();
TextServer.textQ = outputQueue(lhm.size());
jIndex = TextServer.textQ.poll().toString();
int qSize = TextServer.textQ.size();
for(int i = 1; i <= qSize; i++){
jIndex += TextServer.textQ.poll().toString();
}
}

// Using the string textIndex as a cheap queue grab the first value and use it.
String s = jIndex.substring(0, 1);
qnum = Integer.parseInt(s);
out.println(lhm.get(qnum));
out.flush();

if(jIndex.length() >= 2){
  // Here I’m ‘popping’ the first value off the string
s = jIndex.substring(1);
jIndex = s;
} else {
// No more values – need to requeue
out.println(name + “, I told you everything – restarting the list.”);
TextServer.textQ = outputQueue(lhm.size());
s = TextServer.textQ.poll().toString();
int qSize = TextServer.textQ.size();
for(int i = 1; i <= qSize; i++){
s += TextServer.textQ.poll().toString();
}
jIndex = s;
}
} catch (IllegalArgumentException ex){
out.println(“Failed in attempt serve a text to ” + name);
}

return jIndex;
}

%d bloggers like this: