Once again I was frustrated today. Personally I can not stand long, wordy documentation. So while I applaud Amazon for having documentation explaining AWS SDK for Node… just give me a dang example! I would much rather minimal wordiness and just have code samples. The one company who does this right is Stripe and maybe a future post will be about that great payment service–but now I digress.
So even outside of Amazon, I have had a very hard time finding examples with the AWS SDK for Node.JS. So for that reason I’ll try to help fill that void by continuing to post these blogs as I figure out by trial and error.
What is DynamoDB?
I won’t be extensive here, but DynamoDB is essentially a hosted NoSQL solution from Amazon Web Services. It is different from a lot of other NoSQL databases. I actually really like Amazon’s SimpleDB but it seems like they are phasing it out for some stupid reason, so I am getting used to using Dynamo instead.
Why is it good? Because it is really fast, low latency, no worries about storage or scaling, and it is relatively affordable.
Why does it suck? Because there are a lot of limitations on the indexing and querying. And you can’t change your indexes later, so you better get it right the first time!!! I’ll skip the details for a possible futuer post, but long story short…
You must specify a “hash key”, which must be included in every single query (unless you do a full scan). You can have a secondary key called a “range key” optionally. Those two keys together form your primary (unique) key.
Our Application: Logging
I want to move some of our server error/event logging over into Dynamo. You’ll want to log into the AWS DynamoDB Console and create a table. You’ll have to give it a name, keys, and some read/write throughput settings (which I won’t get into). NOTE: You get charged whether you use your table or not, so be sure to delete it after if you’re just testing.
I defined my two keys as follows:
- Hash Key: itemName (String)
- Range Key: microtime (Number)
The item name is something identifying what the error/event is about. So I might do “video.12345” to say we are logging something about that particular video. And then my range key wil be “microtime” which will be an integer of milliseconds since the epoch. Going to the millisecond there should most likely keep the combination of the two keys unique, even if we log two things pretty close together in the code. You could add something additional to the end if you needed to or try to go to the nanosecond or something. But for our purposes, I think it will work fine.
The other fields (called “attributes” in DynamoDB world) do not need to be configured ahead of time, but I will use:
- userID (Number)
- level (String) – debug, info, warning, error, fatal
- message (String)
- details (String)
Note that there is a 64KB limitation that you can store in DynamoDB, but I’m not concerned about that at all for our needs here.
Get Down to the Code
I am assuming at this point that you have set up Node on your server or local machine. You should also have the aws-sdk package installed, if not run this…
npm install aws-sdk
I am not going to go over all this again, but then you need to instantiate and authenticate with your credentials…
var AWS = require('aws-sdk');
AWS.config.loadFromPath('./aws.milesplit.json');
var dd = new AWS.DynamoDB();
Now we are going to put an item (row) into our DynamoDB table…
var tableName = 'YOUR_TABLE_NAME';
putItem = function(itemName, message, userID, level, details) {
var item = {
'itemName': { 'S': itemName },
'microtime': { 'N': new Date().getTime().toString() }
};
if (userID) item.userID = { 'N': userID };
item.level = { 'S': (level || 'info') };
if (message) item.message = { 'S': message };
if (details) item.details = { 'S': details };
dd.putItem({
'TableName': tableName,
'Item': item
}, function(err, data) {
err && console.log(err);
});
};
// Use the function we just created...
putItem('video.1234', 'Video has encoded', 153);
A few important things to note here (I learned the hard way on some of these)…
The values of each attribute is not set directly, you set it as a mapped object as above. Use ‘S’ if you are setting a string attribute value or ‘N’ if you are setting a numeric value. There are some other options in the documentation, but I’ll leave it to them to explain those.
If you are setting a numeric value, you may have to still explicitly convert it to a string. I don’t know why but I ran into an error with setting my microtime. I had to add the toString() method to the end of it, even though I’m saving it as a numeric type in DynamoDB. It doesn’t make complete sense to me, but that’s what worked.
Something else I found out, you can not set an attribute value as an empty string. It will fail to insert. So that’s why you see me seeing if the value is set or not. If it is not set then simply leave off that attribute. Again since it’s a NoSQL database those attributes can be changed, added, or left off on the fly. The only ones requierd are your primary keys (hash key and sometimes range key).
That’s all
I haven’t done any fetching of the inserted rows yet, but that should be easy enough. You can look in the AWS Console and scan or query your table to see that they were inserted.
Have fun!
MileSplit Developer Blog 