This post describes the process of constructing and deploying a simple ToDo list application using the technologies in the title of the post. The app can currently be found at http://evening-beach-6294.herokuapp.com/. The app can add, list and remove ToDos from the application. I used Ubuntu 11.10 o/s to work on this basic project to help me understand these technologies better as I come from a SQL Server /.Net background.
I hope this post will prove useful to anyone else who is new to these technologies as I have tried to describe everything from the ground up so not to confuse someone, if, like me, you are totally new to all this. I’d welcome any feedback about this post as I hope it does help rather than hinder anyone who is interested in the same things. EDIT: The Node.js source code for the app can now be found at https://github.com/lalexgraham/ToDoList.
The reason I used Ubuntu in the end to complete this project was that I found working with Windows very problematic in particular with using Foreman. My post to Sencha forums here explains more.
The starting point for the app was this post, which quickly showed an example of the javascript to work with, especially for the connection up to MongoDB using the native MongoDB driver.
The steps have been broken down into the following:
1. Install git and node.js
2. Install MongoDB
3. Create node.js application
4. Setup MongoHQ
5. Deploy to Heroku
1. Install git and node.js
To get started with node.js on Ubuntu you can write a simple “hello world” script and view it in the browser. To install node.js on Ubuntu you can use git clone command. If you are unfamiliar with git you can google to find out more about it. git clone copies code located at a url to a location on your hard drive. To install git you have to run these commands in a terminal window on Ubuntu:
sudo apt-get update
sudo apt-get install git-core
sudo apt-get install build-essential
sudo apt-get install libssl-dev
If these go ok, then you can install node.js by entering:
git clone git://github.com/joyent/node.git
This installs node.js at your root directory, so you can change directory to it by typing:
cd node
From within the node directory you can configure and install by typing:
./configure
make
sudo make install
That is node.js installed. To test the install enter the following js into a new file on your text editor
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World\n');
}).listen(1337, "127.0.0.1");
console.log('Server running at http://127.0.0.1:1337/');
Save the file as example.js within the node folder and then from the same terminal window, enter the following code for node.js to execute the code
node example.js
Browse to the address http://127.0.0.1:1337 and you should see the output ‘Hello World’. You should also see the ‘Server running at http://127.0.0.1:1337/’ output in your terminal window.
2. Install MongoDB
With node.js up and running, setting up MongoDB to test with was next. The stucture of the database was fairly simple. I just wanted the database to have one collection with one document; “title”. The “title” would contain each ToDo. Installing MongoDB was tricky and took a few attempts. As always its a case of googling and eventually finding the right way. This post describes the way I installed MongoDB, which took the following steps. Note that there are other ways to install mongoDB. The steps below are the apt-get install, but this post suggests a different install which, I think, is the “binary” install:
2.1 Add 10gen package to source.list
2.2 Update package
2.3 Add GPG Key
2.4 Install mongodb-10gen
2.5 Done
2.1 Add 10gen package to source.list
from the directory labelled “FileSystem” which I take to understand as the root folder, you can navigate to and open etc/apt/sources.list using your text editor. Paste the following into the sources.list file:
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
2.2 Update package
Now you have a path to the installation of MongoDB, if you open a new terminal window and type in:
sudo apt-get update
Ubuntu will pick up on the installation and show output in the terminal window as follows (or something similar)
mongodb-10gen – An object/document-oriented database
2.3 Add GPG Key
To install MongoDB, you need to add a key specific to the MongoDB install with the following terminal command
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
Note you have to use the sudo keyword which basically gives you admin privileges just for that command
2.4 Finally you can install MongoDB with the following command:
sudo apt-get install mongodb-10gen
10gen is the company that produces MongoDB. Its not a version.
2.5. Done. Hopefully(!) There are a few steps here, which unfortunately means things can go wrong.
Now you have MongoDB installed, you can test the installation by going into the mongo console:
Open terminal window, enter “mongo”, MongoDB starts with a message similar to..
MongoDB shell version: 2.0.2
connecting to: test
The main configuration file “mongodb.conf” is located at “/etc/mongodb.conf“ where you can amend the config if required.
When writing this post and checking MongoDB on my machine I got an error when trying to launch the console:
Error: couldn't connect to server 127.0.0.1 shell/ mongo.js:84
The solution was to uninstall and reinstall MongoDB.
To uninstall use the following command:
sudo apt-get autoremove mongodb-10gen
I then followed steps 1 to 5 noted above, omitting step 3, the GPG key. For some reason MongoDB had previoulsy been shutdown incorectly, which had affected its config. Reinstalling did not lose any data it seems.
So once I had mongodb running, I could add my database and collection to test with. This can be done from the console. If you are used to SQL databases then the syntax and conventions are very different so take some getting used to. This page from the MongoDB website lists some of the commands that prove useful when using the console, and this is a good tutorial. One of the main differences from a SQL setup such as SQL Server is the power of the commands, many of which are implicit. For example, to create a database “test” use the command “use test;”. Technically speaking this doesn’t actually create the database. The database is created “lazily” by inserting data . Next create the collection “todo”, which is done by typing in ‘db.createCollection(“todo”)’. Once the collection is created you can add a document to it by entering:
> todo = {title: "do the food shop"}
{ "title" : "do the food shop" }
> db.todo.insert(todo);
Note, unlike SQL there’s no need to setup a schema for the collection declaring the title as a string, etc. You just do an insert of data to set up the collection. To do a select of the collection you enter a find command:
> db.todo.find()
{ "_id" : ObjectId("4efca8c598c3d71c7a30ded6"), "title" : "do the food shop" }
>
So, now I have the collection in a database that I want being returned through MongoDB. The next step is to build the node.js application.
3. Create node.js application
I then read up more on node.js, and found this tutorial, which I followed and added things like routing to the ToDo List app. The tutorial also introduced npm (node package manager).
So I factored in some of the suggestions from this post to my ToDo app and here is a run through of the code and where it all fits.
The entry point for the application is index.js:
var server = require("./server");
var router = require("./router");
var requestHandlers = require("./requestHandlers");
var handle = {}
handle["/"] = requestHandlers.list;
handle["/list"] = requestHandlers.list;
handle["/add"] = requestHandlers.add;
handle["/remove"] = requestHandlers.remove;
server.start(router.route, handle);
This script uses several imports, the first one is server.js:
//Load modules
var http = require('http');
var url = require('url');
function start(route, handle) {
function onRequest(request, response) {
var pathname = url.parse(request.url).pathname;
console.log("Request for " + pathname + " received.");
route(handle, pathname, response, request);
}
var port = process.env.PORT || 8888;
http.createServer(onRequest).listen(port);
console.log("Server has started on port " + port);
}
exports.start = start;
The start function extracts the pathname (i.e. “/list”) and passes this to the route function, before setting up a port to listen on for requests.
router.js:
function route(handle, pathname, response, request) {
console.log("About to route a request for " + pathname);
if (typeof handle[pathname] === 'function') {
handle[pathname](response, request);
} else {
console.log("No request handler found for " + pathname);
response.writeHead(404, {"Content-Type": "text/html"});
response.write("404 Not found");
response.end();
}
}
exports.route = route;
The route function tries to associate an array item from the handle array with the request made. If successful, the relevant function is called from requestHandler.js, (list, add, remove):
//Load modules
var http = require('http');
var url = require('url');
var util = require('util');
var querystring = require('querystring');
var appURL = 'http://localhost:5000';
//var appURL = 'http://evening-beach-6294.herokuapp.com';
//Mongoose config
//(would like to put this in separate file: http://j-query.blogspot.com/2011/11/mongoose-and-mongohq.html)
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var toDoSchema = new Schema({
title : String,
});
mongoose.connect('mongodb://127.0.0.1:27017/data');
mongoose.model('toDo', toDoSchema);
// saves to 'todos' collection in 'data' database rather than todo
var toDo = mongoose.model('toDo');
function add(response, request) {
console.log("Request handler 'add' was called.");
post_handler(request, function(request_data) {
var todo = new toDo({
title : request_data.note
});
todo.save(function(err){
console.log("saving"+ request_data.note);
if(!err){
console.log('todoItem saved.');
}
});
});
response.writeHead(301, {'Location':appURL + '/list', 'Expires': (new Date).toGMTString()});
response.end();
}
function list(response) {
console.log("Request handler 'list' was called.");
toDo.find({}, function(err,todos) {
response.writeHead(200, {'Content-Type': 'text/html'});
response.write('Todo List');
response.write('<h1>ToDo list</h1>');
response.write('<ul>');
todos.forEach(function(todo) {
response.write('<li>' + todo.title + ' <a href="remove?id='+todo._id+'">remove</a></li>'); ;
})
var form = ''+
''+
''+
''+
''+
'';
response.end(form);
});
}
function remove(response, request) {
console.log("Request handler 'remove' was called.");
parsedURL = url.parse(request.url, true);
console.log(' _id to remove ' + parsedURL.query['id']);
toDo.findOne({_id: parsedURL.query['id']},function(err,docs){
console.log('deleting' + docs); //Remove all the documents that match!
docs.remove();
docs.save();
// 301 permanent redirect to list\
response.writeHead(301, {'Location':appURL + '/list', 'Expires': (new Date).toGMTString()});
response.end();
});
}
function post_handler(request, callback){
var _REQUEST = { };
var _CONTENT = '';
if (request.method == 'POST')
{
request.addListener('data', function(chunk)
{
_CONTENT+= chunk;
});
request.addListener('end', function()
{
_REQUEST = querystring.parse(_CONTENT);
callback(_REQUEST);
});
};
};
exports.add = add;
exports.list = list;
exports.remove = remove;
The code uses mongoose to connect to MongoDB. I learnt how to use Mongoose from this post.
Aside from the javascript there are a few more files to enable the application to work with Heroku:
Procfile: a file called “Procfile” with no file extension should be added to work with foreman. The foreman section is noted below, but when you enter the command “foreman start”, the foreman process looks for the procfile within the folder and excutes the commands listed. Note the file must be called “Procfile” with a capital P. There is only one line in this file:
web: node index.js
package.json:
{
"name": "mongoHQ-LocalNotes",
"version": "0.0.1",
"dependencies": {
"express": "2.2.0",
"mongoose": "2.0.2"
}
}
package.json contains all the packages that the application uses, such as express and mongoose. This is required by heroku so that it can load the supporting packages when the application is uploaded.
One final thing to mention about the code was that the javascript in the requestHandler.js file contains the config for connecting to MongoDB. This is pointing to a database called ‘data’, but though the collection is called “todo”, for some reason the mongoose driver pluralises this to “todos”. Both the database and collection will be created implicitly when this code runs but if you wish to see the data in the console you must enter the following in a terminal window:
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ mongo
MongoDB shell version: 2.0.2
connecting to: test
> use data
switched to db data
> db.todos.find()
{ "title" : "pick up washing", "_id" : ObjectId("4ee7c23b889913d90c000001") }
{ "title" : "do this", "_id" : ObjectId("4ee9b1996541801e08000010") }
{ "title" : "do that", "_id" : ObjectId("4ee9b22306737c2708000011") }
>
so use todos.find() instead of todo.find().
To enable express to work and other packages you have to install npm, which you will call to install each package. You can install npm with this curl command from a terminal window:
curl http://npmjs.org/install.sh | sudo sh
Once this is done you can install express:
npm install express
and mongoose:
npm install mongoose
The final piece of the jigsaw to get all this working locally before deploying to Heroku is to install Foreman. Foreman is a ruby application, so requires ruby to run it. To install ruby, type the following into a terminal window:
alex@ubuntu:~$ sudo apt-get install ruby1.9.1
alex@ubuntu:~$ sudo apt-get install rubygems
Then to install Foreman:
alex@ubuntu:~$ sudo gem install foreman
on the first try, foreman failed to install with the error message:
errors with Invalid gemspec in [/var/lib/gems/1.8/specifications/foreman-0.26.1.gemspec]: invalid date format in specification: “2011-11-10 00:00:00.000000000Z”
Invalid gemspec in [/var/lib/gems/1.8/specifications/term-ansicolor-1.0.7.gemspec]: invalid date format in specification: “2011-10-13 00:00:00.000000000Z”
The solution was to edit gemspec files by opening nautilus (Ubuntu’s explorer application) with root privileges; Hold down Alt+F2 then type “gksudo nautilus” and browse to, then edit (remove “.000000000Z”) and save the .gemspec files.
The command sudo gem install foreman should now install Foreman. To test this browse to the folder containg the Procfile created earlier and enter the following command:
foreman start
The output should be
19:36:31 web.1 | started with pid 4063
19:36:32 web.1 | Server has started on port 5000
or similar…
4. Setup MongoHQ
The next job was to get the app talking to a remotely hosted database. For this I used MongoHQ. Setting up MongoHQ is fairly simple. All you need to do is sign up, create a database (I created a database called “data”) and then go to the Database Info tab. Just make a note of the Mongo URI such as mongodb://:@environment.mongohq.com:12345/data , which can then be used in the mongoose.connect call in requestHandler.js. You don’t need to create collections as this can be created implicitly through the ToDo app. To test the MongoHQ database, just replace the url to localhost with the Mongo URI supplied by MongoHQ in the mongoose.connect call.
5. Deploy to Heroku
By this time I had all the aspects of the application in place. The final job was to push it onto Heroku. This post from Heroku describes the whole process. Before you put anything onto heroku, you need to register with them first. The next step is to install the heroku toolbelt, which can be done through some commands on your terminal:
>sudo apt-add-repository 'deb http://toolbelt.herokuapp.com/ubuntu ./'
>curl http://toolbelt.herokuapp.com/apt/release.key | sudo apt-key add
>sudo apt-get update
>sudo apt-get install heroku-toolbelt
once this install is complete you can start using heroku commands from the terminal.
Everything is in place to deploy to heroku using the cedar stack. Below is the output from the terminal when deploying my ToDo list app to heroku:
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ git init
Initialized empty Git repository in /home/alex/NodeProjects/MongoHQLocalNotes/.git/
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ git add .
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ git commit -m "init"
[master (root-commit) 644f613] init
Committer: Alex
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:
git config --global user.name "Your Name"
git config --global user.email you@example.com
After doing this, you may fix the identity used for this commit with:
git commit --amend --reset-author
15 files changed, 477 insertions(+), 0 deletions(-)
create mode 100644 Procfile
create mode 100644 Procfile~
create mode 100644 index.js
create mode 100644 index.js~
create mode 100644 model.js
create mode 100644 model.js~
create mode 100644 mongoose.js
create mode 100644 mongoose.js~
create mode 100644 package.json
create mode 100644 package.json~
create mode 100644 requestHandlers.js
create mode 100644 requestHandlers.js~
create mode 100644 router.js
create mode 100644 server.js
create mode 100644 server.js~
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ heroku create --stack cedar
Creating evening-beach-6294... done, stack is cedar
http://evening-beach-6294.herokuapp.com/ | git@heroku.com:evening-beach-6294.git
Git remote heroku added
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ git push heroku master
Warning: Permanently added the RSA host key for IP address '50.19.85.132' to the list of known hosts.
Counting objects: 19, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (17/17), done.
Writing objects: 100% (19/19), 3.93 KiB, done.
Total 19 (delta 6), reused 0 (delta 0)
-----> Heroku receiving push
-----> Node.js app detected
-----> Fetching Node.js binaries
-----> Vendoring node 0.4.7
-----> Installing dependencies with npm 1.0.94
> mongodb@0.9.6-7 install /tmp/build_ewue4f2tjx88/node_modules/mongoose/node_modules/mongodb
> bash ./install.sh
================================================================================
= =
= To install with C++ bson parser do =
= the parser only works for node 0.4.X or lower =
= =
================================================================================
Not building native library for cygwin
mongoose@2.0.2 ./node_modules/mongoose
├── colors@0.5.0
├── hooks@0.1.9
└── mongodb@0.9.6-7
express@2.2.0 ./node_modules/express
├── mime@1.2.4
├── qs@0.4.0
└── connect@1.8.3
Dependencies installed
-----> Discovering process types
Procfile declares types -> web
-----> Compiled slug size is 7.9MB
-----> Launching... done, v4
http://evening-beach-6294.herokuapp.com deployed to Heroku
To git@heroku.com:evening-beach-6294.git
* [new branch] master -> master
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ heroku ps
Process State Command
------- ---------- -------------
web.1 up for 10s node index.js
alex@ubuntu:~/NodeProjects/MongoHQLocalNotes$ heroku logs
2011-12-16T23:53:13+00:00 heroku[slugc]: Slug compilation started
2011-12-16T23:53:19+00:00 heroku[api]: Config add PATH by lalexgraham@hotmail.com
2011-12-16T23:53:19+00:00 heroku[api]: Release v3 created by lalexgraham@hotmail.com
2011-12-16T23:53:19+00:00 heroku[api]: Deploy 1663f77 by lalexgraham@hotmail.com
2011-12-16T23:53:19+00:00 heroku[api]: Release v4 created by lalexgraham@hotmail.com
2011-12-16T23:53:19+00:00 heroku[web.1]: State changed from created to starting
2011-12-16T23:53:20+00:00 heroku[slugc]: Slug compilation finished
2011-12-16T23:53:22+00:00 heroku[web.1]: Starting process with command `node index.js`
2011-12-16T23:53:22+00:00 app[web.1]: Server has started.
2011-12-16T23:53:22+00:00 heroku[web.1]: Error R11 (Bad bind) -> Process bound to port 8888, should be 14296 (see environment variable PORT)
2011-12-16T23:53:22+00:00 heroku[web.1]: Stopping process with SIGKILL
2011-12-16T23:53:23+00:00 heroku[web.1]: State changed from starting to crashed
2011-12-16T23:53:23+00:00 heroku[web.1]: State changed from crashed to created
2011-12-16T23:53:23+00:00 heroku[web.1]: State changed from created to starting
2011-12-16T23:53:24+00:00 heroku[web.1]: Process exited
2011-12-16T23:53:26+00:00 heroku[web.1]: Starting process with command `node index.js`
2011-12-16T23:53:26+00:00 app[web.1]: Server has started.
2011-12-16T23:53:27+00:00 heroku[web.1]: Error R11 (Bad bind) -> Process bound to port 8888, should be 20500 (see environment variable PORT)
2011-12-16T23:53:27+00:00 heroku[web.1]: Stopping process with SIGKILL
2011-12-16T23:53:28+00:00 heroku[web.1]: State changed from starting to crashed
2011-12-16T23:53:28+00:00 heroku[web.1]: Process exited
2011-12-16T23:55:43+00:00 heroku[api]: Scale to web=1 by lalexgraham@hotmail.com
2011-12-16T23:59:34+00:00 heroku[slugc]: Slug compilation started
2011-12-16T23:59:39+00:00 heroku[api]: Deploy f5f5fdb by lalexgraham@hotmail.com
2011-12-16T23:59:39+00:00 heroku[api]: Release v5 created by lalexgraham@hotmail.com
2011-12-16T23:59:40+00:00 heroku[web.1]: State changed from crashed to created
2011-12-16T23:59:40+00:00 heroku[web.1]: State changed from created to starting
2011-12-16T23:59:40+00:00 heroku[slugc]: Slug compilation finished
2011-12-16T23:59:41+00:00 heroku[web.1]: Starting process with command `node index.js`
2011-12-16T23:59:42+00:00 app[web.1]: Server has started.
2011-12-16T23:59:42+00:00 heroku[web.1]: State changed from starting to up
….so all thats left is to now browse to the url provided by Heroku (in this case http://evening-beach-6294.herokuapp.com, and you should see your app on a public url. Now have a cup of tea to celebrate:)
EDIT
I have since done the following…
1.amended the source code, adding a meta tag to the html in the output on requestHandlers.js
uploaded source code to github at https://github.com/lalexgraham/ToDoList
2.uploaded a revised version of the code to Heroku and github
To update both github and heroku you need to enter the following commands whilst in the terminal and browsed to the folder containing the code:
git add .
git commit -m “commit message”
git push heroku master (for heroku)
git push origin master (for git hub)
Alex Graham 