Getting started with CouchDB and MongoDB, Part I: CRUD operations

In a series of blog I am going to write to cover CouchDB and MongoDB as two popular document databases. In each one of the entries I will show how can you do similar tasks with each one of these two so that you get some understanding of the differences and similarities between them. I am not going to do any comparison here but reading the series you can draw conclusion on which one seems more suitable for your case based judging by how different tasks can be accomplished using these two.

Some technicalities:

  • All of the source codes for these series is available at https://github.com/kalali/nosql which you can get and run the samples.
  • All sample codes assume default port numbers unless I mention otherwise in the blog entry itself.
  • All the blog entries use CouchDB 1.3.1 and MongoDB 2.4.5 and thus I expect them to work fine with this version and any newer ones.

A little about these two databases:

  • MongoDB and CouchDB: Both of them are document databases which lets you store freeform data into the storage in form of a document with key/value fields of data. Just look at JSON to see how a document would look like.
  • Apache CouchDB: Provides built-in support for reading/writing JSON documents over a REST interface. Result of a query is JSON and the data you store is in form of JSON all through the HTTP interface. There are tens of access API implemented, each with its own unique characteristics, around the REST interface in different languages which one can use to avoid using the REST interface directly. List of these implementations are available at: Related Projects. Apache CouchDB has more emphasis on availability rather than consistency if you look at it from CAP theorem perspective. CouchDB is cross platform and developed in ErLang.
  • MongoDB: MongoDB does not provide a built-in REST interface to interact with the document store and it does not provide direct support for storing and retrieving JSON documents however it provides a fast native protocol which can be access using the MongoDB Drivers and Client Libraries which are available for almost any language you can think of, and also there are some bridges which one can use to create a REST interface on top of the native interface. MongoDB has more emphasis on consistency rather than availability if you look at it from CAP theorem perspective. MongoDB is cross platform and developed in C++.

Downloading and starting:

  • MongoDB: You can download MongoDB from  the download page and after extracting it you can start it by navigating to bin directory and running ./mongo to start the document store. The default administration port is 28017 and the access port number is 27017 by default.
  • Apache CouchDB: You can download Apache CouchDB from the download page or if you are running some Linux distribution you can get it from the package repositories or you can just build it from the code. After installation run sudo couchdb to start the server.

The basic scenario for this blog entry is to create a database, add a document, find that document back, update the document and read it back from the store, delete the document and finally to delete the database to make the sample code’s execution repeatable.

The CouchDB sample code for the sample scenario:

import javax.json.JsonObject;
import javax.json.JsonReader;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Entity;
import javax.ws.rs.core.Form;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.io.StringReader;
import java.util.UUID;

/**
 * @author Masoud Kalali
 */

public class App {
    private static final String DB_BASE_URL = "http://127.0.0.1:5984/";
    private static final String STORE_NAME = "book_ds";
    private static Client c = ClientBuilder.newClient();

    public static void main(String[] args) {

        createDocumentStore();

        addingBookToDocumentStore();

        JsonObject secBook = readBook();

        String revision = secBook.getStringValue("_rev");
        updateBook(revision);

        JsonObject secBookAfterUpdate = readBook();
        deleteBookDocument(secBookAfterUpdate);

        deleteDocumentStore();
    }

    private static void createDocumentStore() {
        System.out.println("Creating Document Store...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).request().accept(MediaType.APPLICATION_JSON_TYPE).put(Entity.form(new Form()));
        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to create document storage: " + r.getStatusInfo().getReasonPhrase());
        }
    }

    private static void addingBookToDocumentStore() {
        System.out.println("Library document store created. Adding a book to the library...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).path("100").request(MediaType.APPLICATION_JSON_TYPE).put(Entity.text("{\n" +
                "   \"name\": \"GlassFish Security\",\n" +
                "   \"author\": \"Masoud Kalali\"\n" +
                "}"));

        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to create book document in the store: " + r.getStatusInfo().getReasonPhrase());
        }
    }

    private static JsonObject readBook() {

        System.out.println("Reading the book from the library...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).path("100").request(MediaType.APPLICATION_JSON_TYPE).get();
        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to read the book document from the data store: " + r.getStatusInfo().getReasonPhrase());

        }
        String ec = r.readEntity(String.class);
        System.out.println(ec);
        JsonReader reader = new JsonReader(new StringReader(ec));
        return reader.readObject();
    }

    private static void updateBook(String revision) {
        System.out.println("Updating the GlassFish Security book...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).path("100").request(MediaType.APPLICATION_JSON_TYPE).put(Entity.text(

                "{\"_rev\":\"" + revision + "\",\"name\":\"GlassFish Security And Java EE Security\",\"author\":\"Masoud Kalali\"}"
        ));

        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to update book document in the data store: " + r.getStatusInfo().getReasonPhrase());
        }
    }

    private static void deleteBookDocument(JsonObject secBook) {
        System.out.println("Deleting the the GlassFish Security book's document...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).path("100").queryParam("rev", secBook.getStringValue("_rev")).request().buildDelete().invoke();
        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to delete book document in the data store: " + r.getStatusInfo().getReasonPhrase());
        }
    }

    private static void deleteDocumentStore() {
        System.out.println("Deleting the document store...");
        javax.ws.rs.core.Response r = c.target(DB_BASE_URL).path(STORE_NAME).request().accept(MediaType.APPLICATION_JSON_TYPE).delete();
        if (!r.getStatusInfo().getFamily().equals(Response.Status.Family.SUCCESSFUL)) {
            throw new RuntimeException("Failed to delete document storage: " + r.getStatusInfo().getReasonPhrase());
        }
    }
}

I used JAX-RS and Some JSON-P in the code to add some salt to it rather than using plan http connection and plain-text content, but that aside the most important thing that you may want to remember is that we are using HTTP verbs for different actions, for example, PUT to create, DELETE to delete, POST to update and GET to query and read. The other important factor that you may have noticed is the _rev or the revision attribute of the document. You see that in order to update the document I specified the revision string, _rev in the document, also to delete the document I use the _rev id. This is required because at each point in time there might be multiple revisions of the document living in the database and you may read an older revision and that is the revision you can update (you can update a document revision because someone else might have updated the document while you were doing the update and thus your revision is old and you are updating your own revision).

The POM file for the couchDB sample is as follow:

  

    4.0.0

    me.kalali.couchdb
    couchdb_ch01
    1.0-SNAPSHOT
    jar

    couchdb_ch01
    http://maven.apache.org

    
        UTF-8
    

    
        
            org.glassfish.jersey.media
            jersey-media-moxy
            2.2
        
        
            org.glassfish.jersey.core
            jersey-client
            2.2
        
        
            org.glassfish
            javax.json
            1.0-b02
        
    

    
        

            
                org.codehaus.mojo
                exec-maven-plugin
                1.2.1
                
                    
                        run-main
                        package
                        
                            java
                        
                    
                
                
                    me.kalali.couchdb.ch01.App
                
            
        
    


Now, doing a clean install for the CouchDB project sample we get:

Creating Document Store...
Library document store created. Adding a book to the library...
Reading the book back from the library...
{"_id":"100","_rev":"1-99fb78c8607deb8ff966e2e2b3f3f7b3","name":"GlassFish Security","author":"Masoud Kalali"}

Updating the GlassFish Security book...
Book details after update:
{"_id":"100","_rev":"2-27c86bc501856c8bbe301b5a27f8da02","name":"GlassFish Security And Java EE Security","author":"Masoud Kalali"}
Deleting the the GlassFish Security book's document...
Deleting the document store...

If you pay attention you see that the _rev attribute has changed after updating the document, that is because by default read operation reads the latest revision of the document and in the second read after we updated the document the revision is changed and document that is read is the latest revision.

The MongoDB code for the sample scenario:

import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
import com.mongodb.WriteResult;

import java.net.UnknownHostException;

/**
 * @author Masoud Kalali
 */
public class App {
    private static final String STORE_NAME = "book_ds";
    private static MongoClient mongoClient;

    public static void main(String[] args) throws UnknownHostException {

        mongoClient = new MongoClient("127.0.0.1", 27017);

        DBCollection collection = createDocumentStore();

        addBookToCollection(collection);

        DBObject secBook = readBook(collection);
        System.out.println("Book details read from document store: \n" + secBook.toString());

        updateBook(collection);

        DBObject secBookAfterUpdate = readBook(collection);
        System.out.println("Book details after update: \n" + secBookAfterUpdate.toString());

        deleteBookDocument(collection);

        deleteDocumentStore();
    }

    private static DBCollection createDocumentStore() {
        System.out.println("Creating Document Store...");
        DB db = mongoClient.getDB(STORE_NAME);
        DBCollection collection = db.getCollection("books");
        return collection;
    }

    private static void addBookToCollection(DBCollection collection) {
        System.out.println("Library document store created. Adding a book to the library...");
        DBObject object = new BasicDBObject();
        object.put("name", "GlassFish Security");
        object.put("_id", "100");
        object.put("author", "Masoud Kalali");
        collection.insert(object);
    }

    private static DBObject readBook(DBCollection collection) {

        DBObject secBook = new BasicDBObject();
        secBook.put("_id", "100");
        return collection.findOne(secBook);
    }

    private static void updateBook(DBCollection collection) {
        System.out.println("Updating the GlassFish Security book...");
        DBObject matchings = collection.findOne(new BasicDBObject("_id", "100"));
        DBObject updateStmt = new BasicDBObject();
        updateStmt.put("name", "GlassFish Security And Java EE Security");
        updateStmt.put("_id", "100");
        updateStmt.put("author", "Masoud Kalali");
        WriteResult result = collection.update(matchings, updateStmt);
        System.out.println("Updated documents count: " + result.getN());
    }

    private static void deleteBookDocument(DBCollection collection) {
        DBObject object = new BasicDBObject();
        object.put("_id", "100");
        WriteResult result = collection.remove(object);
        System.out.println("Deleted documents count: " + result.getN());
    }

    private static void deleteDocumentStore() {
        System.out.println("Deleting the document store...");
        mongoClient.dropDatabase(STORE_NAME);
    }
}

As you can see in the code we are using the Mongo API rather than using direct HTTP communication and that is because MongoDB does not provide a built-in REST interface but rather it has its own native protocol as explained above. The other important part which will be in comparison with CouchDB is absence of revision or _rev or an attribute with similar semantic role. That is because MongoDB does not keep multiple version of the document Look at CAP theorem for more information on consistency and availability attributes of these two databases.

Now, the POM file for the MongoDB sample is as follow:

 

    4.0.0

    me.kalali.mongodb
    mongodb_ch01
    1.0-SNAPSHOT
    jar

    mongodb_ch01
    http://maven.apache.org

    
        UTF-8
    

    
        
            org.mongodb
            mongo-java-driver
            2.10.1
        
    

    
        
            
                org.codehaus.mojo
                exec-maven-plugin
                1.2.1
                
                    
                        run-main
                        package
                        
                            java
                        
                    
                
                
                    me.kalali.mongodb.ch01.App
                
            
        
    

The output for doing a clean install on the MongoDB sample will be as shown below:

Creating Document Store...
Library document store created. Adding a book to the library...
Book details read from document store:
{ "_id" : "100" , "name" : "GlassFish Security" , "author" : "Masoud Kalali"}
Updating the GlassFish Security book...
Updated documents count: 1
Book details after update:
{ "_id" : "100" , "name" : "GlassFish Security And Java EE Security" , "author" : "Masoud Kalali"}
Deleting the the GlassFish Security book's document...
Deleting the document store...

The sample code for this blog entry is available at GutHub:Kalali

This blog entry is wrote based on CouchDB 1.3.1 and MongoDB 2.4.5 and thus the descriptions are valid for these versions and they may differ in other versions. The sample codes should work with the mentioned versions and any newer release.


From NetBeans to IntelliJ IDEA, Week 1

I have been using NetBeans for a long time now; Because it is easy to use, easy to understand and explain to others, it has impressive Java EE support, etc. I was working on Java EE projects or projects  involving NetBeans RCP applications with 40-50 modules (1m loc or so) which are considered small applications. I didn’t need to have all modules opened or even when I needed NetBeans handled it pretty well. After joining GlassFish team things started to change a little. GlassFish codebase was new to me and the number of maven modules in GlassFish project was way larger than what I could easily open in NetBeans IDE to be able to navigate around in the code, to find the usages of a method, or to locate all of the implementing classes of a particular interface, etc.

I opened a RFE in NetBeans bug-tracking system but till that RFE got addressed I needed to work and thus I decided to give IntelliJ a try and see how it fares with GlassFish codebase and how easy it will be for me to adopt it as the IDE of choice for my day to day work on GlassFish.

First days impressions (Positive):

  • Can import the entire GlassFish codebase and start using it, no delay in code assisting popup and amazingly no lag after the initial scanning and indexing of the entire codebase.
  • Find usage works almost perfectly, sometimes it says that the class/es are outside of the codebase which might be my fault when importing the project
  • Code completion assistant is fast, when I say fast I mean really fast.
  • I like the fact that I can choose between different available LAFs. [The CDE/Motif LAF is still part of the JRE which denotes the overzealousness of keeping the backward compatibility of the platform?]
  • I like the “Store Current Layout as Default” feature which let me store a layout and use it when the layout get too messed-up.
  • I like the “Autoscroll to source” and “Autoscroll from source” though I’d rather have the action to do it manually when the mentioned options are disabled and I want to locate a class in the project view

First days impression (Not positive ones): Some of what I am missing might be available somewhere in the IDE but I haven’t find them yet….

  • I was not able to switch between different profiles for the module, for example IDE, release, etc. Maybe it is somewhere but I have not found it yet.
  • I think the color schema is hard to read because some of the used colors are low contrast, for example the comments, annotations, are hard to read and required some adjustment to be easier to read.
  • I opened some views that I cannot close, IDETalk, commander, Ant Build… they are all on auto-hide in the vertical bar in the left side without any close button or close action in the context (right click) menu to remove the tab from the side bar. Same goes for view tabs appearing in the bottom bar and left bar. But the good thing is we can rearrange them for better accessibility.
  • I think the local history feature in NetBeans has a better UI and usability than IDEA. In NetBeans the local history is shown in the same tab as the code and not in an extra window same goes for  diff window, etc.
  • The code formatter works better in NetBeans (SHIFT+CTRL+F) compared to IntelliJ (CMD+ALT+L). I like the NetBeans hotkey better as it has something from Formatting in the hotkey. (I tried couple of unindented unglized code snippet in both).
  • The Favorites view does not allow me to add a file system directory to a favorites list which limits the favorites view usage to project members (classes, packages, modules) …

I will post another blog entry after I used IntelliJ for another two weeks to give you a more in depth comparison of how they work for Mavan projects (my comparison will be mostly on Editors and Project/ navigation and refactoring features. Nothing on other languages support, Java EE, JavaFx, application server support, etc.)

My thoughts on JSR 351, Java Identity API

Identity, something that we hear more often these days with the whole web 2.0 and social services and more and more web based public services growing around us. The identity notion is an integral part of a security system in distributed services. Developing effective software system require an effective security and access control system which java provides, not exactly in the way that it should be in 2011 but it does provide what is the bare bone necessity to develop applications and frameworks on top of it and  benefit from its presence. The identity API is going to ease the interaction between the identity providers and those who consume the identity and trust the identity providers in addition to governing and managing the identity attributes.

I was studying the JSR details and it seems to be covering everything required for the identity attributes governance and the required API for both ends of the usage including the client API the governing/producing API. The identity producing and consuming is not new and there are fair number of public identity producers like facebook, twitter, etc. and also products that system integrators can use  like OpenAM as an open source product or gorilla Commercial software products like ORACLE identity management  or IBM tivoli identity management software, etc.

In a very simple set of words, the JSR 351: The Java Identity API will be as successful as it is going to be adopted. No adoption and it will endup dying some dark corner…  Design a simple and elegant API and try to ship it with some free and easy to use service implementations and it may get some momentum, otherwise it will be a goner and people will stick with what they have. I like the new features that it is going to introduce in the decision making or authorization part but we should see how well it will be adopted by identity providers to develop the services that provides the interaction point between the JSR interface and their repositories.  Pushing it as JSR wont really do that much without a wide adoption in the community. Look at how many implementation of the JSR 115 and JSR 196 exits to be plugged into application servers supporting the contract and you will get what I am referring to by community adoption.

An analysis on the monthly increase of NetBeans IDE active users count

Few weeks ago I was talking to Jiří Kovalský about NetBeans and its 1,000,000 active users and I asked him to check whether it is possible to have the number of active users by month or not and today he got back to me with a nice chart showing number of active users for each month since Jan 2004. Jiri is  NetBeans community manager and also he is the first person in the NetBeans team I get acquaintance long ago during NetCat50.

Here I attached the chart and then I will do an short analysis of what the chart can represent.

Number of NetBeans Active users per month

Now a brief overview of the chart:

  • NetBeans team started to count the active users in Jan 04
  • During summer and new year eve there is a decline in the number of active users each year and as overall number of users growth, this decline can be seen more clearly.
  • Number of active users is increasing continuously.

Now I want to merge the above chart with another table which is different NetBeans versions release dates.

NetBeans versions release dates

  • The last line in the bar is for June 2011, The July numbers are not calculated yet.
  • It took NetBeans two years, until Jan 2006, to get 200,000 active users at the beginning but the growth in number of active users was accelerating from the beginning as chart suggests.
  • In its next 3 years, from Jan 2006 to Jan 2009 number of users increased by 400,000 to a total of 600,000 active users which means the user growth accelerated quite well. This is the post NetBeans 5 ear when each version’s  changelog had quite a large number  number of bug fixes, performance improvements and new features.
  • The biggest increase in the number of users in duration of one year can be seen between June 2010 and June 2011 with about 200,000 users. This is the second year that ORACLE was in charge of Sun and its products.
  • It looks like that after NetBeans 6.9 the number of active users is increasing faster than before and the reason is clearly the stability and performance improvement in addition to tons of new features in the core and support for PHP and C++.

As a long time user of NetBeans IDE I should say that NetBeans has come a long long way to become the IDE that we use in our daily jobs nowadays. The number of features introduced in the IDE and the number of bug fixes is enormous. You can find and try NetBeans 4 or 5 and compare it to NetBeans 7 to understand the huge distance between these two.

NetBeans seen several shifts in its direction specially during netbeans 6 time when More languages were being supported in IDE and diverse set of SOA and XML development features were being included in the IDE. Then Again another shift happened and all those features and language supported were dropped and NetBeans team put more effort into the core to make the core more stable and feature rich which as you can see in the chart has payed off pretty well.

The 1,000,000 active users number is not just a number, it shows that a vast, versatile and living community is behind NetBeans IDE as users, contributors, and the core development team. Long live the good community and the good IDE.

A Simple Netbeans Module , a Gmail Checker…

Some times ago I participate in NetCat 50 – a process for NetBeans IDE version 5 QA -During NetCat I find that NetBeans RCP is something that I can Pick for later Swing based clients.

I start learning NetBeans RCP platform during NetCat 50 , and to evaluate my learnings I write a simple Module for NetBeans IDE .

The module is a GMail Checker , Module make it possible for you to see your new emails within your IDE and allows you to rotate between subjects.

I write Down a Tutorial on building NetBeans Module and as a Case study I showed how user can build GMail checker him/herself.

To build that module I used another Open Source project , http://g4j.SourceForge.Net, which is Java library to access GMail service.

After installing the module you will see a new toolbar in your IDE , the toolbar will be like following image :

by clicking on toolbar , it will start connection to Gmail. a NetBeans Standard progress bar will show the the progress in IDE bottom line

a NetBeans Standard progress bar will show the the progress in IDE bottom line

after it fetch information from GMail it will show you a something like

which tell you how much new email you have , and how much space of your GMail is occupied Now you can allow GMail Checker to rotate between subjects or do it yourself using provided buttons

You can configure the GMail checker Options trough Its configuration panel which is an standard NetBeans configuration panel.

Configuration panel is something like the following image. Gmail Checker Option Dialog

I set up an update center for my further NetBeans module , right now , you can check this UC and get your GMail Checker from there.
http://www.Solarisict.com/updates.xml
If you prefer to install the module from your local drive then , you can download all in one archive which contains all required modules for GMail checker.

you can get the source code for GMail checker from NetBeans web site , this archive contain all source code , sound files and images that I used to build this module the link to get the zip archive is here

To add a new update center to NetBeans IDE and many other NetBeans Tips and tricks take a look at Geertjan Weblog and for more NetBeans related information look at another big blog , Roumen Weblog