Monitoring ZFS pools performance using zpool iostat

Performance, performance, performance; this is what we hear in all software development and management sessions. ZFS provides few utility commands to monitor one or more pools’ performance.

You should remember that we used fsstat command to monitor the UFS performance metrics. We can use iostat subcommand of the zpool command to monitor the performance metrics of ZFS pools.

The iostat subcommand provides some options and arguments which we can see in its syntax shown below:

iostat [-v] [pool] ... [interval [count]]

The -v option will show detailed performance metrics about the pools it monitor including the underlying devices.  We can pass as many pools as we want to monitor or we can omit passing a pool name so the command shows performance metrics of all commands.  The interval and count specifies how many times we want the sampling to happen what is the interval between each subsequent sampling.

For example we can use the following command to view detailed performance metrics of fpool for 100 times in 5 seconds interval we can use the following command.

# zpool iostat -v fpool 5 100
The output for this command is shown in the following figure.

The first row shows the entire pool capacity stats including how much space were used upon the sampling and how much was available. The second row shows how many reads and writes operations performed during the interval time and finally the last column shows the band width used for reading from this pools and writing into the pool.

The zpool iostat retrieve some of its information from the read-only attributes of the requested pools and the system metadata and calculate some other outputs by collecting sample information on each interval.

And GlassFish v3 is Here

The long awaited and the most looked upon version of GlassFish released today. GlassFish v3 fully implements Java EE 6 specification which means EJB 3.1, Servlet 3, JAX-RS, JPA 2, Contexts and Dependency Injection for Java EE, Bean validation, Java EE profiles and so on.


GlassFish is not only the most up to date application server but it also benefits from a very good architecture. GlassFish architecture provides extensions points and contracts which 3rd party developers can use to add functionality to GlassFish (even the administration console is plubable).


GlassFish v3 is an important milestone in GlassFish life because now it is fully based on OSGI modularity framework which means GlassFish can be integrated into a bigger OSGI system.


In addition to Java EE profiles, GlassFish v3 is also available as an embedded application server which we can use for testing purpose or any other kind of in process use cases.


What I like the most about GlassFish is its close integration with many other well developed products like OpenESB, OpenSSO, IDEs, and so on in addition to its superb performance and administration channels.


GlassFish v3 adds another reason to make one consider it as the prime option for deploying simple and complex application, and that is its extensibility which so far made it possible to host different kind of scripting language based applications like application based on RoR or PHP in the same process which hosts the Java EE applications.


GlassFish v3 is available for download at:


Like GlassFish v2, Sun Microsystems provides support for GlassFish if you need a higher level of assurance and guranteed support. to get more information on the provided support model take a look at GlassFish Enterprise Home-page.


And if you are interested there is a GlassFish V3 Refcard published by DZone and authored by me which introduces GlassFish v3 in more details and gives you all you need to start working with GlassFish v3.



I have an Upcoming Book about GlassFish which is due to be published on April 2010 by Packt Publishing. The book mainly discuss GlassFish security (administration and configuration), Java EE security and using OpenSSO to secure Java EE applications in general and Java EE web services in particular.


To learn more about Java EE 6, you can take a look at Sun Java EE 6 white paper located at:


Another way for you to meet GlassFish folks, other GlassFish community members and to learn more about GlassFish v3 is to join the Virtual Conference about GlassFish v3 which is supposed to take place on 15th of December.

JPA and a First tier application performance problems

Well, this is the second time I am using the new blogging platform of This time it was much easier and with less hurdle.

I want to write about JPA, performance, caching and related things. Though it wont be very well organized but it may come useful for new JPA developers. recently I have assigned a task to work on performance problems of a first tier application and further extend it with some new forms and reports and address some usability glitch in the software. The application was developed by a company which no longer exists. The developer company was dissolved before they finishing the development cycles.

The performance issue was very important for the customer, so I start investigating the performance problem first. First things first, I run the application on client machine and hell it was slow, very slow indeed. it took some 15 seconds for the application to open a Jframe with two auto completing JCombobox on it. I though maybe it is the first time I am accessing the database and application is trying to initialize a soft client side cache. So, I closed the JFrame and open it again. the result was almost the same. I checked network and DNS resolving was fast and no substantial network delay was in place. I thought the problem is either in a poor development or something is wrong on database machine.

I asked an operator about the performance and I understand that is a data access problem. and I should focus on that area.

I opened a terminal to Database server machine and checked how much load it is taking when I open that particular JFrame, well there was no substantial load on the server. indeed there was no load on the server. I briefly check to see what kind of indexing we have on two tables where I though are used to load items of JComboBox/se and hell, No index was created. The developers either was crazy not to have any index except for the default PK index or they have forgotten to execute their post installation script.

I asked for the source code of the application to see what is going on in the application code. The application is based on TopLink Essential which is a part of GlassFish project, contributed by Oracle for data access. I had some previous experience with Toplink and I was sure that there is nothing wrong with the persistence provider itself.

Application source code was documented but not enough to understand the code easily. I locate that JFrame form and checked how they are loading the content of JCombobox/es. Well, they were calling an static method from a DAO class which returns a list an Object named A then it was iterating over the list and loading the JComboBox with A.property1 of. I though why on earth they are fetching the entire object to use on property?

I checked the second JC and I found the same steps, A DAO method were called which returns a list and then the JC was loaded using D.property1.
I start looking at these two entities to see what is going on with them and I find that each of them has a one to many relation with at least one other entities. So we basically had something like:

A —–1—–N—–>B——1———-N——>C
D —–1—–N—–>E

There were no FetchType determined for the relationships so, Toplink assume FetchType.EAGER for the relationships by default. The FetchType.EAGER means that JPA eagerly loads all related records from database and create all corresponding objects before we even start using them.
What does that means? It Means that for the first JComboBox that client application creates num(A)+num(B)+num(C)+num(A)*num(B)*num(C) objects just to show a list of names in a JComboBox. Very funny hu?

I create a development database setup the development environment (NetBeans works well with ANT based projects) and start changing the code. I didn’t change the fetch types instead I changed the JPA queries to ensure that I am only loading one field and not all fields which have eager fetching relationships. I compiled and problem for this particular form were solved. I added two indexes to ensure that I am getting the result as fast as possible.

I was thinking about JPA caching and why JPA was not able to improve the overall performance after I closed the form and re-opened it. I think the data load was more than what JPA could cache by default.

I will blog more about these application and its problems. I believe it was either a prototype of the developers were intended to reiterate over the code to improve the performance.

GlassFish version 2 monitoring capabilities

GlassFish version 2 monitoring capabilities

GlassFish version 2 provides good mentoring information about different aspects of application server and all of your applications that are deployed into GlassFish but before it can show those statistical information you need to configure its monitoring service. Figure 1.

Although this statistical and monitoring information and the way that they are presented is not as luxurious as weblog and websphere one’s but they are good enought to help you track what is going in your server.

To enable monitoring service of GlassFish you need Commence with openning administration console, Locate the configuration node and expand it now select monitoring now choose all components that you want to monitor, select high for more details and sure more overhead and low for less details and less overhead. Figure 1.

Configure your monitoring

Now you should be able to see some monitoring detailsby going to monitoring pages, monitorig pages are placed under different nodes in administration console based on your installation profile. If you are not sure about your glassfish profile, follow the developer profile item.

  • For the cluster or enterprise profile In the tree component, select Common Tasks, then click View Monitoring Data.
  • For the developer profile, in the tree component, select the Application Server node. Click the Monitor tab.

Glassfish categorize monitoring information under three categories, these categories are as follow:

Call Flow:shows you how many requests are replyed successfully or with a failure and how much time consumed for that request in different layer of application server. Figure 2

Call Flow

Runtime:this category provides information about JVM, Garbage Collection, HTTP service, Listeners (http/s), ect. for example under JMV node of these category you can use memory item to view fine grained details about your application server heap. Figure 3


Applications:Let you select an application and view details about that application components, for example you can view how much time a servlet has called and how time its call taked. Figure 4


Resources:here you can view detailed information about all of your resources like connection pools, connectors, jms connection factories, etc. For example you can view fair amount of details about your connection pool in order to hunt possible connection leake or to tune your connection pool. Figure 5


Transactions:These allows you to view what is state of transactions that are running inside your application server.

It is my first blow entry written from my Ubuntu 7.04 installation and I am happy with my switch from OpenSuse to Feisty Fawn 🙂