Oct 31, 2012

Global JNDI support in Virgo Server 3.5 for Tomcat


This rather long post analyses four solutions to the problem of using data sources, and more in general JNDI in Virgo, the 4th being the one I recommend and decided to use, which consists in leveraging Tomcat's built-in JNDI provider in Eclipse Virgo Server for Apache Tomcat.

If you are not interested in the reasons why I dropped the first three, jump directly to the fourth.

Even if the post is mostly focused on JDBC data sources, once Tomcat JNDI provider is exposed to the application it can be used for any type of resource, not only data sources.

Most of the credits for this solution go to my colleague Stefano Malimpensa.

1. JDBC data sources in OSGi

The most correct approach to obtain a JDBC data source in a pure OSGi enterprise application consists in using the OSGi JDBC Service (see the official OSGi JDBC specification).  In Virgo, that means using Gemini DBAccess.

However, in my humble opinion Gemini DBAccess is not an optimal solution for a number of  reasons:

  • Gemini DBAccess is currently available only for Derby, and to use a different database you need to write your own implementation. Not a big issue but some extra effort anyway.
  • To integrate DBAccess with EclipseLink you should probably use Gemini JPA, which is affected by a bug that prevents connection pooling from working and is therefore not usable in production https://bugs.eclipse.org/bugs/show_bug.cgi?id=379397
  • When using DBAccess with Gemini JPA you need configure connection parameters in each bundle's persistence.xml. I find this inconvenient because it is necessary to repackage the bundles every time the connection parameters change, and due to the modular nature of OSGi one complex application may include several bundles with persistence units.
  • If DBAccess is used without GeminiJPA, DBAccess will provide only a data source factory, and it will be the responsibility of your code to instantiate and configure the data source (e.g. pass in user name, password etc). In such case you would need to support a configuration file to let system administrators easily change connnection parameters, which is again extra effort.
  • DBAccess requires the OSGi registry, which means it would not work with legacy code or third party libraries written for J2EE. 
As the name implies, Gemini DBAccess is tailored to data base resources. If you want a single, unified approach for looking up any type of resource, then it's not a good fit for you.

2. Full JNDI in OSGi

 At this point you may want to try Gemini Naming,  which implements the OSGi JNDI service specification. Even Gemini Naming is in my humble opinion not an optimal solution:
  • There is no configuration console, nor a configuration file: the only way you can bind resources in the JNDI namespace is programmatically. This implies a lot of boring initialisation code, and you probably need to support a configuration file to let system administrators easily change configuration parameters, which is again extra effort.
  • Gemini Naming requires the OSGi registry, which means it would not work with legacy code or third party libraries written for J2EE.

3. Local JNDI declared inside a Web App


Virgo supports JNDI lookups for data sources inside a Web App. To achieve this you have to:
  • Include in the Web application the JDBC driver(s) and the pool implementation (e.g. Apache DBCP or Tomcat JDBC) as jars in your WEB-INF/lib folder
  • Configure web.xml to list the usual JNDI resource-refs
  • Include a Tomcat context.xml file in the Web App and configure it as explained here and here
The above will work for JDBC data sources but has the following draw backs:
  • You must include the JDBC driver and pool in every Web App of yours. This means that each Web App will have its own pool, even if they connect to the same database, and that you must repackage the WAR if you need to update the JDBC driver
  • JNDI lookup will work only in a thread originated by a HTTP request. This means that application bundles that are not WARs will not be able to obtain the data source via a JNDI lookup, unless their code is executed by a thread started by the Web container. In fact, the JNDI lookup will fail from threads created by Equinox: this is for example the case of code that observes OSGi framework lifecycle events (BundleListener) and need access the database when a bundle is installed or uninstalled.
In my case the main show stopper to this solution is the thread issue described above, because my application must access the database (and therefore the data source) from threads that are not always started by the Web container. If you are interested in the historical roots of this apparently strange limitation, I recommend reading this post by Neil Bartlett.

4. Tomcat global JNDI registry


Another option is available, which is not affected by any of the above issues and limitations and it consists in using Tomcat's global JNDI support. You gain a general purpose well tested and well documented JNDI registry capable of deploying any type of resource, not only data-sources (can even be extended to support custom resource types http://tomcat.apache.org/tomcat-6.0-doc/jndi-resources-howto.html#Adding_Custom_Resource_Factories).

Please note that this solution will make the global JNDI namespace available, disabling the Web App java:comp/env context. In order words, you cannot use the java:comp/env prefix in your JNDI lookups. This should be an acceptable limitation, given that the java:comp/env prefix in any case would work only within a Web App and not from a plain OSGi bundle.

In order to use the Tomcat  JNDI registry for data sources in Virgo the following mandatory steps are required:

  1. Create a bundle fragment for Catalina to extend Virgo's Tomcat with connection pooling support. The Virgo distribution contains in fact a stripped down version of Tomcat that removes the libraries required for JDBC pooling. Luckily enough, you can create a fragment to contribute the libraries back to Catalina. You just need to make sure that your fragments are placed in the bundle repository folder, not in pickup.
  2. Create a bundle fragment for Catalina to make the required JDBC driver(s) available to the server and the application
  3. Create a fragment that gets the JNDI context from Tomcat and that registers in the JVM a global InitialContextFactoryBuilder
  4. Edit tomcat-server.xml and define your global resources

All the above fragments are a convenient method for extending Tomcat/Catalina to use third party libraries whose Java packages were not originally imported by the Virgo bundles. For further details refer to this post by Glyn Normington, the project lead of Virgo. If you are working with Virgo his personal blog is a must read!



1. The pool bundle fragment

Here is a sample for Apache DBCP. The MANIFEST.MF below is added to the DBCP JAR, that's why there is no Bundle-Classpath. Mind the fragment host header.

Manifest-Version: 1.0
Fragment-Host: com.springsource.org.apache.catalina
Bundle-ManifestVersion: 2
Bundle-Name: Tomcat DBCP
Bundle-SymbolicName: org.apache.tomcat.dbcp
Bundle-Version: 7.0.27
Bundle-Vendor: apache
Bundle-RequiredExecutionEnvironment: JavaSE-1.6
Import-Package: javax.management,
 javax.management.openmbean,
 javax.naming,
 javax.sql,
 org.apache.juli.logging


2. The JDBC driver bundle fragment

Here is a sample for PostgreSQL. The MANIFEST.MF below is added to the PostgreSQL driver JAR, that's why there is no Bundle-Classpath. Mind the fragment host header.

Manifest-Version: 1.0
Fragment-Host: com.springsource.org.apache.catalina
Bundle-ManifestVersion: 2
Bundle-Name: PostgreSQL JDBC Driver
Bundle-SymbolicName: org.postgresql.jdbc.catalina
Bundle-Version: 9.2.1000
Bundle-Vendor: postgresql
Bundle-RequiredExecutionEnvironment: JavaSE-1.6


3. JNDI bridge fragment

Create a fragment with the following MANIFEST.MF that includes the two classes below and place it in the repository folder.

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: JNDITomcatBridge
Bundle-SymbolicName: jndi.tomcat.bridge
Bundle-Version: 1.0.0.qualifier
Bundle-Vendor: org.example
Fragment-Host: com.springsource.org.apache.catalina;bundle-version="7.0.26"
Bundle-RequiredExecutionEnvironment: JavaSE-1.6
Import-Package: org.apache.catalina.mbeans;version="7.0.26"

The code below consists of two classes.

The first, GlobalJNDILifecycleListener, is a GlobalResourcesLifecycleListener subclass that downcasts the server instance to get the global JNDI context and that registers in the JVM NamingManager a custom InitialContextFactoryBuilder which wraps the JNDI context obtained from Tomcat.

Other options are of course possible, including doing everything in the custom implementation of InitialContextFactoryBuilder without subclassing the listener, but whatever approach you adopt, it is important to make sure that the invocation to NamingManager.setInitialContextFactoryBuilder occurs only once in the life of a Virgo server instance, because the method will fail and raise a runtime exception if it is called more than once.


import javax.naming.Binding;
import javax.naming.Context;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import javax.naming.spi.NamingManager;

import org.apache.catalina.Lifecycle;
import org.apache.catalina.LifecycleEvent;
import org.apache.catalina.Server;
import org.apache.catalina.mbeans.GlobalResourcesLifecycleListener;
import org.apache.juli.logging.Log;
import org.apache.juli.logging.LogFactory;

public class GlobalJNDILifecycleListener extends GlobalResourcesLifecycleListener {
    private static final Log log = LogFactory.getLog(GlobalJNDILifecycleListener.class);

    @Override
    public void lifecycleEvent(LifecycleEvent event) {
        super.lifecycleEvent(event);
        if (Lifecycle.START_EVENT.equals(event.getType())) {
            Server server = (Server) event.getLifecycle();
            Context ctx = server.getGlobalNamingContext();
            ContextFactory factory = new ContextFactory(ctx);
            try {                
                NamingManager.setInitialContextFactoryBuilder(factory);
                log.info("Published Global Naming as default InitialContext");
                logJNDIEntries(ctx, null);
            } catch (NamingException e) {
                log.error("Naming Exception:", e);
            }
        }

    }

    private void logJNDIEntries(Context context, String prefix) throws NamingException {

        NamingEnumeration<Binding> namingEnumeration = context.listBindings("");

        while (namingEnumeration.hasMoreElements()) {
            Binding binding = namingEnumeration.next();
            String nameEntry = binding.getName();
            String fullName = (prefix == null || prefix.equals("") ? nameEntry : prefix + "/" + nameEntry);
            String entryClassName = binding.getClassName();
            if (Context.class.getName().equals(entryClassName)) {
                Context ctx = (Context) binding.getObject();
                logJNDIEntries(ctx, fullName);
            } else {
                log.info("Found: " + fullName);
            }
        }
    }

}
import java.util.Hashtable;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.naming.spi.InitialContextFactory;
import javax.naming.spi.InitialContextFactoryBuilder;

/**
 * Simple implementation of {@link InitialContextFactory} that returns the global {@link InitialContext}
 * obtained from Tomcat
 *
 * @author giamma, stefano
 *
 */
public class ContextFactory implements InitialContextFactoryBuilder, InitialContextFactory {

    private Context context;

    public ContextFactory(Context context) {
        this.context = context;
    }
   
    @Override
    public InitialContextFactory createInitialContextFactory(Hashtable environment) throws NamingException {
        return this;
    }

    @Override
    public Context getInitialContext(Hashtable environment) throws NamingException {
        return context;
    }

}

4. tomcat-server.xml

 

Replace the listener with yours and declare your JNDI resources:
 
 <-- Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />  -->
 <Listener className="com.example.catalina.mbeans.GlobalJNDILifecycleListener"/>
 
<GlobalNamingResources>
 <Resource name="jdbc/db"
           type="javax.sql.DataSource" username="user" 
           password="password" 
           factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory" 
           driverClassName="org.postgresql.Driver" 
           url="jdbc:postgresql://127.0.0.1/db" 
           maxActive="20" maxIdle="10"/>
</GlobalNamingResources>


You can now happily perform plain old new InitialContext().lookup() from every method of every class of your OSGi application deployed in Virgo, regardless of the origin of the thread.

Oct 2, 2012

Developing for Virgo server with Eclipse PDE

Last week I pushed to github an Eclipse plug-in released under the EPL license that allows using PDE plug-in projects with Virgo Tools. Source code, binaries and documentation are available here:   http://github.com/giamma/pde2virgo
 
The plug-in is pretty simple: it contributes a new builder to a PDE plug-in project and the builder takes care of copying the content of the META-INF folder, as well as any library referred by the Bundle-ClassPath header, to the project binary folder, where Virgo Tools expect those resources to be located.
 
To develop for Virgo using PDE you also have to properly configure the PDE target platform so that your plug-ins compile against the Virgo bundle repository. Plus, you must add the OSGi Bundle Nature to your PDE Plug-in projects, otherwise Virgo Tools won't let you deploy them to the server instance. More details are available in the github project wiki https://github.com/giamma/pde2virgo/wiki
 
I originally developed the plug-in in my spare time when looking for a solution that would allow me to develop OSGi bundles for Virgo Server using PDE plug-in projects rather than Virgo Tools' OSGi bundle projects. The plug-in is still affected by a couple of minor issues, but it works well and has been used by about ten colleagues of mine in the last three months. I decided to release it to the public because of this discussion https://bugs.eclipse.org/bugs/show_bug.cgi?id=329198 and because Eclipse gave me a lot in the last 8 years, so it was my turn to give back a little.

After some months developing with Eclipse Virgo Server for Apache Tomcat I can state that I am really glad I chose it as the target application server: it is a high-quality, lightweight OSGi enterprise app server capable of blending traditional Web Applications with OSGi applications, that provides great diagnostic features, meaningful error messages and that can significantly reduce your development time thanks to a fast start-up time and the ability to hot-swap OSGi bundles. Its Eclipse   integration, Virgo Tools, still needs some work, though.
 

Oct 1, 2012

EclipseLink and extensible entitites

[Originally shared on G+ the 14th of August 2012]

If you are interested in EclipseLink support for extensible entities, besides the official documentation you may want to have a look at this forum thread:
http://www.eclipse.org/forums/index.php/mv/msg/357791/880331/#msg_880331

The last post is under my name but credits go to my colleague Paolo Di Patria for writing a working test case.

Eclipse and the Ubuntu global menu

[Originally shared on G+ the 7th of August 2012]

For those working with Eclipse on Ubuntu, if you don't like the fact that the Eclipse menu does not blend into Unity's global menubar, you may want to try this hack https://bugs.launchpad.net/eclipse/+bug/618587/comments/46

You may experience some refresh glitches from time to time, but I personally prefer the menu to be integrated in Unity, especially because I use shortcuts way more than the menu.

First contact with Virgo Server, Virgo Tools, Eclipse Link

[Updated version of a post originally shared on G+ on the 22nd of June 2012]

My team and I have been working on an OSGi enterprise prototype since April 2012. We decided to use Eclipse Virgo Tomcat Server as our application server. Here is the experience collected so far, using Virgo 3.5M4 and version 1.0.0.201205110321-MILESTONE of the Virgo tools, both pre-release versions.

Documentation and other sources of information

Virgo comes with a lot of official documentation, and the documentation quality is generally high. On the other hand, if you look over the Web for articles, blog posts, forum posts etc you won't find that much besides the official documentation and the Eclipse community forums.
And while the quality of Virgo is generally very high, keep in mind that if you experience an uncommon issue, chances are that you will have to solve it on your own.
A good knowledge of the OSGi specification is really required to work with Virgo. Even long time Eclipse plug-in developers may have troubles developing proper OSGi bundles, because Eclipse plug-in developers are typically used to requiring bundles rather than importing packages, and because of the different boot delegation of Equinox when used in Eclipse, see http://blog.springsource.org/2009/01/19/exposing-the-boot-classpath-in-osgi/

Error reporting

Virgo is very good in giving you precise error and diagnostic information if your OSGi bundles fail to be resolved. The Virgo error messages are much much better than Equinox's and this is very very valuable.

Attention points

Even if Virgo is an OSGi application server, and even if it is based on Equinox, Virgo does not support split packages and does not favor required bundles.
I totally agree that split packages are evil, a very dangerous pitfall of the OSGi specification. So I was originally pretty happy with this decision, till I wanted to use the Equinox Registry in my application. In fact, I found out that the org.eclipse.core.runtime package is split across multiple bundles, and could not find a way to have the Equinox Registry bundles work "as is" in Virgo. A user in the Virgo Eclipse forums advised me to create a Plan (a virgo deployment artefact) that would include the bundles with the split package, but that did not solve my issue. I eventually worked around the problem by repackaging the Equinox registry bundles in a single bundle. Maybe other solutions are possible, but this was the fastest solution. The second important attention point is that Virgo authors dislike the use of Require-Bundle manifest header. I again understand the decision and I agree that one should prefer Import-Package, but what I dislike is that Virgo offers an alternative header attribute Import-Bundle that is Virgo specific, and does not belong to the OSGi specification. It is a sort of macro which will make Virgo enhance your bundle headers at deploy time to make them import all packages of the imported bundle. While this may appear as a convenient macro, it is Virgo specific and it will prevent your application from running in other OSGi containers or OSGi application servers (unless of course you modify the MANIFEST). So I have decided not to use it.

IDE Tools

If you are a long time Eclipse plug-in developer like me you are probably used to the PDE and love it, and you may expect to be able to develop bundles for Virgo using PDE.
Unluckily this is not possible, and you have to use the Virgo Tools, a totally different toolset. You could develop bundles with PDE, export them as binaries and deploy to Virgo, and it will of course work, but if you want to be able to build and run from the workspace your projects cannot be PDE plug-in projects but must be Virgo Tools' OSGi Bundle projects. The bad thing here is that developing with Virgo Tools is not as smooth as developing with PDE tools.
Here are a few examples:
  1. Much like PDE, Virgo Tools add a container to your project build path. The container should automatically populate the classpath according to bundle dependencies. In practice, the container is often out of sync resulting in spurious build errors, and you have to open the MANIFEST, add a blank, save and build to fix them. Plus, if you don't use autobuild, you'll need to build twice: once for the container to be updated after you change the MANIFEST and a second time for the code to be compiled after the container has updated. 
  2. If you want to import a package from another bundle in the workspace, you must enlist the other bundle in your Project References property page for the container to see it, but the MANIFEST editor will not help you, you'll need to switch to the source view and manually add the imported package. 
  3. If one of your workspace bundles exports packages that are included in some embedded JARs and you need another bundle to use classes from those packages, you'll have again to manually import the package, but you'll also need to add those JARs to your Java Project Buildpath, otherwise the dependant bundle will not compile, because the container will never see the JARs. 
There are a number of other smaller issues but the above are the most annoying.

Runtime Bugs

We spotted 3 defects, two related to the EclipseLink - Gemini JPA integration:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=381369
https://bugs.eclipse.org/bugs/show_bug.cgi?id=379397

And one in the EcilpseGemini Web component, part of Virgo Server:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=380362

As usual the Eclipse community has been great in providing support and advice to workaround the issues.

Great Virgo features

  1. Very good diagnostic tools. Logs, dumps, detailed wiring information, all those features simplify problem determination of malfunctioning OSGi applications. 
  2. User regions. You can deploy bundles in isolated user regions (exported packages and offered OSGi services will be visible and usable only by bundles in the same region). You can have the same bundles deployed more than once in different user regions, because virgo will qualify the bundle symbolic name with the region name. 
  3. Deployment artifacts galore. You can deploy to Virgo Tomcat Server bundles, plain WAR files, Plans, PARs, Web Application Bundles (WAB) and all that flexibility allows you to deploy OSGi enterprise applications and plain old Web Applications within the same Virgo instance. Plain WARs can even load classes from the deployed OSGi bundles. 
  4. Hot swapping. You can easily remove/add a deployment artefact (e.g. a bundle) from/to the server, without stopping the server and the application. Of course you need to make proper use of OSGi ServiceTrackers, the OSGi Extender pattern and write your application code to deal with such use cases. 
  5. No more issues with 3rd party libraries When working with traditional J2EE application servers a frequent issue consists in classloading problems that may originate when your application and the app server use different versions of the same third party library. In such cases, depending on the app server of use, you have different ways of bypassing the problem, but none of them is really perfect, and in general, I wish I'd never have to deal with such issues. I am glad OSGi and Virgo free me from those problems.

Conclusion

All in all Eclipse Virgo is an exciting piece of technology. It's a full featured OSGi and Web container packaged with lots of useful diagnostic features that enables blending traditional Web Applications with OSGi (enterprise) applications. True modularity is visible in Virgo itself, which comes in 3 different flavors (Nano, Kernel, Tomcat Server). I believe it's a solid product and probably the best option for OSGi enterprise applications.

How to fix unreadable Eclipse tooltips in Ubuntu 12.04

[Originally posted on G+ on May 7th 2012]

Similarly to what happened in earlier versions of Ubuntu, the default tooltip colors of the Ambiance theme (Ubuntu default) results in unreadable Eclipse tooltips (black foreground text on black background).

In earlier versions of Ubuntu you could edit the Ambiance theme via Gnome control panel and change the tooltip colors. Unluckily, the new "Appearance" settings dialog of Ubuntu 12.04 allows you to select the theme but you cannot tweak its colors anymore.

So here is how you can fix the problem.
Install gnome-color-chooser:
sudo apt-get install gnome-color-chooser 
run the tool and switch to the Specific tab. Activate the Tooltip group and set your tooltip colors (I use black foreground on white background). Then, at the bottom of the dialog select the value Theme Default for the profile combo and press apply. The next time you start Eclipse it will honour your tooltip colors.

For those who may be interested, gnome-color-chooser will change your ~/.gtkrc-2.0 file.

Jericho HTML Parser

[Originally posted on G+ on February 6th 2012]

Yesterday evening I was looking for a HTML parser that would be tolerant wrt malformed documents. I first looked at Jericho HTML http://jericho.htmlparser.net (I read about it a while back), and when looking for a comparison between Jericho and CyberNeko HTML Parser http://sourceforge.net/projects/nekohtml/ (which is mentioned in the Jericho home page), I stumbled upon JSoup http://jsoup.org/

I immediately fell in love with JSoup because:

  1. it's open source under the MIT license
  2. it's DOM oriented and declares to support HTML 5.0
  3. it's a very small library with no dependencies (CyberNeko to the contrary depends on Apache Xerces)
  4. it declares Android compatibility
  5. it offers a decent API for performing HTTP requests (design looks similar to Apache HTTP Client)
  6. it is fully documented and comes with several code examples

I made a quick test fetching and parsing a couple of HTML documents resulting from the flow of a dynamic website and everything worked fine.

Don't know how it behaves on more complex pages, but the positive comments I read in Stack Overflow and other sites give me good hope.