Dienstag, 25. März 2014

A general approach to migrating EMF models persisted into SQL based backends/stores


With Eclipse EMF you can model your business domain (meta modeling) and persist instances of it into all kind of backends. As time goes by, you tend to add functionality to your application domain which involves changing the model (which is known as 'model evolution'). If you happen to change/evolve your model such that it is not compatible with its previous version, then, you need to prepare for complex migration procedures to transform one version (an old version) of your model into the a newer version.
For Eclipse EMF, there are several frameworks that are capable of persisting models into SQL based backends:
  • Eclipse CDO
  • Eclipse Teneo
  • Eclipse Texo
This article will demo a general approach to migrating EMF models that are persisted via Eclipse Teneo.
Eclipse Teneo is a framework which allows for connecting models via Hibernate to an SQL based database.
This is done internally by translating Eclipse Ecore meta models to Hibernate mappings which are translated themselves to SQL DDL statements.
The key point of the following approach is to grab the generated Hibernate mappings and use another tool that is capable of comparing two mappings (generated at two different points in time) of the same model. This tool is introduced in the next chapter: Liquibase.

Comparing Databases with Liquibase

Liquibase is capable of comparing two different 'Databases' with each other and generating a 'diff script' for execution in SQL backends.
A 'Database' can be a connection to a JDBC/SQL database or it can be some other sort of meta structure that describes a JDBC/SQL database. This is the key point of Liquibase for integrating Ecore meta models into it. As said, with Teneo we can generate mappings of our meta models which Liquibase can use as 'Database' for creating change sets which we can use to apply to another database for model evolution purposes.

A small demo

Just imagine a domain class called Book which looks like this (revision 0 of the meta model):

(This is borrowed from the 'Extended Library Model' available when installing the EMF SDK.)

Now, let us add some additional attributes to the book EClass (revision 1 of the meta model):
The changes include:
  • a new attribute of type float called 'price' (this is no incompatible change, but used to demonstrate some simple migration steps later)
  • changed the multiplicity of author to * and renamed it to 'authors'
Now, we need to compare both meta model revisions with each other and generate the 'diff script' (Liquibase migration script):
  1. Extract revision 0 from your source repository
  2. Extract revision 1 from your source repository
  3. Start Teneo to generate the Hibernate mapping of revision 0
  4. Start Teneo to generate the Hibernate mapping of revision 1
  5. Call Liquibase to compare mapping of revision 0 with mapping of revision 1 and generate the script
  6. Review and adapt the generated migration script
This is a very abstract algorithm, but was implemented for a customer and wrapped up in a JFace wizard for simplification:
The wizard collects all needed parameters from the user/developer and runs the above algoritm. In this case, the source repository was subversion and thanks to the Subversive plugin, no or minimal code was needed to choose and checkout the revisions.

The migration script looks like this (excerpt):

Liquibase handles 'migration steps' by the notion of 'change sets'. In our case, it has generated thre change sets (other change sets are not shown in the screenshot due to simplification of the demo):
  • an 'addColumn' instruction to add the new column 'price' to the table
  • a new junction table ('createTable' instruction) to reflect the many-to-many association of Book.authors and Writer.books (which are both set as eOpposites)
  • a 'dropColumn' instruction to remove the 'old author' column from the table 
In the review process of the generated migration script, we would have to review each instruction and check whether to keep it as-is, for example, the addColumn instruction which does not do any harm to the database, or to provide custom instructions for properly transitioning incompatible changes, for example the dropColumn instruction which leads to loss of information. In short, before executing the dropColumn instruction, we have to provide a new instruction to transform all rows of Book that contain an author value/reference into new rows in the new many-to-many table.
For example:

select i.book_id, i.author_id, 0 into book_authors from item i where i.author not null

(This is pseudo-sql and may have to be adjusted for the target DBMS, but you can tell Liquibase which DBMS is valid for each custom SQL.)


We have seen a possible approach to migrating Ecore meta model instances persisted into SQL backends using Teneo by leveraging the functionality of Liquibase.

As Liquibase operates on JDBC enabled databases, it can also be applied to the other EMF frameworks that persist models to SQL backends, for example CDO.

Another interesting point is to use EMF Compare (MPatch format) to generate Liquibase conditions to check whether each meta model change has been covered by the migration script.


Liquibase: http://www.liquibase.org/
EMF: http://eclipse.org/modeling/emf/
EMF Teneo: http://wiki.eclipse.org/Teneo
EMF CDO: http://www.eclipse.org/cdo/
EMF Compare: http://eclipse.org/emf/compare

Montag, 1. Juli 2013

Eclipse E4, EMF and XWT in Action


This article will show the Eclipse technologies stack consisting of Eclipse E4, EMF and XWT in action. First, we will have a look at the functionality the application has to provide, then see how this can be done using those Eclipse frameworks.

The application should be as in the following screenshot:

Requirements: User must be able to...

  • ... create a new layered geographic map or load a list of existing ones
  • ... create new layers (Markers/Points-of-interests, Google, OpenStreetMap, OGC WMS, GML, KML, ...)
  • ... re-order the layers of a selected map
  • ... edit layers, add new Points-of-interests (POIs) to a Markers layer and/or geocode them
  • ... show a preview of the map
All business cases can already be handled by the Geoff project. The interesting part is what frameworks we want to use to implement the requirements. Obviously, as the title says, we have chosen Eclipse E4 as the platform, EMF for modeling the business domain and XWT for declaratively defining the UI via XML.

Modeling our business domain

We are not going to describe the full domain model, but present a simplified one which looks like the following:

There is a base object GeoMap which has a list of Layers which can be any of Markers, OSM, Google, etc.
The full ecore meta model is available in the Geoff git repository. To simplify the UI development, we will use  the generated edit code of the model as well.

Defining the E4 application model

There is a left pane with some parts providing the editing capabilities and there is a right pane which hosts a preview of the configured map:

Providing the contents of the parts in XWT

The entry point of the application is the map selection/creation part:

This UI consists of a TreeViewer with the maps configured so far and a Toolbar (style=VERTICAL) with the actions.

A name is given to the TreeViewer via name="mapsViewer". This allows other objects to reference it. The input is configured using XWT's powerful binding expression: {Binding path=input}. This tells XWT to bind the 'input' property of the data object (provided to the root of the UI) to the 'input' property of the JFace TreeViewer. As we are using the generated EMF Edit code of our business model, we want to use an AdapterFactoryContentProvider as a content provider for out TreeViewer. This is done using the Make object which will use E4's DI engine to create and inject a new instance of the provided class. Remember that the regular AdapterFactoryContentProvider of EMF Edit does not allow DI, so, we have to redefine the class as folows:

As you can see, we are using the @Inject annotation to get missing input, in this case, the DI will provide an AdapterFactory instance to the constructor. Furthermore, we make the content provider be able to hide all children in our root object (GeoMap), that is, we can make the content provider to only return a flat list of the elements. Background: JFace's ListViewer does not allow to show an object's image out of the box.
The label provider of our TreeViewer is configured in the same way.

Next comes a highlight of the XWT E4 integration efforts...
How would you listen for the TreeViewer's selection and put it to your IEclipseContext?

The E4 way would be:

So, lots of boilerplate code! In XWT this can be reduced to just an 'Export' instruction:

This tells XWT to listen to the viewer's singleSelection property and export it into the current IEclipseContext. Remember that Eclipse Contexts are hierarchical: if you are in a Perspective, then each part inside that Perspective has its own context. So, we have to walk one level up from the current context. In the Java code we have called context.getParent(). In the XWT Export instruction we do this by providing the level argument. In this case level="1" just means one level up, so, the export will be done in the Perspective's context and the object is available to all other parts in that Perspective.

Next, the ToolBar on the right side of the TreeViewer:

The UI definition should be self-explanatory, but there is one simplification here. We use XWT's selectionEvent attribute on each ToolItem to define the handling code. For instance, selectionEvent="newMap" tells XWT to look for a method in the handler class that is called 'newMap' and execute it once the ToolItem is pressed. The handling method looks like this:

The first thing to note is that the key point to using the E4 integration of XWT is to subclass your part's POJO from E4XWTUI.
Next, we annotate all fields we are interested in to be injected and define the event handler method newMap() which will be called once the ToolItem is pressed.

The next big step is to make different XWT UIs communicate with each other. In the prior step we listen to the TreeViewer's single selection property and export it into the current Perspective's IEclipseContext to make it available to other parts.
One other part is the Map Details Part:

Once the selection of the maps viewer changes, it will be exported for use with other UI parts.
The Map Details Part's UI looks like this:

To listen to IEclipseContext variables/objects that are set/reset, we have to use the Var instruction:

x:name="map" tells XWT to name this object "map", so other objects can reference it inside the same UI definition. varName="geoMap" tells it to listen for a variable called "geoMap" inside the active IEclipseContext chain.

Next, we can define all other UI controls, for example:

The interesting part is again the powerful expression: text="{Binding elementName=map, path=value.(g:GeoMap.name)}

This means to bind the GeoMap object's name property which can be obtained from the previous Var instruction to the text property of the SWT Text control.

The remaining UI is done following these principles.


The most important part of this article was to show XWT's E4 integration to create very complex and custom SWT UIs. This was achieved by providing the building blocks to make parts communicate with each other by publishing variables and listening to them.
The benefits we get when using these technologies stack:
  • E4's powerful DI engine and application model
  • EMF for rapid business domain modelling and code generation
  • strict separation of UI and business logic using XWT (no single SWT widget was instantiated in this application in Java code, all done in XML)
  • loosely coupled UIs that can communicate with each other via the IEclipseContext hierarchy

Freitag, 21. Juni 2013

Geocoding location names with Geoff

What is geocoding?

Imagine you have some data in a database that (logically) could be visualized on a geographical map, but you are missing the geographic coordinates (typically latitude/longitude pairs), i.e. you only have the object's logical name:
  • city names
  • postal addresses, street names
  • points of interest: restaurants and bars, libraries, shops, etc.
In order to get the coordinates, you could look up those names in a search engine and show it on a map like Goolge Maps does, then, grab its coordinates and assign them to the objects.
This process is called geocoding: assigning geographical coordinates to a location name.

Querying geocoding services

As you guess, manually geocoding location names can take a lot of time if you have a bunch of objects.
Luckily, this procedure can be automated. There are (public) geocoding services available that can be used to automatically lookup location names and get a geocoded equivalent:
  • Commercial geocoding APIs/webservices providers (Google, Microsoft, Nokia, etc.)- may be 'free' but constrained depending on how many queries you execute
    • Obtain a proper key from those providers
  • OpenStreetMap's geocoding (public) API/webservice - not to be queried in productional environments
    • Host the OpenStreetMap planet files yourself or 
    • consume from a (commercial) provider that has done this already

Using Geoff's API to query a geocoding service

The Geoff project (which is a proposed project at the Eclipse Locationtech IWG)  has a simple interface (IGeocodingService) that acts as a facade/frontend to the public/commercial geocoding services:

public List executeQuery(String query);

It takes a query string as parameter (a location name, for example) and returns a list of potential matches of type POI (point of interest) which has a LongLat (geographic coordinates) and service specific properties.
The result is a List of best matches of geocoded POIs as in general it is not always possible to uniquely identify the location name when automatic geocoding is done. In such cases, you will have to check and resolve the best match.

Currently, there is a default implementation that queries OpenStreetMaps's Nominatim to geocode location names. Other implementations may follow.

Visualizing geocoded location names with Geoff

Once you have finished the geocoding process, you can use Geoff to visualize the POIs on a map (in your Eclipse RCP application). The following steps will walk you through a simple use case...

Setup the map

Geoff has an EMF ecore meta model to define a map object.

1. create the root map container
GeoMap map = MapFactory.eINSTANCE.createGeoMap();
MapOptions opts = MapFactory.eINSTANCE.createMapOptions();

2. create an OpenStreetMap base layer
OSM layer = LayersFactory.eINSTANCE.createOSM();
LayerOptions opts = LayersFactory.eINSTANCE.createLayerOptions();

3. create a second layer that will contain some POIs (Markers)
Markers markersLayer = LayersFactory.eINSTANCE.createMarkers();markersLayer.setName("Markers");map.getLayers().add(markersLayer);

4. a) obtain a list of location names from some backend data source
String[] cities = new String[] { "Berlin", "Paris", "London", "Madrid", "Rom" };

4. b) geocode the location names and add to second layer
IGeocodingService geo = IGeocodingService.Util.getFirstFound();
for (String city : cities) { List results = geo.executeQuery(city); if (!results.isEmpty()) { POI poi = results.get(0); Marker marker = MarkersFactory.eINSTANCE.createMarker(); LonLat lonLat = TypesFactory.eINSTANCE.createLonLat(); lonLat.setLon((float) poi.getLatLon().getLon()); lonLat.setLat((float) poi.getLatLon().getLat()); marker.setLonLat(lonLat); markersLayer.getMarkers().add(marker); }}

Setup the UI container (this part is SWT/E4 specific)

public void createUI(Composite parent) {
GeoMap map = ... //map object populated in first step
GeoffMapComposite mapComposite = new GeoffMapComposite(parent, SWT.NONE);
mapComposite.setMap(map, "map");


The output should be similar to the following screenshot...

How was that done?

GeoffMapComposite is a wrapper for SWT's Browser widget. It runs the JavaScript framework OpenLayers. Once you call GeoffMapComposite.setMap(), it will use Geoff's code generation functionality to generate OpenLayers specific JavaScript code at run-time and execute it in the wrapped Browser widget.

Mittwoch, 27. März 2013

Geoff: Visualizing geographical maps in Eclipse RCP/RAP applications

The Eclipse Foundation has set up a new industry working group, namely LocationTech.
Its main focus is on

- Storage and processing of massive data volume
- Model driven design
- Desktop, Web, and mobile mapping
- Real time analysis of business critical data

With geoff (geo fast forward), I have proposed a new project that aims at providing a solution to be used for existing (or maybe new ones, too) Eclipse RCP applications to enable visualization of geographical (geo spatial) maps. It uses model driven design (EMF) to provide a declarative configuration of a layered map that can be rendered via the JavaScript framework called OpenLayers 2.12.

EclipseSource's Ralf Sternberg was very cooperative to host an RAP version of the current state on their demo server - Thanks a lot!

Geoff RAP Demo Application

You can follow the progress on GitHub: Geoff on GitHub

Freitag, 28. Mai 2010

Chained Commands Execution

Chained Commands Execution

What are commands?

The eclipse command API [1] allows for declaratively contributing UI parts (menu/toolbar items, window controls, ...) to the UI anywhere in your workbench window.
In general, this requires to
  1. create a command definition
  2. create a handler or handlers (which are conditionally enabled upon a specific expression)
  3. define a menu contribution to be shown in the UI which the user can use to trigger the execution of the command

As the command API does exist since eclipse 3.3, there are plenty of (re-)usable commands:

What are chained commands?

As you might have noticed, the core API just allows to call one specific command per menu contribution (menu/toolbar item, ...).
Often, there is a need to execute several commands by just one UI button. To allow this, you have to define a composite command that is manually written to call command X, Y, and Z.
If you want to execute command A, B, and C in a sequence, you have to provide another composite command.
Things will become more complicated if those wrapped commands require parameters to be passed to them.

To fix this, we just define a chain of commands.

A chain of commands consists of an arbitrary number of commands tied together to be executed (conditionally) in a sequence.

Example: Imagine you have a web application based on RAP [2]. Once the user accesses the web application, he is given a login form where he has to provide his login data. Once he presses the login button, he is redirected to a home perspective. If the login fails, an error message will occur.

The plugin.xml may contain this extension:




















                  value="Login denied using: user={login.user}">































This is the menu contribution extension point.
Our contributions will not go into a menu/toolbar/..., instead they will be added to a chain (locationURI="chain:chains.demo.login" where the chain's id is chains.demo.login).
Next, we define three commands to be executed by the chain:

1. the myapp.login command will authenticate the user
2. the ui.showMessage will show an error message if the user could not be authenticated
    - this will only happen if the previous command has been executed successfully, you can check this by comparing the "previous" variable's boolean value
3. the last command will change the active perspective, if the first command succeeded

Next, we have to define the chain execution menu item:
















This command menu item will execute the chain defined by the id "platform.executeChain". That is our composite command that handles chain executions.


The source of the chain command is available on google code at


Use a subversion client to checkout.


[1] http://wiki.eclipse.org/index.php/Platform_Command_Framework
[2] http://eclipse.org/rap/

Dienstag, 12. Januar 2010

Single Sourcing RAP/RCP apps


You read about RAP [1] to be able to do pretty much that is possible with RCP, and thus, decided to provide a web version of your eclipse plugin.
So, you install and setup the RAP tooling [2]. You change your target platform to compile against the RAP runtime. Several compile errors arise. After some research you read about RAP not being fully compatible with its RCP counterpart and that you will have to apply single sourcing techniques to get your plugin work on RAP.
A bit more of your research reveals the following approaches used for single sourcing:
  • use reflection to determine whether a specific functionality is available
  • swap non-working code into a bundle fragment
  • use osgi services for each functionality to be single sourced
  • use byte code weaving to patch non-working code at runtime
  • etc. (Are there any more to be listed here?)

This approach

This approach is based on a central single sourcing service (think OSGi). It provides an interface for registering and executing wrapped code by a user defined id:

Let us explain this by a simple example: In your RCP you are adding a mouse move listener to a control:

Composite comp = new Composite( parent, SWT.None );
comp.addMouseMoveListener( new MouseMoveListener() {
 public void mouseMove( MouseEvent e ) {
  System.out.println( "mouse location: "
   + e.x + ", " + e.y );
} );

As you might know, RAP does not support mouse move events, and thus, this has to be single sourced.
We, first, have to register the wrapped code, i.e. the part that is not runnable on RAP:

ISingleSourcingService service = ISingleSourcingService.INSTANCE.get();
service.registerExec( "my.execs.addMouseMoveListenerToComposite",
 new IExec() {
  public Object run( Object... params ) {
   Composite comp = (Composite) params[ 0 ];
   comp.addMouseMoveListener( new MouseMoveListener() {
    public void mouseMove( MouseEvent e ) {
     System.out.println( "mouse location: "
      + e.x + ", " + e.y );
   } );
   return null;
 } );

Then, we can call the single sourcing service to execute the wrapped code:
Composite comp = new Composite( parent, SWT.None );
service.execute( "my.execs.addMouseMoveListenerToComposite", null, comp );

This code can be used on both of the platforms, since it does not cause any errors anymore.

Missing clues

Where to register wrapped code?
The preferred way is a separate plugin to host the code. When running on RCP, you will have to make sure that this plugin has been started. On RAP, you would just exclude that plugin from deployment. The other way around is valid as well.

Where to find the sources?
You can checkout the sources from http://erdalkaraca.googlecode.com/svn/trunk/org.eclipse.labs.singlesrc.

[1] http://eclipse.org/rap/introduction.php
[2] http://eclipse.org/rap/gettingstarted.php