Tuesday 3 November 2009

GXT 2.0.1 - Grid update scrolling problem

I have hit and finally found a workaround for an annoying (and apparently fixed but not released to community) bug with updating grid rows where the row is not currently in view. The problem is that the Grid jumps to the last updated row in the result of one or more rows being updated via the ListStore that backs the Grid.

The problem can be seen in the following simple example









001 package com.cloudscapesolutions.gridupdatetest.client;
002 
003 import java.util.ArrayList;
004 import java.util.List;
005 
006 import com.allen_sauer.gwt.log.client.Log;
007 import com.extjs.gxt.ui.client.Style.LayoutRegion;
008 import com.extjs.gxt.ui.client.data.BaseModelData;
009 import com.extjs.gxt.ui.client.data.ModelComparer;
010 import com.extjs.gxt.ui.client.data.ModelData;
011 import com.extjs.gxt.ui.client.store.ListStore;
012 import com.extjs.gxt.ui.client.widget.ContentPanel;
013 import com.extjs.gxt.ui.client.widget.Viewport;
014 import com.extjs.gxt.ui.client.widget.grid.ColumnConfig;
015 import com.extjs.gxt.ui.client.widget.grid.ColumnModel;
016 import com.extjs.gxt.ui.client.widget.grid.Grid;
017 import com.extjs.gxt.ui.client.widget.layout.BorderLayout;
018 import com.extjs.gxt.ui.client.widget.layout.BorderLayoutData;
019 import com.extjs.gxt.ui.client.widget.layout.FitLayout;
020 import com.google.gwt.core.client.EntryPoint;
021 import com.google.gwt.user.client.Command;
022 import com.google.gwt.user.client.DeferredCommand;
023 import com.google.gwt.user.client.Random;
024 import com.google.gwt.user.client.Timer;
025 import com.google.gwt.user.client.ui.RootPanel;
026 
027 /**
028  * Entry point classes define <code>onModuleLoad()</code>.
029  */
030 public class GridUpdateTest implements EntryPoint {
031     private Viewport viewport;
032 
033     public void onModuleLoad() {
034         Log.setUncaughtExceptionHandler();
035 
036         DeferredCommand.addCommand(new Command() {
037             public void execute() {
038                 onModuleLoad2();
039             }
040         });
041     }
042 
043     /**
044      * This is the entry point method.
045      */
046     public void onModuleLoad2() {
047         viewport = new Viewport();
048         viewport.setLayout(new FitLayout());
049         viewport.setBorders(false);
050         ContentPanel mainPanel = new ContentPanel();
051         mainPanel.setBodyBorder(false);
052         mainPanel.setHeaderVisible(false);
053         mainPanel.setLayout(new BorderLayout());
054         final ListStore<ModelData> store = new ListStore<ModelData>();
055         store.setModelComparer(new ModelComparer<ModelData>() {
056 
057             @Override
058             public boolean equals(ModelData m1, ModelData m2) {
059                 String m1Key = m1.get("One")
060                 String m2Key = m2.get("One")
061                 if (m1Key == null || m2Key == null) {
062                     if (m1Key == null && m2Key == nullreturn true;
063                     return false;
064                 }
065                 // Log.info("Comparing " + m1Key + " and " + m2Key);
066                 return m1Key.equals(m2Key);
067             }
068 
069         });
070         List<ColumnConfig> configs = new ArrayList<ColumnConfig>();
071         configs.add(new ColumnConfig("One""One"100));
072         configs.add(new ColumnConfig("Two""Two"100));
073         configs.add(new ColumnConfig("Three""Three"100));
074         ColumnModel columnModel = new ColumnModel(configs);
075         final Grid<ModelData> grid = new Grid<ModelData>(store, columnModel);
076 
077         for (int i = 0; i < 100; i++) {
078             BaseModelData bmd = new BaseModelData();
079             bmd.set("One", i+"");
080             bmd.set("Two", i* 1000000.0);
081             bmd.set("Three"0);
082             store.add(bmd);
083         }
084         Timer t = new Timer() {
085 
086             @Override
087             public void run() {
088                 int toUpdate = Random.nextInt(100);
089                 BaseModelData bmd = new BaseModelData();
090                 bmd.set("One", toUpdate +"");
091                 bmd.set("Two", toUpdate * 1000000.0);
092                 bmd.set("Three", Random.nextInt());
093                 Log.info("Updating " + toUpdate);
094                 
095                 // DO NOT USE store.contains() IT DOESN'T USE THE 
096                 // COMPARATOR AND THE BaseModelData DOES NOT IMPLEMENT
097                 // EQUALS
098                 if (store.findModel(bmd== null) {
099                     // Darn it
100                     Log.error("ModelData not contained in store:" + toUpdate);
101                     store.add(bmd);
102                 }
103                 else {
104                     store.update(bmd);                    
105                 }
106                 
107             }
108             
109             
110         };
111         t.scheduleRepeating(1000);
112         
113         mainPanel.add(grid, new BorderLayoutData(LayoutRegion.CENTER));
114         viewport.add(mainPanel);
115         RootPanel.get().add(viewport);
116         viewport.layout();
117     }118 }





If you run the example with GXT 2.0.1 you will set a table of 100 rows and every few seconds the rows in view will change as the data is updated. The change brings the last row updated in the grid into view in the visible region of the viewport, moving the scroll bars. This is not the behaviour I would have expected, and as I say from a post on the GXT forum it looks like the problem has been found and fixed and is available for paying customers.

I based my solutions on the suggestion in the post to change the update behavior so that events are not triggered as updates are delivered and a refresh is called afterwards. This solution actually works well on a couple of fronts as it also allows for the bulk delivery of updates to the grid which fits better with my real application usage of an updating ListStore/Grid where I am retrieving updates to a dataset from the server periodically to update the ListStore. So the updated code for my example is as follows.











001 package com.cloudscapesolutions.gridupdatetest.client;
002 
003 import java.util.ArrayList;
004 import java.util.List;
005 
006 import com.allen_sauer.gwt.log.client.Log;
007 import com.extjs.gxt.ui.client.Style.LayoutRegion;
008 import com.extjs.gxt.ui.client.data.BaseModelData;
009 import com.extjs.gxt.ui.client.data.ModelComparer;
010 import com.extjs.gxt.ui.client.data.ModelData;
011 import com.extjs.gxt.ui.client.store.ListStore;
012 import com.extjs.gxt.ui.client.widget.ContentPanel;
013 import com.extjs.gxt.ui.client.widget.Viewport;
014 import com.extjs.gxt.ui.client.widget.grid.ColumnConfig;
015 import com.extjs.gxt.ui.client.widget.grid.ColumnModel;
016 import com.extjs.gxt.ui.client.widget.grid.Grid;
017 import com.extjs.gxt.ui.client.widget.layout.BorderLayout;
018 import com.extjs.gxt.ui.client.widget.layout.BorderLayoutData;
019 import com.extjs.gxt.ui.client.widget.layout.FitLayout;
020 import com.google.gwt.core.client.EntryPoint;
021 import com.google.gwt.user.client.Command;
022 import com.google.gwt.user.client.DeferredCommand;
023 import com.google.gwt.user.client.Random;
024 import com.google.gwt.user.client.Timer;
025 import com.google.gwt.user.client.ui.RootPanel;
026 
027 /**
028  * Entry point classes define <code>onModuleLoad()</code>.
029  */
030 public class GridUpdateTest implements EntryPoint {
031     private Viewport viewport;
032 
033     public void onModuleLoad() {
034         Log.setUncaughtExceptionHandler();
035 
036         DeferredCommand.addCommand(new Command() {
037             public void execute() {
038                 onModuleLoad2();
039             }
040         });
041     }
042 
043     /**
044      * This is the entry point method.
045      */
046     public void onModuleLoad2() {
047         viewport = new Viewport();
048         viewport.setLayout(new FitLayout());
049         viewport.setBorders(false);
050         ContentPanel mainPanel = new ContentPanel();
051         mainPanel.setBodyBorder(false);
052         mainPanel.setHeaderVisible(false);
053         mainPanel.setLayout(new BorderLayout());
054         final ListStore<ModelData> store = new ListStore<ModelData>();
055         store.setModelComparer(new ModelComparer<ModelData>() {
056 
057             @Override
058             public boolean equals(ModelData m1, ModelData m2) {
059                 String m1Key = m1.get("One")
060                 String m2Key = m2.get("One")
061                 if (m1Key == null || m2Key == null) {
062                     if (m1Key == null && m2Key == nullreturn true;
063                     return false;
064                 }
065                 // Log.info("Comparing " + m1Key + " and " + m2Key);
066                 return m1Key.equals(m2Key);
067             }
068 
069         });
070         List<ColumnConfig> configs = new ArrayList<ColumnConfig>();
071         configs.add(new ColumnConfig("One""One"100));
072         configs.add(new ColumnConfig("Two""Two"100));
073         configs.add(new ColumnConfig("Three""Three"100));
074         ColumnModel columnModel = new ColumnModel(configs);
075         final Grid<ModelData> grid = new Grid<ModelData>(store, columnModel);
076 
077         for (int i = 0; i < 100; i++) {
078             BaseModelData bmd = new BaseModelData();
079             bmd.set("One", i+"");
080             bmd.set("Two", i* 1000000.0);
081             bmd.set("Three"0);
082             store.add(bmd);
083         }
084         Timer t = new Timer() {
085 
086             @Override
087             public void run() {
088                 int toUpdate = Random.nextInt(100);
089                 BaseModelData bmd = new BaseModelData();
090                 bmd.set("One", toUpdate +"");
091                 bmd.set("Two", toUpdate * 1000000.0);
092                 bmd.set("Three", Random.nextInt());
093                 Log.info("Updating " + toUpdate);
094                 
095                 // DO NOT USE store.contains() IT DOESN'T USE THE 
096                 // COMPARATOR AND THE BaseModelData DOES NOT IMPLEMENT
097                 // EQUALS
098                 if (store.findModel(bmd== null) {
099                     // Darn it
100                     Log.error("ModelData not contained in store:" + toUpdate);
101                     store.add(bmd);
102                 }
103                 else {
104                     store.setFiresEvents(false);
105                     store.update(bmd);
106                     store.setFiresEvents(true);
107                     grid.getView().refresh(false);
108                     
109                 }
110                 
111             }
112             
113             
114         };
115         t.scheduleRepeating(1000);
116         
117         mainPanel.add(grid, new BorderLayoutData(LayoutRegion.CENTER));
118         viewport.add(mainPanel);
119         RootPanel.get().add(viewport);
120         viewport.layout();
121     }
122 }





The relevant changes are in lines 103-109.
1. Turn off event firing for the store
2. Update the store, as many times as you need for different rows
3. Turn events back on
4. Refresh the Grid which will refetch the data.

Hopefully there will be a release of GXT soon which fixes the problem at source.

Monday 2 November 2009

GXT BaseModelData not Serializable Workaround

I have been using the open source version of Ext GWT/GXT a great Widget library for GWT for some time now. As the product starts to mature the open source releases are starting to lag the paid for subscription releases as the guys at Ext JS start to try and generate some revenue out of the product they have developed, which is fair enough, I guess.

The latest open source version at the time of writing is 2.0.1.

However in my latest project I needed to be able to Java serialize some of my GXT data using Java serialization. The main data type in the GXT world is the BaseModelData which is a default implementation of the ModelData interface. BaseModelData is marked as java.io.Serializable unfortunately the implementation is not serializable in reality as it contains an RCPMap object which is not java.io.Serializable.

This issue has been raised with the guys at Ext JS, and apparently has been fixed in a release post 2.0.1, but as there is no open source release it is not available for me to use at the moment.

I looked at the possibility of patching up the source, but its not a trivial fix and due to the limitations of GWT security white list you can not override standard Serialization or Externalization to work around the issue.

So I have taken a different approach for the time being and have introduced my own implemenation of the ModelData interface which allows me to use the GXT widgets whilst at the same time being able to serialize them on the server side.

The implemenation is


import java.io.Serializable;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;

public class View /*extends BaseModelData*/ implements ViewI {

   
@Override
   
public boolean equals(Object obj) {
       
if (obj == null) return false;
       
return getKey().equals(((View)obj).getKey());
   
}


   
private static final long serialVersionUID = -1847195052298237356L;

   
private Map<String,Serializable> properties;
   
private String type;
   
private String keyProperty;

   
public View() {
    }
   
   
   
public View(String type, String keyProperty) {
       
super();
       
this.type = type;
       
this.keyProperty = keyProperty;
   
}


   
public String getKeyProperty() {
       
return keyProperty;
   
}

   
public void setKeyProperty(String keyProperty) {
       
this.keyProperty = keyProperty;
   
}

   
public void setType(String type) {
       
this.type = type;
   
}


   
@Override
   
public Serializable getKey() {
       
return get(keyProperty);
   
}

   
@Override
   
public String getKeyField() {
       
return keyProperty;
   
}

   
@Override
   
public String getType() {
       
return type;
   
}

   
@Override
   
public void setKey(Serializable key) {
       
set(keyProperty, key);
   
}


   
@Override
   
public <X> X get(String property) {
       
if (properties != null) return (X)properties.get(property);
       
return null;
   
}


   
@Override
   
public Map<String, Object> getProperties() {
       
if (properties == null) properties = new HashMap<String, Serializable>();
        Map<String,Object> returning =
new HashMap<String, Object>();
       
for (Map.Entry<String, Serializable> entry : properties.entrySet()) {
           
returning.put(entry.getKey(), entry.getValue());
       
}
       
return returning;
   
}


   
@Override
   
public Collection<String> getPropertyNames() {
       
return properties.keySet();
   
}


   
@Override
   
public <X> X remove(String property) {
       
if (properties == null) properties = new HashMap<String, Serializable>();
       
return (X)properties.remove(property);
   
}


   
@Override
   
public <X> X set(String property, X value) {
       
if (properties == null) properties = new HashMap<String, Serializable>();
       
return (X)properties.put(property, (Serializable)value);
   
}

}



where ViewI is a simple extension of the ModelData interface to add some other bits and pieces I needed.



import java.io.Serializable;

import com.extjs.gxt.ui.client.data.ModelData;

public interface ViewI extends ModelData, Serializable {
   
String getType();
    Serializable getKey
();
   
void setKey(Serializable key);
    String getKeyField
();
}



Hopefully it wont be too long before GXT releases an open source version with a workaround for this issue, but in the mean time feel free to use this code/simple approach to work around the issue.

Saturday 26 September 2009

Shout out for OpenCSV

I just wanted to give a shout out to a nice library I stumbled upon when working on my latest Google App Engine for Java project, a site for displaying my running clubs Road Race results for the last few years at http://bsrcroadraceresults.appspot.com.


I have a need to upload a lot of data from old Excel files up into the DataStore. I searched around for an easy way to convert the Excel to a set of annotated POJOs and came across OpenCSV. Turns out to be totally trival to consume a CSV file and use the data to populate the fields of a POJO using standard bean conventions.



Will post a code snippet to show how when back at my dev machine.


Having done the POJO conversion I then used XStream to turn them into an XML file and a little servlet I have been working on to consume the XStream recreating the POJOs on the server and putting them in the DataStore, but more of that later.

Thursday 17 September 2009

Google App Engine (GAE) for Java and XStream

XStream is a powerful and often incredibly useful tool for turning Java classes into XML and back again. If have used it for all sorts of data transformation, configuration loading and saving, persistence needs over the past few years.

In my recent investigations of Google App Engine for Java (GAE/J) I have come across the problem of how to seed an application I am developing with the initial set of data that it needs. At the time of writing GAE doesn't have much in the way of a data loading/management API, especially on Java.

So I thought I would look at using my trusty old friend XStream and file upload to send data up into the application to populate the Datastore.

Unfortunately using XStream with GAE proved to be less than straightforward mainly due to the current security restrictions in the GAE environment. GAE is a sandboxed environment with white listing of a limited set of classes.

As a result there are a couple of issues with using XStream in GAE

  • The standard out of the box XStream uses classes that are not permitted by the sandboxed GAE environment

  • A common usage pattern for XStream is to use it to create an object input or output stream that can be used to read and write serializable objects, but this works by using subclassing of ObjectInputStream and ObjectOutputStream, again this is not allowed by the GAE security model


  • However all is not lost. I found a couple of interesting discussions about the security issues here and here the result of which was a JIRA entry with a number of patches that work around the Serialization security issues.

    I took the gae3-xstream.patch and applied it and tested it and found my problems were half solved. I had hit the second issue I listed above.

    I was trying and failing to create an ObjectInputStream in GAE to unmarshall an XStream representation of a serialized stream of objects.

    The serialized object representation had been created using

    ObjectOutputStream out = xstream.createObjectOutputStream(stringWriter);
    for (WavaBase wavaBase : wavaBases) {
        out.writeObject(wavaBase);
    }


    As a result you get a nicely formed XML document with an object-stream root.


    <object-stream>
        <wavabase>
            <id>F10K</id>
            ...
        <wavabase>
        <wavabase>
            <id>M10K</id>
            ...
        <wavabase>
        ...
    <object-stream>


    But this form of document requires use of an object stream to reverse the process.


    ObjectInputStream in = xstream.createObjectInputStream(inputStream);
    List<WavaBase> wavaBases = new ArrayList<WavaBase>();
    while(true) {
        WavaBase wavaBase = (WavaBase)in.readObject();
        if (wavaBase == null) break;
        wavaBases.add(wavaBase);
    }


    Here is where we hit the issue with XStream on GAE as we are not able to call createObjectInputStream() without getting a security violation.

    But all is not lost. If we adopt a different approach for creating our XML document containing the XStream serialized data we require we can work about the stream security problems.

    The alternative approach is to create a list or set of the data that we want to save and then to use toXml() to form the document, this doesn't require or use the custom input and output streams.

    To create the serialized object representation use

    List<WavaBase> clonedWavaBases = ...;
    ...
    xstream.toXML(wavaBases, stringWriter);


    which gives us a subtly different XML generated


    <list>
        <wavabase>
            <id>F10K</id>
            ...
        <wavabase>
        <wavabase>
            <id>M10K</id>
            ...
        <wavabase>
        ...
    <list>


    To turn the XML back into Java objects in the Google App Engine we call


    List<WavaBase> wavaBases = (List<WavaBase>)xstream.fromXML(inputStream);

    By-passing the issues with object stream handling.

    Using this approach has allowed me to load the data I need into my first GAE application Wava Calculator which is a simple GWT/GXT application for calculating age adjusted times for runners.

    Wednesday 16 September 2009

    Google App Engine Java (GAE) Persistence Gotcha

    Working with GAE/J (Google App Engine for Java) I hit a snag whilst trying to upload some data into my google web app. I was using a set of tricks and tools (more of which later) to reconstruct some data from an XML file description of the data generated with XStream.

    As a result the data being mapped into the application had to some extent to match the objects which had produced it. As a result I have a SortedMap field in one of my data objects.

    @PersistenceCapable(identityType=IdentityType.APPLICATION)
    public class WavaBase implements Comparable <WavaBase> {
        @PrimaryKey
        @Persistent
        private String id;
        @Persistent
        private String sex;
        @Persistent
        private Double standardInSeconds;
        @Persistent
        private String distance;
        @Persistent
        private Double distanceInKilometers;

        private SortedMap<Integer,WavaData> wavaData;
        @Persistent(mappedBy="wavaBase")
        private SortedSet<WavaData> wavaDataSet;
         ...
    }

    The sorted map is not annotated as @Persistent but here is the gotcha. GAE will still attempt to persist it resulting in a most unhelpful warning

    java.lang.UnsupportedOperationException: FK Maps not supported.
        at org.datanucleus.store.appengine.DatastoreManager.newFKMapStore(DatastoreManager.java:401)
        at org.datanucleus.store.mapped.MappedStoreManager.getBackingStoreForMap(MappedStoreManager.java:766)
        at org.datanucleus.store.mapped.MappedStoreManager.getBackingStoreForField(MappedStoreManager.java:637)
        at org.datanucleus.sco.backed.SortedMap.(SortedMap.java:82)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        ...

    Looking at the trace you can see that datanucleus is attemting to deal with the SortedSet even though it is not marked for persistence.

    To ensure that a private member of a @PersistenceCapable class is not processed by the persistence store you need to "explicitly" mark the member as @NotPersistent, this tells the persistence layer to ignore the member.

    So the class becomes

    @PersistenceCapable(identityType=IdentityType.APPLICATION)
    public class WavaBase implements Comparable <WavaBase> {
        @PrimaryKey
        @Persistent
        private String id;
        @Persistent
        private String sex;
        @Persistent
        private Double standardInSeconds;
        @Persistent
        private String distance;
        @Persistent
        private Double distanceInKilometers;

        @NotPersistent
        private SortedMap<Integer,WavaData> wavaData;
        @Persistent(mappedBy="wavaBase")
        private SortedSet<WavaData> wavaDataSet;
         ...
    }

    There may be a way to switch the default handling of members of a @PersistenceCapable class so that members need to be explicitly included in persistence rather than excluded, but I haven't found it yet.

    Daddy can pix it!

    In my house, shared with my lovely wife and two kids, there is a believe amongst the smaller members of the family, that I can fix anything. If something is broken it is delivered sometimes with great ceremony into Daddy's oppice (office) to be pixed (fixed).

    Now I would love to claim myself some DIY guru with special powers to mend or repair, or even better to make or invent.

    Sadly in the "real" world I am a simple fella who can do basic tasks and who has a tube (currently missing in action) of Super Glue that can be used to mend and repair many a tragic accident.

    Ok, so where am I going with this. Outside of my family life I am for want of a better description a software engineer. I have worked developing software at various levels of the software stack for over 20 years now. I believe I am a pretty decent software engineer and have worked for big firms, small firms and in recent years I have run my own company with a couple of friends and ex-colleagues.

    Increasingly as I go through my working life I am amazed at the progress we make in terms of the software and tools that are available to us, and often at the same time appalled by just how difficult is it to get technologies working outside of a simple use case that the developer had in mind when writing it, and worse still just how difficult it can be to make differernt tools, libraries and products work together.

    So I increasing spend time trying to discover and integrate new and existing technologies to produce ways of increasing productivity and making the building of effective solutions that will scale and be resilient.

    The goal of this blog to try and provide a place to bring together the successful (and failed) attempts I have at technology intergration in the hope that it will
    • be useful to others attempting to follow a similar path
    • be a reminder to myself if no one else as to what I did and why and what tips and tricks I learned
    • give me a place to appraise the technologies I find and maybe generate some debate with others about where the technologies might go
    As I write this I find myself wondering if anyone but I will ever read it, and whether it matters. But I can but try and embrace the 21st century.

    In the coming days and weeks I hope to blog on some of the technologies I am currently working with such as
    • The Old - Spring, XStream, Java, Hibernate
    • The New - Google App Engine for Java
    • The Borrowed - GigaSpaces and Caching, GWT, GXT
    • The Blue - Eclipse Tooling
    I hope to provide some software super glue enabling others to quickly link, integrate, fix and reuse some of the great software out there in the world.

    Ok, enough already, on with the show!