Scala on Android 101 - Proguard, XmlParser and Function2AsyncTask 12

I’ve neglected my blog for a while so I though I’d share my latest experiments with running Scala on Android. In order to try a couple of things with Scala on Android I’m creating an Activity which  basically trying to consume a WebService and display the results in a List.Please keep in mind that I’m only learning Scala now so I’ll gladly get any comments and/or suggestions regarding my solutions.These are the problems I’ve found  :

  • PROBLEM #1
    • The scala-library.jar is too big to dex so you have to use ProGuard to keep things small. Try making it work properly with maven and you’ll have issues with getting all the plugins working in the right order!
  • SOLUTION #1
    • Ditch maven! “But having it manage my dependencies is so cool…” I hear you say.. well let’s use Ivy:

Just use one of the numerous ANT build scripts for Android and add the targets for Ivy and ProGuard and the scala repository to your ivysettings.xml:

.....
<url name="scala-tools.org">
     <artifact pattern="http://scala-tools.org/repo-releases/[organisation]/[module]/[revision]/[module]-[revision].[ext]" />
  </url>
....

add your dependencies to your ivy.xml:…

<dependency org="org/scala-lang" name="scala-library" rev="2.8.0.r18462-b20090811081019"/>
<dependency org="org/scala-lang" name="scala-compiler" rev="2.8.0.r18462-b20090811081019"/>
<dependency org="net/sf/proguard" name="proguard" rev="4.3"/>
....

Add some nice Ivy targets:

<taskdef resource="org/apache/ivy/ant/antlib.xml"
     uri="antlib:org.apache.ivy.ant" classpath="${ivy.jar.dir}/ivy.jar"/>

 <target name="init-ivy" depends="download-ivy">
 	<ivy:settings file=”${basedir}/ivysettings.xml” />
        <ivy:retrieve />
</target>

  <target name="download-ivy">
    <mkdir dir="${ivy.jar.dir}"/>
    <get src="http://www.integratebutton.com/repo/
       ${ivy.install.version}/ivy-2.0.0-beta2.jar"
      dest="${ivy.jar.file}" usetimestamp="true"/>
  </target>

a nice ProGuard target:

    <target name="proguard" depends="compile">
      <taskdef resource="proguard/ant/task.properties"
               classpath="${external-libs}/proguard-4.3.jar" />
      <proguard>
-injars ${outdir}/classes:${external-libs}/scala-library-2.8.0.r18462-b20090811081019.jar(!META-INF/MANIFEST.MF,!library.properties)
-outjars ${outdir}/classes.min.jar
-libraryjars ${android-jar}
-dontwarn
-dontoptimize
-dontobfuscate
-dontskipnonpubliclibraryclasses
-dontskipnonpubliclibraryclassmembers
-allowaccessmodification
-keep public class <your main class here>
-keep interface scala.ScalaObject
      </proguard>
    </target>

and you’re good to go!

  • PROBLEM #2
    • Scala uses the Java SAX parser which kept complaining about the use of namespace prefixes so I had to come up with a way to toggle this feature.
  • SOLUTION #2
    • Use my own SAX parser instance with the right features toggled :
object XmlParser {
  // Workaround for namespace prefix
  private val namespacePrefixes = "http://xml.org/sax/features/namespace-prefixes"
  val parser = javax.xml.parsers.SAXParserFactory.newInstance()
  parser.setNamespaceAware(false)
  parser.setFeature(namespacePrefixes, true)

  def load(i:InputStream) = XML.withSAXParser(parser.newSAXParser()).load(i)
}
  •  PROBLEM #3 :
    • AyncTask requires you to override an abstract method with varargs which is a no no right now in scala Ticket #1459
  • SOLUTION #3
    • Create a java class which extends AsyncTask, I called it MyAsyncTask:
import android.os.AsyncTask;
public abstract class MyAsyncTask<T1,T2,T3> extends AsyncTask<T1,T2,T2>{
	protected T2 doInBackground(T1 …f) {
		return doInBackground(f[0]);
	}
	abstract protected T2 doInBackground(T1 f);
}

As you can see now doInBackground gets a single argument… which in my current implementation is a Function =)I’ve created an AsyncTask class in scala which extends MyAsyncTask and gets a Function as it’s constructor arg :

class AsyncTask(f:()=>Unit) {
    def doInBackground {
        new _root_.pt.inevo.android.meo.MyAsyncTask[Function0[Unit],Void,Void]  {
           override protected def doInBackground(f: () => Unit):Void = {
               f()
               return null;
           }
          def onProgressUpdate(progress:Int*) { }
         override def  onPostExecute(result:Void) { }
   }.execute(f); 	} }

I also added an implicit to convert from Function into my new AsyncTask class and wrapped it in an object:

object AsyncTask {
     implicit def function2AsyncTask(f: ()=>Unit):AsyncTask=new AsyncTask(f)
}

With this I can simply have regular functions and just call doInBackground:

 val getChannelList = () => {
      for( c <- EPG.getChannelList \\ "Channel" ) {
         Log.v("getChannelList", c \ "Name" text )
}
override def onCreate(savedInstanceState: Bundle) {
      super.onCreate(savedInstanceState)
      setContentView(R.layout.main)
      getChannelList.doInBackground
}

I hope this helps some of you get started with Scala on Android. I’ll be working on the layout stuff trying to get ListView and the Adapters to work properly.P.S: I’m using the great DataBinder Dispatch library for Scala to call the service through REST so my Service object is something like:

 object EPG {
    val req = :/("services.sapo.pt") / "EPG"
    def getChannelList=Http(req / "GetChannelList" >> as_xml)
}

where the as_xml function is defined in my XmlParser class as:

def as_xml(res:InputStream) = XmlParser.load(res);

Reuse your Flex SDK with Sprouts 3

Well, I just started taking a look at Sprouts to manage my Flex builds by I didn’t want to download the Flex SDK again!
As soon as I saw that Sprouts was trying to download the flex_sdk_3.zip file I just Ctrl-C and did the following:

  1. touch ~/.sprouts/cache/sprout-flex3sdk-tool-3.3.0/flex_sdk_3.zip
  2. ln -s /opt/Adobe Flex/flex_sdk_3 ~/.sprouts/cache/sprout-flex3sdk-tool-3.3.0/archive

This just fools Sprouts which is advised against in the google groups discussions but saves my some time :P

I did a similar process for the Flash Player 10 which I had downloaded but you could also use the one that came with the Flex SDK. Basically you need to :

  1. touch  ~/.sprouts/cache/0.7/sprout-flashplayer-tool-10.22.1/flash_player_10_linux_dev.tar.gz
  2. mkdir ~/.sprouts/cache/0.7/sprout-flashplayer-tool-10.22.1/archive
  3. ln -s <wherever>/flash_player_10_linux_dev ~/.sprouts/cache/0.7/sprout-flashplayer-tool-10.22.1/archive/flash_player_10_linux_dev

This is a simple hack but might be useful to some of you.

Create RoR model diagrams with yUMLmeRails 118

I just hacked together a simple plugin for RoR apps in order to show a simple model diagram.

If been a fan of RailRoad since the beginning, it generates simple dot and neato diagrams from your rails application.

Now I just found out about the cool http://yuml.me beta and decided to use RailRoad to create yUML diagrams.

I created a GitHub project so you can look at the code and make whatever changes you want. The project page has more info on the plugin.

I was also thinking of adding a Shoes app to create the diagrams but for now I just use it to display the diagram.

Please keep in mind that the source’s all messy and is far from finished,  perhaps someone else would like to make it a whole lot better ;)

Rails + PostgreSQL accent insensitive search 96

For a project we’re currently working on the need for accent insensitive search came up.

I google for a couple of solutions, most fo which required applying a translate() to the records or using to_ascii()  in order to replace accented characters with plain ones. For me this should be a considerable performance hit as the DB had to apply the function to every record before performing the query.

I decided to go with a case insensitive using the operator ‘~*’. Basically, for each character which has accented variations I introduce an expression (ex: a -> (a|à|á|â|ã) resulting in a query like :
select * from my_table where name ~* '(a|à|á|â|ã)ndr(e|é|è|ê)'

This will search for the name “André” in the database.

Since I’m using ruby I created a new method in the String class called accent_insensitive_regexp:

class String

@@ACCENT_INSENSITIVE_REGEXES=[
“(a|á|à|â|ã)”,”(e|é|è|ê)”,”(i|í|ì)”,”(o|ó|ò|ô|õ)”,”(u|ú|ù)”,”(c|ç)”
]

def accent_insensitive_regexp
res=self
@@ACCENT_INSENSITIVE_REGEXES.each {|exp|
res.gsub! Regexp.new(exp), exp
}
res
end

end

Now i can my case and accent insensitive search working by using something like:

User.find :all, :conditions=>[" name ~* '?' ",name.downcase.accent_insensitive_regexp]

Those who’ve tried to use the downcase and upcase methods in Rails will notice that this will not work properly since these methods to not deal with the accented characters.
For dealing with that issue I changed those two methods in the String class to:

alias_method :old_upcase, :upcase
alias_method :old_downcase, :downcase
@@DOWNCASE_ACCENT_CHARS=”çàáèéìíòóùúâêôãõ”
@@UPCASE_ACCENT_CHARS=”ÇÀÁÈÉÌÍÒÓÙÚÂÊÔÃÕ”

def upcase
self.old_upcase.tr(@@DOWNCASE_ACCENT_CHARS,@@UPCASE_ACCENT_CHARS)
end

def downcase
self.old_downcase.tr(@@UPCASE_ACCENT_CHARS,@@DOWNCASE_ACCENT_CHARS)
end

And there you have it. I quick and dirty hack to get this stuff to work properly in RoR. Please notice I haven’t spent much time cleaning the code I just though this could get someone with similar problems started.

JavaFX + Maven on Ubuntu 20

Well, I’ve been following JavaFx for a long time but I’ve only started trying out yesterday.

I need to create some GUI for testing some libraries I’ve been working on and I felt this was the right time to try out JavaFX.

The first challenge was to get the JavaFX Preview SDK working on my Ubuntu machine.

After googling around I found that the MacOSX version of the SDK works on Linux so I downloaded it from here.

Then I got the Netbeans JavaFX Plugin from the JavaFX_NB_Daily project. After installing the nbms I decided to switch the SDK provided by the plugin with the one I downloaded from Sun’s JavaFX page. For that I went into my netbeans directory “~/netbeans6-1″ onto the “javafx” directory, renamed the sdk dir and unziped sun’s sdk. This step is probably not necessary but I’d like to make sure I was working with Sun’s JavaFX SDK.

Well, with everything setup I started netbeans, create a project with the sample weather app and it ran just fine :)

The next step was to get maven to compile a JavaFX project. I normally have several dependencies for each new project and I’ve really come to appreciate Maven for dealing with these for me.

Again I googled for it and found this: http://m2-javafxc.sourceforge.net/. Well it did what I needed, just had to figure out how to use my javafxc compiler. This is what I came up with in the end:

<?xml version=”1.0″ encoding=”UTF-8″?>
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>pt.inevo.javafx</groupId>
<artifactId>javaFXMaven</artifactId>
<name>
javaFXMaven</name>
<version>1.0</version>
<url>http://www.inevo.pt</url>
<properties>
<javafx.home>/home/nfgs/netbeans-6.1/javafx/javafx-sdk1.0pre1</javafx.home>
<netbeans.hint.useExternalMaven>true</netbeans.hint.useExternalMaven>
</properties>
<dependencies>

<dependency>
<groupId>javafx</groupId>
<artifactId>javafxrt</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>system</scope>
<systemPath>${javafx.home}/lib/javafxrt.jar</systemPath>
</dependency>
<dependency>
<groupId>javafx</groupId>
<artifactId>javafx-swing</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>system</scope>
<systemPath>${javafx.home}/lib/javafx-swing.jar</systemPath>
</dependency>
<dependency>
<groupId>javafx</groupId>
<artifactId>Scenario</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>system</scope>
<systemPath>${javafx.home}/lib/Scenario.jar</systemPath>
</dependency>
<dependency>
<groupId>javafx</groupId>
<artifactId>javafxgui</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>system</scope>
<systemPath>${javafx.home}/lib/javafxgui.jar</systemPath>
</dependency>
</dependencies>
<pluginRepositories>
<pluginRepository>
<id>m2-javafxc</id>
<name>Sourceforge M2-javafxc static repo</name>
<url>http://m2-javafxc.sourceforge.net/m2repo</url>
</pluginRepository>
</pluginRepositories>
<build>
<sourceDirectory>src/main/javafx</sourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
<compilerId>javafxc</compilerId>
<include>**/*.fx</include>
<compilerArguments>
<jfxHome>false</jfxHome>
</compilerArguments>
</configuration>
<dependencies>
<dependency>
<groupId>net.sf.m2javafxc</groupId>
<artifactId>plexus-compiler-javafxc</artifactId>
<version>0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>javafx</groupId>
<artifactId>javafxc</artifactId>
<version>0.1-SNAPSHOT</version>
<scope>system</scope>
<systemPath>${javafx.home}/lib/javafxc.jar</systemPath>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>fxsketch.Main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
</build>
</project>

Now I can store my code in src/main/javafx and just open my project with netbeans (since it reads the pom.xml file).
I’m using the external maven since I’ve had a few problems with the internal version.

I hope this helps some of you. I’m really starting to appreciate JavaFx and can’t wait to get the Media SDK and the SG3D to really come up with something interesting ;)

Dealing with UTF-8 with appengine’s bulk loading 27

We just uploaded our first app to Google Appengine’s servers.

It’s called YouDIG - You Draw I Guess - and we believe it to be a fun game to play online.

Being a word based game we had to have an easy way to upload new Riddles to the online app. So, I created a very simple bulkload.py which just calls the ImportCSV function from the SDK. The problem was that it doesn’t work with UTF-8 files!

Here’s what I did (I’m still a Python newbie, so feel free to send me your comments/suggestions):

1 - Edited the google/appengine/tools/bulkload_client.py

def ContentGenerator
....
  if rows_written > 0:
    yield rows_written, unicode(content.getvalue(),'utf-8')

  def PostEntities
  ....
  body = urllib.urlencode({
    constants.KIND_PARAM: kind,
    constants.CSV_PARAM: content.encode("utf-8"),
    })

this basically unicodes everything and encodes it as UTF-8 before sending the POST request.

2 - Created my Loaders (similar to the ones described in the docs):

class RiddleLoader(bulkload.Loader):

  def HandleEntity(self, entity):
    entity['approved']=True
    return entity

  def __init__(self):
    bulkload.Loader.__init__(self, 'Riddle',
      [('word', Riddle.lowerCase),('level', str),
       ('language', Language.get_key_by_code ),
       ('category', Category.get_key_by_sys_name)])

if __name__ == '__main__':
  mybulkload.main(RiddleLoader())

This is a simple loader that extends the bulkload.Loader. The “Language.get_key_by_code” and “Category.get_key_by_sys_name” are static functions that allow me to get the entity based on a string. This way I can import Languages, Categories and Riddles and have the relations set using simple string keys (since I don’t know and entity’s key before it’s saved!).

The main difference from the standard bulkloading is in the main method. The “mybulkload” which is a class extending “BulkLoad” and allows to receive UTF-8 CSV POST data.

3 - The mybulkload package:

def utf_8_encoder(unicode_csv_data):
  for line in unicode_csv_data:
    yield line.encode('utf-8')

class MyBulkLoad(BulkLoad):
""" A handler for bulk load requests.
"""

  def Load(self, kind, data):
    Validate(kind, basestring)
    Validate(data, basestring)
    output = []

    try:
      loader = Loader.RegisteredLoaders()[kind]
    except KeyError:
      output.append('Error: no Loader defined for kind %s.' % kind)
      return (httplib.BAD_REQUEST, ''.join(output))

    buffer = StringIO.StringIO(data)
    reader = csv.reader(utf_8_encoder(buffer), skipinitialspace=True)

    entities = []

    line_num = 1

    for row in reader:
      try:
       output.append('\nLoading from line %d...' % line_num)
       entities.extend(loader.CreateEntity([unicode(cell,'utf-8') for cell in row]))
       output.append('done.')
      except:
       exc_info = sys.exc_info()
       stacktrace = traceback.format_exception(*exc_info)
       output.append('error:\n%s' % stacktrace)
       return (httplib.BAD_REQUEST, ''.join(output))

      line_num += 1

     for entity in entities:
      datastore.Put(entity)

     return (httplib.OK, ''.join(output))

def main(*loaders):
"""Starts bulk upload.
Raises TypeError if not, at least one Loader instance is given.
Args:
loaders: One or more Loader instance.
"""
  if not loaders:
   raise TypeError('Expected at least one argument.')

  for loader in loaders:
    if not isinstance(loader, Loader):
     raise TypeError('Expected a Loader instance; received %r' % loader)

  application = webapp.WSGIApplication([('.*', MyBulkLoad)])
  wsgiref.handlers.CGIHandler().run(application)

if __name__ == '__main__':
  main()

i just copied this stuff from the __init.py__ in google/appengine/ext/bulkloa, added the utf8_encoder function  and extended the Bulkload class overloading the Load method.

Here what I used:

reader = csv.reader(utf_8_encoder(buffer), skipinitialspace=True)

to encode the stuff send to the CSV reader and:

entities.extend(loader.CreateEntity([unicode(cell,'utf-8') for cell in row]))

to unicode everything before creating the entities.

Perhaps there was an easier way but this is working for me so I hope this can help some of you.

Next I’ll write an entity eraser to bulk delete entities from the AppEngine’s production servers…

Flex FlickrFlow 3

I needed a simple Flickr gallery viewer to show user’s photos but I knew I’d seen a couple of cool Papervision based viewers so I decided to search for one.

I found FlickrFlow and downloaded the source code and started hacking my way through it.

First I made the background white and reduced the reflection size, this just required me to fill the bitmap white and create a smaller gradient box for the reflection.

Then I decided it made no sense to use the full size image to show only a smaller one, so I switched to the AS3 Flick API and used the GetSizes API call to get the URL for the “Small” sized image.

This worked and now it is a lot faster than it used to be.

Still, FlickrShow would first fetch all the images and only then start loading then into CoverFlow, so I changed this as well….

Finally, I chose to always select the middle image instead of having a fixed index for selection.

There are still a couple of bugs to fix, mainly regarding mouse click events which sometimes don’t work unless you first use your mouse wheel or keyboard… in the meantime here is a small sample which searches for images with the tag “Madeira Island”.

(The white background doesn’t look good here, I should make it transparent but I have to fix the gradiend before…)

Apache proxy for crossdomain problems 25

Well, the Flash player requires a crossdomain.xml file to reside on every server you call for remote services.

We needed to display RSS feeds for user’s blogs and we couldn’t enforce every single blog provider to have a crossdomain file. The solution was to use a local proxy to perform requests on our behalf.

Common solutions use a server side script to fetch the contents but I didn’t want to have yet another PHP, JAVA, Ruby, etc script laying around so I used Apache’s mod_rewrite with a very simple rule:

RewriteEngine On
RewriteRule ^/my_rss_proxy/(.*) $1 [P]

Since we were already using Apache’s proxies to forward request to my HAProxy I just added this simple rule and now we can use:

http://my_server_url/my_rss_proxy/http://rss_url

RIA Wars 1

The growing bandwith has allowed developers to provide rich Web based user interfaces and thus bring together Web and Dektop applications.

You have Javascript with several new libraries sprouting almost everyday, this allows us to bring live to otherwise static HTML pages through manipulation of the DOM and using AJAX calls to get data from the server.

Nonetheless javascript depends on the browser’s implementation and your source code is there for anyone to see. AJAX calls are pretty cool but XML is a major overhead with large messages and slow parsing.

Adobe has a huge browser penetration, close to 98% if i’m not mistaken. Most people dislike Flash and think it is only used by designers to create sexy and annoying banners. The truth is that Flash uses Mozilla’s Tamarin VM for running ActionScript. Both Javascript and ActionScript are ECMAScript but with ActionScript you have the advantage that it get’s compiled and your source code is not there for anyone to see (unless they disassemble it). Besides, to reproduce most of the functionality you get with Flash you need a lot of javascript libraries, especially if we’re using Flex. Flex provides us a growing number of prebuild components and the UI is specified using MXML with offers a clear separation of controller and view. What’s more upcoming versions of Flash will enable browsers to cache commonly used flash modules (like the Flex framework) so, you only download the library the first time you visit a site using Flex.

The AMF protocol, used by Flash is now Open and there is a reference implementation freely available. This binary protocol is a lot faster than using XML or JSON and it supports pushing information thus we don’t need to have clients constantly pooling our server.

There are some more contenders in this RIA war, JavaFX and Silverlight.

JavaFX suffers, for now, from the same problems every JAVA applet has. It does not fit well in a page, it takes too long to load and there are still many people without JAVA. When JAVA offers an easier instalation and preloads in the OS’s memory then we could have a serious alternative but for now.. well it is just not up to the challenge.

Silverlight has some cool ideas, it allows us to program in a myriad of languages since they share a common runtime, has hardware acceleration and, since it is from Microsoft will come preinstalled in Windows machines. Still, there is no real showcase (Microsoft is using it to build a part of their site now in beta), there have been a couple of releases which were incompatible with each other (you had to keep reinstalling the darn thing), and to get a development environment up and ready it took ages just to figure out the right one!

I really like the idea of having Desktop like applications on the Web but i’d also like to have them in my desktop. Most of these allows developing both for the Web and for the Desktop but Flash and Java might just have an edge on this one. Adobe AIR allows you to develop some pretty cool desktop apps and JAVA has been improving Swing with LaF.

To sum up, I like the fact that Javascript has been evolving at a fast pace but I still feel stranded by browser dependencies, I don’t like to have the source code exposed and I just think that Adobe’s offer is a better deal. ActionScript is a pretty decent language and the Flex framework offers us a lot of cool components. Add to that the AMF protocol with the Open Source Data Services and you have a very good solution for RIA development. What’s more you have a couple of 3D libraries,Thermo’s coming up ( pixel shaders ;) ) and we’ll get hardware acceleration!

Mobile Platforms thoughts for 2008 6

Nowadays tech junkies have a real problem.. there are just too many cool gadgets to carry around.

The convergence has started and I personally believe that the cell phone will become the “do it all” device of the future. We already have cell phones with decent cameras, PIM software and GPS. Add to this a good storage solution and  you got everything you’ll need on a daily basis.

The problem so far has been the platform. You’ve got Windows Mobile which has been around for a while, still it seems sluggish and is missing the “cool” factor many of us geeks miss… Nokia has the Maemo platform for the N800 which is pretty cool but there is no GSM support. Nokia also has the Symbian OS which allows us to develop in C and Python but it is not compatible with most other manufacturers.  Java has been around for most mobile platforms and there are some pretty decent apps and games but you get the feeling that JAVA is not build right into the platform.. some phones even have a separate JAVA area for you to start your JAVA applications.

Apple has released the sexy iPhone and it is really appealling. Still, the SDK took long to come out and the platform remains too closed for any serious investment in it. Besides that, here in Europe we need 3,5G to have decent Internet access and the iPhone does not provide that.

Then, Google came up with Android. It’s impressive… Google can generate quite a hype around a mobile platform for which there is no hardware yet! This seems to be the way to go.. this way when it arrives at the market there should be lots of cool free software around the web for it, especially with money being offered to the best apps!

Nonetheless Apple’s iPhone still has the sex appeal that most phones only dream of having….

Adobe has been focusing on AIR, Thermo, etc.. they have come up with great technologies for Web and Desktop but delayed the Adobe Flash Lite. Personally I’m not very interested in a stripped down AS2 only Flash version… this could be because they’re hopping the hardware will evolve to support the whole Flash runtime… The actionscript engine uses the TamarinVM so it should be ported to most of mainstreammobile platforms soon.

Well, this post is becoming too long, I just though i’d write here my 2 cents. I’ll be keeping and eye on Google’s Android and the iPhone.. wouldn’t it be a beautiful mariage?

Next Page »