Archive for the ‘Techie’ Category

Hints for posting patches to Android

Wednesday, March 11th, 2009

I’ve had a few (very minor) patches accepted by the Android community. Although they’re fairly explicit about what needs doing, here are a few findings which might grease the wheels for others.

  • When uploading using repo, you do not get an opportunity to input any message explaining the patch. Instead, your git commit message will be used, so get it looking nice! (You can use git commit --amend to tweak it).
  • It appears to be undocumented what will happen if you have multiple commits per topic branch. Are they all uploaded? Separately or individually? Which commit message is used? (At least, I can’t find it documented anywhere). You can merge git commits using git rebase -i (amongst other commands). But gerrit appears to have functionality for patches depending on other patches; perhaps this is what happens? But I’m not brave enough to try.
  • Likewise, I can’t find anywhere explaining what happens if you have repo downloaded a patch before uploading a new one. Is that marked as a dependency?
  • gerrit appears to assume that your commit messages are formatted according to an unwritten standard for Git commit messages.
  • I can’t find documented how reviewers for patches get allocated. It seems to be a bit of a black art. I’ve had a few patches sail through, but my very first one – an obvious typo fix in a comment – is still sitting there, presumably awaiting somebody. Who? Who knows.

On the whole, though, the system worked exactly like it said on the tin! I can’t wait to see how easy or hard it is, in practice, to get changes submitted to the Symbian Foundation…

Git versus Mercurial

Tuesday, January 27th, 2009

A few months ago I adopted Mercurial for a project on which I was working. At that time I was faced with a choice between Mercurial or Git, and Mercurial seemed the right choice because it behaved similarly to Subversion, with which nearly everyone is familiar. Git, however, seemed to be developed by-hackers-for-hackers. Although this often yields the best feature set it does, in my view, tend to lead to confusion (and occasionally architectural omissions).

(That said, I was tempted by Git; I’d closely followed the discussion which led to its development on the Linux kernel mailing list years ago, so I felt ahead of the game!)

Only four months later, and it looks to me like I backed the wrong horse. The list of open source projects using Mercurial is impressive – OpenJDK, Mozilla, OpenSolaris, Symbian OS – but they tend to be corporate efforts, and therefore aren’t likely to be trend-setters. All the cool kids seem to be hanging around the Git block – Linux, Perl, X.org, Qt, Ruby on Rails, Wine, Samba, GHC, Scratchbox, VLC and not to mention Android. That sort of momentum of grass-roots-led open source projects is, I reckon, bound to push Git ahead irrespective of the merits of the two.

So what are the merits of the two? From a few hours’ research, most people seem to agree on these facts:

  • Mercurial behaves more like subversion, in terms of command-line syntax. Some see this is as good, some as bad.
  • Git has squillions of commands. Likewise.
  • Git is rubbish on Windows. Subversion has the excellent TortoiseHg. Efforts to produce an equivalent for Git seem to have stopped.
  • Git has no knowledge of file renames. Most people seem to think this is a disadvantage, but the more enlightened opinions note that this is actually an advantage, because Git will automatically notice portions of files that are similar, and can therefore keep track of changes even if a file is split into two. Allegedly. Cool!
  • Git has freakishly good Subversion integration. It appears you can almost ask it to treat a Subversion branch on a server as just another Git branch.
  • The physical structure of Mercurial is possibly more suited to integration with other tools/IDEs/etc., in that there are clear libraries. Tools would need to launch processes to integrate with Git.
  • Both are a bit rubbish with projects composed of many smaller projects, though it sounds like Git’s submodules support is more ‘core’ than the Subversion ‘forest’ extension designed to support this. I may be wrong.

I’ve also heard two other things, but I can’t justify them: one is that Git has a more coherent branching strategy than Mercurial (there’s no reason why you need to create a new repository for a new branch), and the other is that Git is rubbish with UTF-16 text.

Behind the scenes, the concepts are very similar. The only real difference is that Mercurial works on changesets, which can only build on the relevant prior changeset, whilst Git works in terms of entire trees of files (which, behind the scenes, may be stored as deltas).

So, when I eventually switch from Subversion to a distributed system, I’ll probably be switching to Git. I hope there’s a viable path for progress on the Windows version by then.

Calling Eclipse/OSGi Plug-in APIs from outside OSGi – part three

Tuesday, January 27th, 2009

Previously I have tried to explain how it’s possible to call Eclipse APIs from outside Eclipse. It was all a bit painful.

I decided to automate the process as much as possible. I’ve now got a “Run in OSGI” API which enables you to write code like this:


package com.macrobug.testOutside;

import com.macrobug.osgiWrapper.OsgiWrapper;
import com.macrobug.testinside.IResults;
import com.macrobug.testinside.MyAPI;

public class Main {

  public static String resultsHolder;
  
  public static void main(String[] args) {
    try {
      OsgiWrapper.runInOsgi(MyRunnable.class.getName(),"/Users/adriantaylor/Desktop/testInsideExport/eclipse","com.macrobug.testinside");
      System.out.println("Results holder is "+resultsHolder);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
  
  public static class MyRunnable implements Runnable {
    public void run() {
      MyAPI myapi = new MyAPI();
      IResults results = myapi.doFirstApi();
      resultsHolder = results.getResults();
    }
  }
}

Note that MyAPI is an API exposed by a normal Eclipse plug-in, whilst this code is running in a normal Java VM outside of Eclipse.

The code works by creating two new class loaders. Firstly, it fabricates an entire new Eclipse plug-in which depends upon the packages you want. It steals the class loader from that plug-in; this class loader has access to the APIs in those packages. It then creates a new class loader which delegates some requests to that class loader, but also allows access to whatever classes can be loaded via the normal means.

Finally it asks that new class loader to load the runnable, and then runs it.

The code is here if you’re interested.

But I’m not sure I’ll ever use it, or advocate its use to my customers. It’s still just too horrendous. The point of this whole exercise is to allow my customers, who may have arbitrarily complex Java systems involving their own wacky class loaders (e.g. in a Java web serving environment) to embed my code and use my APIs, even if I use OSGi internally. But this solution won’t work for that: it will only work where the existing system is simple.

So I am still not sure what to do. I really, really, want to use OSGi inside my projects but that makes it virtually impossible to embed them into an existing complex Java world. It’s daft.

Parallels bug fixes

Monday, January 12th, 2009

Just a quick note to say that Parallels has fixed both of the two bugs that mean the Symbian emulator wouldn’t run from a shared Mac drive. This first one was a problem with file attributes. The second one is excessive case-sensitivity of the file system. Top marks to Parallels for fixing them! (Fixed in build 3810).

Running OSGI/Eclipse plug-ins from within a normal Java application – Part 2 of 2

Monday, November 24th, 2008

In Part One I explained how to start up a whole Eclipse/OSGi plug-in system from a normal Java application. This worked fine, except there was no way to exchange data between the two.

I’ve finally figured out a way. (It was more difficult than I expected!) So here is Part Two, where I describe how.

First of all, let’s talk about what is not possible.

  • It’s not possible to allow external code (the ‘testExternal’ project from the last article) to access classes within any existing plug-in in the embedded Eclipse. At least, not without using reflection. (See this bug).
  • It’s not possible to allow the plug-ins to load code from JARs and libraries outside of Eclipse. (Exporting the Eclipse product fails).

So what are we left with?

Fortunately, there’s a way of creating a Third Kind of thing. This is called a ‘framework extension bundle’ (I think). This is a type of Eclipse plug-in which is an extension to the OSGi framework. The various classloaders running in the Eclipse world always check with any such extensions to see if a class can be loaded through them. So, the classes in such plug-ins are globally available to all other Eclipse plug-ins. Better still, Eclipse doesn’t attempt to load the classes in this plug-in… it assumes it will have been loaded as part of the process of loading Eclipse itself. That means you can load it using the classpath mechanism, which means it’s accessible to your code outside Eclipse as well as your code inside it.

Specifically, here’s what you need to do.

  1. As before, organise your code into internal code which will run inside Eclipse, and external code which will run outside. But this time, you need a third thing: crossover code which will provide the data structures that need to be accessible from the internal and external code.
  2. Create a new Eclipse Plug-in Project for this crossover code.
  3. Create classes, etc. inside that plug-in.
  4. Now the clever bit: in the MANIFEST.MF for that plug-in, add the following line:
    Fragment-Host: system.bundle; extension:=framework
    That means that the classes in this plug-in are regarded as part of the system – just like java.lang.String and similar. They will be made available to all plug-ins within the whole Eclipse system. Furthermore, they will be loaded using the standard Java class loader and the normal class path, which means they’re also accessible outside Eclipse.
  5. In your ‘internal’ plug-in (which as you’ll remember was an Eclipse IApplication) you can now access these data structures, fill in details, and return them. Note that you do not need to specify your ‘crossover’ plug-in as a dependency of your internal plug-in: it’s always available.
  6. Specifically your IApplication might look like this:
    import com.me.crossover.ResultsHolder;

    public class MyInternal implements IApplication {

      public Object start(IApplicationContxt context) throws Exception {
        IThingy foo = ; // retrieve from other plug-ins etc.
        ResultsHolder rh = new ResultsHolder();
        rh.name = foo.name;
        return rh;
      }

      public void stop() {

      }
    }

  7. In your ‘external’ project, add the ‘crossover’ project to the build path.
  8. In your ‘external’ code, take the value returned from org.eclipse.core.runtime.adaptor.EclipseStarter.run and cast it to the ResultsHolder. You can now access its fields, call its methods etc.

Now, when you run your external system, ensure you add the ‘crossover’ plug-in to the class path. Bob’s your uncle.

Obviously it’s a right pain to have to ‘trans-ship’ information from the data structures used within Eclipse into other classes which you can access externally. But right now, that seems unavoidable.

Running OSGI/Eclipse plug-ins from within a normal Java application – Part 1 of 2

Thursday, November 13th, 2008

Eclipse is a great plug-in system for Java. But superficially, it appears a bit like some allege the GPL to be – a virus! It appears that, if any of your code is an Eclipse plug-in, the whole Java system has to be.

Not true, it turns out! You can embed a whole Eclipse system, with all sorts of plug-ins, in an existing Java interpreter process. But frankly, it’s a fiddle. This note explains how. A subsequent posting will explain how you can exchange information between the Eclipse and non-Eclipse parts of the process.

But first of all, why? If you’ve got some code in Eclipse plug-ins that you wish to run from an existing Java system, your options are either to spawn a new process to run it, or to follow this recipe and run the plug-ins inside your existing Java interpreter. Why would you want to do the latter?

  • No overhead of starting a new process
  • No overhead of recompiling the class file ‘hotspots’ to native code each time
  • No overhead of running the ‘static’ cod to construct static data members
  • Easier exchange of information – freely pass Java classes back and forth

Obviously, there’s not usually any point in spawning a whole Eclipse GUI from some external Java code. But there are many libraries and whole systems written as Eclipse plug-ins, making use of the powerful OSGi plug-in system at its core. Using those is often something that an external Java program would want to do.

Enough of the ‘why’. What about the how?

  1. Work out exactly how you need to interact with your Eclipse system. You’re going to be running some code inside Eclipse, which will need to perform all the interactions with your external code. Your ‘internal’ code has full access to al the Eclipse plug-in APIs. Ideally, the information exchange between the ‘internal’ and ‘external’ code should be minimal, since it’s tricky.
  2. Create a new Eclipse plug-in project. This will house your ‘internal’ code. Let’s assume we’re going to call it ‘testInternal’.
  3. Edit the MANIFEST.MF for testInternal needs to depend upon the Eclipse plug-in(s) whose APIs you wish to use, and also on org.eclipse.core.runtime.
  4. Under Extensions, add an extension to the extension point org.eclipse.core.runtime.applications. You’ll need specify a ‘run’ element pointing at a class which implements IApplication. Go ahead and create that class. Meanwhile, the relevant part of your plugin.xml should look something like:
       <extension
             id="testInternal"
             point="org.eclipse.core.runtime.applications">
          <application
                cardinality="singleton-global"
                thread="main"
                visible="true">
             <run
                   class="testInternal.TestInternal">
             </run>
          </application>
       </extension>
    
  5. In your testInternal class that you just created, fill in the code you want to run into your start method. It’s a good idea, for now, to put some statements that will show output indicating whether this code has run. For example:
      public Object start(IApplicationContext context) throws Exception {
        System.out.println("Got into start");
        String[] args = (String[]) context.getArguments().get(
            "application.args");
        // Interact with Eclipse plug-ins as much as you like
        System.out.println("Exited start");
        return EXIT_OK;
      }
    
  6. Note that in the above code, we’re not really attempting any information exchange with the world outside of Eclipse plug-ins. If you happen to want to exchange Strings, you’re fine: you can read the arguments using the code shown. You can return any type of object at all, but as we’ll see later there are problems related to class loaders, so in practice you will need to return one of the core Eclipse objects such as a String, a Long etc.
  7. In this project, create a new Product Configuration. (File – New – Other – Plugin Development – Product Configuration).
  8. This Product Configuration should include all the plug-ins that you need. That includes your testInternal plug-in, but also all those on which it depends. (The “Select Required” button is helpful here). Make sure you specify your newly created ‘application’ extension as the one to launch.
  9. Export this Product Configuration. File – Export – Eclipse Product – Directory.
  10. You’ve now got a standalone Eclipse installation which will run your ‘application’ code and call into the plug-ins it needs to call into. It’s worth testing this. Simply run the Eclipse executable that’s been created, and you should see the results of the print statements in your code.
  11. Now we need to attack the ‘external’ code and get it to run the ‘internal’ code from within the same java process. But first, an interlude: here’s how the normal Eclipse launch process works. It is relevant, trust me!
    • The eclipse.exe launcher is complex, but its net effects when launching Eclipse amounts to approximately: java -jar org.eclipse.osgi.XYZ.jar -configuration configuration/ .... (Here, XYZ is just a version number).
    • The java process looks for, and finds, a ‘main’ function (as specified in the manifest inside that jar) within org.eclipse.core.runtime.adaptor.EclipseStarter.
    • The ‘main’ function initialises OSGi and Eclipse, loads plug-ins according to the configuration file at configuration/config.ini and then passes control to the start function of the relevant IApplication.
  12. We need to do the same in our code. Fortunately, org.eclipse.core.runtime.adaptor.EclipseStarter has some static methods we can call, to mimic the action of its main. Here’s the code you need.
      private static final String INSTALL_DIR = "...";
    
      public static void launchTestInternalFromOutside() {
        System.setProperty("eclipse.application", "testInternal.testInternal");
        System.setProperty("osgi.configuration.area", INSTALL_DIR+"/configuration");
        System.setProperty("osgi.install.area","file://"+INSTALL_DIR);
        System.setProperty("osgi.framework","file://"+INSTALL_DIR+
          "/plugins/org.eclipse.osgi_3.4.2.R34x_v20080826-1230.jar");
        Object o;
        try {
          System.out.println("About to start Eclipse");
          o = org.eclipse.core.runtime.adaptor.EclipseStarter.run(new String[]{"myParam"},null);
          if (o != null) {
            System.out.println(o.toString());
          } else
            System.out.println("o was null");
        } catch (Exception e) {
          e.printStackTrace();
        }
      }
    
    
  13. Obviously, you also need to add the org.eclipse.osgi.XYZ.jar to the classpath of this code.
  14. INSTALL_DIR is the directory of the Eclipse that you exported.
  15. The various properties are the parameters that Eclipse and OSGi needs to get started. (I couldn’t find these documented anywhere; a bit of debugging revealed what they needed to be, but you may have less luck – fiddle with them).
  16. And that’s it. Run this code, and you should see the whole Eclipse framework started, plug-ins loaded, and your ‘testInternal’ code run all from the same Java process.

So what are the caveats?

  • Data exchange between your ‘internal’ and ‘external’ code is tricky. The reason is that Eclipse plug-ins use their own class loaders (as explained in this excellent article). So, if your internal code returns a com.frangible.Sprocket, and your external code tries to cast that to a com.frangible.Sprocket, you’ll get a ClassCastException. Huh? That’s because there are two separately loaded versions of the Sprocket class, loaded by different class loaders. As far as Java is concerned, they’re separate classes. I haven’t exactly worked out the best way round this, but Part Two of this article will reveal my findings when I do. If I do.
  • That org.eclipse.core.runtime.adaptor.EclipseStarter.run method is slow. In all fairness, it has to load a lot of code. There are other static methods in the same class which should, hopefully, allow you to ‘start’ Eclipse once but then ‘run’ it many times. Presumably, this means all the heavyweight plug-in loading will occur once, and subsequent times you’ll be able simple to run your testInternal code without all that overhead. Worth looking into.
  • Something strange is going on upon exit. Control is always returned to my launchTestIntrnalFromOutside method, but the process doesn’t necessarily actually exit. Perhaps Eclipse has some sort of shutdown hook. I have yet to look into this.

So, this is a work in progress but it’s a start. Good luck!

Distributed version control systems

Friday, October 31st, 2008

One of the projects I’m working on right now involves two developers in different continents.

For various reasons it would be awkward to set up a proper source code repository on a mutually accessible server, so instead I’m experimenting with a distributed version control system: Mercurial. For those who haven’t come across these before, the idea is that each developer has their own complete repository. Into this repository they check in their own changes. They can also ‘pull’ changes from anyone else’s repository.

There are no restrictions on where you can pull changes from, nor the order in which you take them (apart from the inevitable merging pain if you do something illogical).

So far, it’s looking really good! What’s really surprised me is how similar the command-line syntax is to Subversion. If you’re familiar with Subversion, you might want to try playing with Mercurial, as I think distributed systems are probably going to be the way of the future…

PS it will be interesting to see what the Symbian Foundation does for this sort of thing.

How to parallelise Symbian programs without changing the code

Thursday, October 23rd, 2008

The Smartphone Show this year seemed to have a lot of emphasis on different types of ‘runtime’ which can be used on Symbian phones. In this instance a ‘runtime’ seems to mean any programming environment: be it native Symbian C++, Java, Flash, Python or even AJAX stuff in the browser.

There was also a lot of emphasis on multi-processing, since it seems pretty clear that SMP is going to be the best way to save power on phones in the future. (Two slow processors use less power than one fast one).

Symbian OS provides all the usual techniques for multi-processing: threads, semaphores, and mutexes. Yet we all know these make programming difficult. So here are some miscellaneous thoughts on other things which could be done.

A declarative run-time

I tend to think that, as we get to a world with dozens or hundreds of cores on the desktop, we programmers are going to have to give up on issuing specific sequences of instructions, and instead just describe the desired relationships between our variables. The runtime then figures out how to calculate this based on the available hardware resources.

Such a scheme requires these relationships to be described in a functional or declarative programming language like Haskell. Maybe we need a Haskell runtime for Symbian OS?

I can imagine that working quite well to calculate the screen image of a game, for example. But interfacing with the rest of the system would probably need to be done using traditional programming techniques.

Software transactional memory

STM is an alternative to locks. In short, you just go ahead and make your writes to memory, and if they conflict with somebody else’s writes, the system will roll them back.

Ideally we need a paging system which somehow makes copies of pages automatically for each thread, and only merges them together upon a commit.

Parallelising active objects

Most Symbian OS programmes are specified as a series of active objects, which define how the thread responds to certain incoming events. A parallelised active scheduler could simply distribute the active objects across multiple threads in order that they can run in parallel on multiple cores. But, there would need to be restrictions: it’s always been assumed that active objects run one-at-a-time, and hence there’s no need for locking of data structures. Either all active objects would need to use locks whenever accessing potentially shared data structures, or each object would need to be annotated to give rules about which ones the scheduler could run simultaneously.

Still, the fact that Symbian programmes do have this semi-declarative way of saying what code to run in response to which stimulus does give them an advantage over other operating systems where event loops can be structured in myriad ways. It does open the door to thinking about parallelising programs automatically.

In practice, I think it’s probably hopeless. Except…

Paralellising active objects + software transactional memory

If you imagine each active object as a single transaction in STM, then each one can run independently, operating on its own copy of the data. When the active object has completed, if no other active object has operated on the same data, then its changes are committed. Otherwise, they’re reverted and the active object is run again.

Then, the restrictions needed get much less: just making sure each active object doesn’t do any input/output operations (otherwise it can’t be reverted). (Non-blocking operations can simply be postponed until the transaction is committed).

So, all we need to do, is find a way to annotate the behaviour of active objects like that; implement a paging scheme and APIs for STM, and write an active scheduler which can distribute active objects across threads… and suddenly we have a way of parallelising all existing Symbian programmes.

Ha ha ha. Of course there are millions of complications. But it’s interesting to muse.

Results of Stray Event Scanner evaluation at a customer site

Tuesday, August 19th, 2008

As I mentioned, about a week ago I visited a customer site to find out if our Stray Event Scanner tool actually did the trick, on their codebase. That customer has kindly said I can post some results here.

Results were mixed!

Proportion of this customer’s codebase scanned 5% (estimate).
False positives reported by the tool 47
Of which are things I might be able to fix 37
are intrinsic limitations 8
are already fixed in a later development version of the tool 2
Interestingly, 35 of these false-positives were in one smallish area of code. It seems that some Symbian OS code conforms to the norms expected by the tool, whilst some code really gets is unnecessarily overexcited. All the rest of the code we scanned only produced 12 false-positives.
Real bugs found by the tool 14
Of which tool was precise about the problem 6
tool was wrong, or vague, about the specific problem but did identify the right bit of faulty code 8
Actual bugs filed in this customer’s defect-tracking system 10 (because the other four were duplicates)

Conclusions. Good things first…

  • As expected, the tool found plenty of bugs. Nearly all of these bugs were real problems which could cause E32USER-CBase 46 panics under some circumstances, which are awful to diagnose.
  • In fact one of them was a duplicate of an existing E32USER-CBase 46 bug, which they’d been searching for for some time, but had been unable to find. Result!
  • The tool broadly worked as planned. There were no hopeless incompatibilities with their source code.
  • With the exception of that one component, there were far fewer false-positives than I realised.

Bad things…

  • There were lots of false-positives in that one component. Many of these will be impossible to remove using the current analysis method.
  • In some cases the tool was not very accurate about specifying exactly what was wrong with the code. It would identify the right area of code, but a lot of examination and thought was required to identify whether there really was a problem there. Again, there’s not much that can be done about this: stray event problems are intrinsically complicated and somebody technically experienced will be required to analyse the results.

Next steps…

  • See how many of those false-positives I can remove easily
  • Fix a few other minor bugs that cropped up
  • Decide whether the tool is ready to leave the beta stage!

DTrace is very good

Monday, August 18th, 2008

One of the things which would make Macrobug’s life much easier is if Symbian OS had DTrace or an equivalent. DTrace is a dynamic tracing framework which allows arbitrary APIs and function calls to be instrumented at run-time without any changes to any code. That way, the Macrobug tools would be able to hook in and monitor what’s happening inside the device without any changes to the kernel or application binaries.

Such days are probably some way away, but I can look forward to it!

Meanwhile, just as an example of how great DTrace is on a real system, I’ve just used it to diagnose a problem with my Mac. Macs include a backup mechanism known as Time Machine. This does incremental backups, in my case onto a RAIDed NAS server sitting elsewhere in my house (which in turn has parts of it rsync’ed to an off-site backup regularly).

Time Machine backs up automatically every hour in the background, and keeps as many incremental backups as it can fit onto the backup disk. There’s therefore an incentive for the backups to be smallish, as then your history will go back further. I found that most Macs seem to back up about 1-3 MB every hour, whereas mine always backed up 105-110MB. What was this mysterious file which was changing every hour that seemed to be 105MB big?

To diagnose this, I used Apple’s whizzy DTrace front-end, known as Instruments, to attach to the “backupd” process which does the time machine backups. I used some preset DTrace probes which can instrument the APIs used to read and write files, thus giving me lists of all the reads and writes, including their sizes, filenames, and even call stacks. And there’s even a nice UI to look at the results:

Instruments UI

The mysterious 105MB file turned out to be the article database for my RSS reader. It appears that they store all the articles in one file (messages.db) instead of many small ones. So, each time it refreshes (more than once per hour) the file changes, and Time Machine chooses to back it up.

So, the solution is simple – check whether there’s a fixed version of Vienna, or move to a new RSS reader, or tell Time Machine to ignore this file.

As for Symbian, perhaps there will be something like DTrace one day! I can’t wait!