macrobug

Benno explains how to build simple native programs for Android. But any significant pre-existing Linux software package uses a complex array of libraries and a correspondingly complicated build process, typically based around GNU Autoconf. That build process usually involves a “configure” script whose purpose is to run tiny programs to experiment with the system, to find out its capabilities and set up the code appropriately.

Terrific, except it doesn’t work if, say, you’re building on an Intel x86 box, but trying to build for Android, which is based around Linux and an entirely different CPU. Fortunately, the Scratchbox project exists to solve this problem. It provides you with a special Linux universe, which uses the QEMU emulator to run ARM binaries. Hence: autoconf can run its experiments and all will be well.

It worked surprisingly well and surprisingly simply. Good work by the Scratchbox team.

Maybe this is how Benno produced his strace binary which he used to dig up some interesting stuff from Android. I’m not sure. Either way, I’m going to be building strace here as an example of how to get this stuff to work.

1. You’re really going to need Linux. Specifically, it sounds like a Debian-based distribution is by far the most likely to work. Since I’ve got a Mac, I had to install Ubuntu Linux 7.10 within a Parallels virtual machine. This is easier said than done, in itself, but once the X-server problems were sorted out it was reliable.
2. First we need to install Scratchbox. It’s available as Debian packages. You can install it either using command-line tools, or the GUI package manager. So naturally I chose half-and-half.
   1. Edit (as root) /etc/apt/sources.list, and add this line: deb http://scratchbox.org/debian stable main
   2. Use the Synaptics Package Manager (in the System menu) to look at the available packages. You’ll need:
      • scratchbox-core
      • scratchbox-libs
      • qemu (not 100% convinced this is really required; I think it may be provided with Scratchbox)
      • scratchbox-devkit-cputransp

      You’ll also need a toolchain. Which toolchain(s) to choose was a bit of a mystery. I’d like to have seen a package using CodeSourcery’s latest compiler, 2007q3. But it seems the Maemo platform, the main user of Scratchbox, is stuck on an older compiler. To be honest, more of an issue than the compiler is probably the version of libc in use within the Android platform. I haven’t figured that out yet, so I’ve been producing statically linked binaries for the moment. Anyway, for my toolchain, I chose scratchbox-toolchain-arm-linux-2006q3-27, which appeared to be the most recent.

   3. Wait for these to install…
3. As per the Scratchbox install document and these Maemo-focussed instructions, add your user to scratchbox: sudo /scratchbox/bsin/sbox_adduser adrian
4. Log out and log in again until the groups command reveals you’re in the sbox group
5. Run scratchbox
6. Run sb-menu
7. Create a new target. Configure it thusly:
   1. Select the compiler you installed, arm-linux-2006q3-27
   2. Select cputransp devkit
   3. Select qemu-arm-0.8.2-sb2 emulation
   4. No to rootstrap
   5. Yes to files. Choose C-library, /etc, Devkits and fakeroot



8. Exit sb-menu. You will be deposited back at the scratchbox shell, but this time the shell is able to run ARM binaries. So things like configure should work.
9. Fetch your source. I built /scratchbox/packages/hello-world.tar.gz and also strace 4.5.15. In theory you ought to be able to build anything (so long as static linkage is OK, and so long as the configure script doesn’t do anything too wacky that qemu can’t emulate). For your first build, perhaps stick to the hello world example?
10. Unzip it/install it/etc. into your scratchbox home. What appears to be /home/xyz is actually located at /scratchbox/users/xyz/home/xyz so that’s where to put the files. (The reasons scratchbox maintains a separate root is so that there’s no chance any of your important system files can be blatted whilst building things for a different architecture).
11. ./configure LDFLAGS=–static (that’s minus-minus static, in case the blog software munges it)
12. make
13. That’s it! You now need to move the generated binary from here onto your Android emulator in the tried and trusted way, then execute it:

MacBook:Desktop adriantaylor$ file hello
hello: ELF 32-bit LSB executable, ARM, version 1 (SYSV), for GNU/Linux 2.6.14, statically linked, for GNU/Linux 2.6.14, not stripped
MacBook:Desktop adriantaylor$ cd android_sdk_darwin_m3-rc20a/tools/
MacBook:tools adriantaylor$ ./adb push ../../hello /system/bin/hello
2006 KB/s (563210 bytes in 0.274s)

In my case, rinse and repeat for strace, then…

MacBook:tools adriantaylor$ ./adb shell
# /system/bin/hello
Hello World!
# /system/bin/strace /system/bin/hello
execve("/system/bin/hello", ["/system/bin/hello"], [/* 8 vars */]) = 0
uname({sys="Linux", node="(none)", ...}) = 0
brk(0)                                  = 0x80000
brk(0x80c70)                            = 0x80c70
syscall_983045(0x80430, 0x7dd88, 0, 0x10, 0x80430, 0x7e734, 0x7ea38, 0xf0005, 0x28, 0x8, 0x4, 0x10, 0, 0xbe94cb08, 0x11da8, 0x8834, 0x60000010, 0x80430, 0, 0, 0, 0xc8c8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = 0
brk(0xa1c70)                            = 0xa1c70
brk(0xa2000)                            = 0xa2000
fstat64(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
write(1, "Hello World!n", 13Hello World!
)          = 13
munmap(0x40000000, 4096)                = 0
exit_group(0)                           = ?
Process 547 detached

Maybe I’ll eventually get some time to get dynamic linkage working, which shouldn’t be too hard, just a matter of working out which libc to use. Famous last words?

Update: yes, dynamic linkage may be too much to hope for. The libc is apparently derived from BSD rather than glibc. And I can’t get these tips to work.

A couple of links: some investigation and some more information on it.

Java is widely regarded as easier to develop for than C++, etc. I tend to agree with that. In C++ it’s easy to find some nasty question to separate the decent developers from the amateurs (for example, ask them how a vtable works).

But even Java has some “low-level” lurking features which only the better Java developer understands. One of these is the class loader. Java normally has a class loader which will search the CLASSPATH environment variable, and/or any JAR file specified, when asked to load a given class. But this can be replaced - and often is.

For example, in Eclipse, each plugin has its own class loader. This means that plugins cannot conflict by having two classes with the same name; it also means that Eclipse can enforce a clear dependency tree on the plugins (the classloaders interact to allow downstream plugins to load classes from upstream plugins).

More or less the same applies to J2EE environments on web servers. Again, this is done so that the different… errr… web thingies have separate environments.

Still other classloaders - such as for good old Java applets - load classes dynamically across a network.

I’m hardly an expert on this stuff, but it seems I know more than some. The vendors of a library I wanted to use load their configuration files using getResourceAsStream(), which is delegated to the class loader. I explained this wasn’t very convenient for people whose class loaders didn’t refer to useful directories. They said that I should just edit the CLASSPATH and insisted that “all Java environments have a CLASSPATH” despite me repeatedly telling them it’s not true. For Eclipse and web applications, at least, the CLASSPATH is irrelevant.

The library is popular and otherwise appears well-designed and reasonably well-written.

All of which makes me think that any decent Java developer would be able to answer an interview question such as “what are some advantages of using a different class loader?” as a good filter at interview for Java developers.

Having said that, I’ve always been rubbish at interviewing so don’t listen to me.

In Excel, suppose you want to produce a table like this:

Obviously you want to use the SUM formula to add up the total number of beasties. That means you can’t literally type “2 oxen” into the cell - you’ll have to just type “2″ but make up a number format to present it as you wish.

That’s easy for singular quantities: the number format would be 0 “sheep”.

But for plurals you have to go a bit further. It turns out you can put multiple formats in the box, separate them with semicolons, and add conditions. So the desired format is this: [=1]0 “ox”;0 “oxen”.

This adds all sorts of potential. What about [=0]”None”;[<4]”A few”;”Lots”? Or [=2]”Company”;[>2]”A crowd”?

I’ve found a neat way to survive using Eclipse on a small screen!

Get rid of the Outline view (I did that ages ago). That frees up a third of your screen which you can devote to code.

The cunning bit is - then - to use Ctrl-O to bring up the Quick Outline view, and type the first few letters of the thing you wish to navigate to. Quicker than using the Outline view in the first place, and takes no screen space!

I haven’t seen this full sequence written down anywhere (only in various partial fragments), so here it is for anyone who finds it useful. Here’s what you need to do if you want to add a Help button to your Eclipse wizard.

1. In your wizard class, call setHelpAvailable(true).
2. In your wizard page class, override performHelp() with code like this:

   public void performHelp() {
     PlatformUI.getWorkbench().getHelpSystem()
       .displayHelp(CONTEXT_ID);
   }
   

3. CONTEXT_ID should be something like “ .mywizardhelp”.
4. Add a help context extension to your plugin. It should look something like:

   <extension
            id="com.wotsit.doodah.wizardhelp"
            point="org.eclipse.help.contexts">
         <contexts
               file="helpContexts.xml"
               />
      </extension>
   

5. Create that helpContexts.xml file in the root of your plug-in. (It can be localised in fragments, incidentally). It should contain something like:

   <contexts>
         <context  id="wizardhelp">
           <description>Gets help about this wizard</description>
           <topic href="html/wizard.html"  label="My wizard help"/>
           <topic href="html/toc.html"  label="General help"/>
     </context>
   </contexts>
   

6. Finally, of course, create the help HTML itself (which you probably did already if you selected the easiest settings when creating your plug-in in the first place).

The bit I couldn’t find documented anywhere was the single line of code which ties it all together! So there might be a better way to do that, but it seems to work.

Useful for test plans, etc.

Insert a field each time you want a number produced, with the formula “Seq XYZ”, where XYZ is any arbitrary label you want. (It won’t appear; it just gives a name to the sequence and ensures that each item with that name gets a unique number).

More advanced tips here.

Back in November I was having difficulty getting the S60 3.1 SDK to run on Parallels. At that time I had access to some Symbian OS source code (through a contract) so was able to debug it. (Albeit not the S60 version, so hacking binaries was needed in the end.)

Now I’ve run into the same problem with the UIQ3.1 SDK, and my previous fix doesn’t quite work! And of course now I have no access to any source code to debug it. However I think I’ve found a fix that works.

Fortunately, my previous instructions contained enough pointless irrelevant information to enable me to figure out a fix this time round.

In short, here’s the change you need to make to your epoc32\release\winscw\udeb\ecust.dll binary.

0002e40: 0000 5959 5dc2 0400 b8bc 1041 00e9 ee5b  ..YY]……A…[
0002e50: 0000 5589 e56a ff68 9d2f 4000 64ff 3500  ..U..j.h./@.d.5.
0002e60: 0000 0064 8925 0000 0000 5053 5657 83ec  …d.%….PSVW..
0002e70: 2851 b8cc cccc cc8d 7c24 04b9 0a00 0000  (Q……|$……
0002e80: f3ab 5989 4dbc c745 c000 c93d 006a 0fc7  ..Y.M..E…=.j..
0002e90: 45fc ffff ffff ff15 4042 4100 50c7 45fc  E…….@BA.P.E.
0002ea0: ffff ffff ff15 4442 4100 8d45 c450 c745  ……DBA..E.P.E
0002eb0: fcff ffff ffff 1548 4241 0083 f800 0f84  …….HBA……
0002ec0: a800 0000 8b45 c048 4848 4848 4848 4848  …..E.HHHHHHHHH
0002ed0: 4848 4848 4848 4848 4848 4848 4848 4848  HHHHHHHHHHHHHHHH

The reason the previous fix didn’t work is that the 84a8 instruction I suggested modifying previously no longer exists. Judging by the other stuff around (in particular the series of 4848484848…) this appears to be the same area of code, but no 84a8. Who knows why - maybe a new compiler version, or slightly changed code, or something.

Fortunately, though, in my previous post I’d noted that it was a loop over 4000000 iterations. 4000000 in hex is 0×3d0900. A bit further up (both in the old code and the new) we see 0009 3d00. Changing the digit before the 9 from a 0 into a c worked. It means the loop runs (many) more times, and therefore Parallels’ high-resolution timer is sufficient to distinguish the start and end of the loop.

(In fact, fiddling around with this number, I find that changing it from 4000000 to 8000000 is sufficient. You may experience differences…)

Yuck.

The standard Java libraries come with a convenient package for reading and creating Zip files (java.util.zip). The class representing a Zip file entry has a method isDirectory() which can tell you whether the entry is a directory.

Unfortunately there is no counterpart, setDirectory(), that can be used when you are creating a zip file.

Usually this doesn’t matter, because when you zip up a file and provide it with a path, the unzipper will create all the directories required to store the file in that path. The only problem, then, is if you need to store empty directories which contain no associated files.

Several solutions are found on the web for this, usually involving storing a slightly different type of filename. The Java API documentation itself suggests that any path ending in a slash will be regarded as a directory. None of these solutions work.

The reason none of them work is that there is actually a bit in the Zip file specification which is used to represent directories (in fact, the relevant bit of the Zip file structure is entitled ‘external attributes’ and is supposed to represent the DOS equivalents - set bit 4 to represent a directory). You can find PHP, etc. code on the web setting this field to 16 to represent directories. The Java zip classes simply don’t set that bit.

Solutions:

• Use the zip library from Apache Commons, which does support these things.
• Use the Zip comments field (or special filenames) to store metadata about the entries you put in your zip. This only works if you also control the unzipping process.

This is amazing. More details here.