Well, I still can’t find any software which will allow me to use the standard HP drivers with the DeskJet. (See parts one and two for the explanation). But, there’s another open-source alternative to Gutenprint’s HP driver: HPIJS - which works great! It’s still not brilliantly fast, but it supports multiple paper types, and most importantly, duplex printing - so I’m happy!
Part two of my saga of trying to get my HP DeskJet 940C to print nicely from MacOS X despite it being attached to my ReadyNAS. Bonjour Browser, we can see that the Airport Express is advertising:
So, not IPP at all.
What we need to do, then, is get the ReadyNAS to advertise itself as a remotely controllable USB device, instead of (as well as?) an IPP printer. But do we need _pdl-datastream, or _riousbprint, or both?
Specifically, this is how it appears over Bonjour from the AirportExpress:
_pdl-datastream._tcp. 192.168.0.6:9100 txtvers=1 qtotal=1 note=Living Room product=(HEWLETT-PACKARD DESKJET 940C) pdl=application/vnd.hp-PCL priority=5 usb_MFG=HEWLETT-PACKARD usb_MDL=DESKJET 940C usb_C ty=HEWLETT_PACKARD DESKJET 940C _riousbprint._tcp. 192.168.0.6:10000 txtvers=1 qtotal=1 note=Living Room product=(HEWLETT-PACKARD DESKJET 940C) rp=HEWLETT-PACKARD DESKJET 940C HU1B96N1ZZBH pdl=application/MLC,application/PCL,application/PML priority=1 usb_MFG=HE
Well, looking at a Wireshark dump of the transaction, we’re talking to port 10000: the remote USB service. Unsurprising, really. To be honest it’s more surprising that the Airport Express provides the other service (which makes it look like an HP JetDirect to PCs, I suppose).
So, we “just” need a package we can install on the ReadyNAS using apt, which exposes the USB device using this protocol. Sadly I can’t find such a Linux project, and nor can I find documentation on the protocol so I can’t have a go at writing one myself. Which is probably a good thing, as I have more important things I should be doing!
Summary: it’s not impossible after all, somebody just needs to write some software!
I’ve just bought a ReadyNAS NV+, a NAS box onto which I will do all my backups. It’s great: amongst other things you can log into it using SSH, and use apt-get to install packages, which opens up lots of opportunities such as running my Subversion server on it.
I also wanted to share my old-ish HP DeskJet 940C inkjet from it. Upon connecting it to the USB port on the ReadyNAS, it appeared in Bonjour and I was able to print with it from my Mac - but only using the Gutenprint driver, rather than HP’s own driver. This was not a problem when it was attached to my old Power Mac G4 sitting in the corner… but I’d hoped to retire that. Unfortunately, the Gutenprint driver is slower, doesn’t support different types of paper, nor duplex. It’s good, but it’s not good enough.
I’ve spent a couple of hours researching the problem and have concluded that it’s impossible to solve. Here’s my reasoning. If anyone can find a way round these problems that would be terrific!
MacOS X uses a printing system called CUPS. This knows about the printers installed, and manages queues of documents for them. You can control CUPS using the Printers preference page, or by logging into your own Mac on port 631 using a web browser.
The settings for each CUPS printer consist of three things behind the scenes:
The key part is the PPD. This specifies all the capabilities of the printer. It also specifies how to process the data for the printer… and this is done in terms of the “cupsFilter” element. It describes whether the printer can accept a given MIME type directly. If not, it can also specify that the printer can accept that MIME type if the data is fed through a certain programme. CUPS will chain together these “filters” to convert from the original data type to one which the printer can actually accept. For example,a PICT image might be converted to PDF, then to a general bitmap, then to a bitmap format suitable for the printer.
Here are the cupsFilter details for my HP DeskJet 940c…
| Driver | Connection | Filters |
|---|---|---|
| Gutenprint | Over IPP to ReadyNAS | *cupsFilter: “application/vnd.cups-raster 100 rastertogutenprint.5.1″ |
| HP | Locally over USB | *cupsFilter: “application/pdf 0 /System/Library/Printers/Libraries/PrintJobMgr/Contents/MacOS/PrintJobMgr” |
| IPP to my Mac G4 | *cupsFilter: “application/pdf 0 -” *cupsFilter: “application/pictwps 0 -” *cupsFilter: “image/* 0 -” |
So, for Gutenprint, a general CUPS raster (which CUPS intrinsically knows how to accept) is converted by the program “rastertogutenprint5.1″ to the data format specific for the printer. CUPS then sends that across the network to the ReadyNas, and all is well.
If you’re using the standard HP driver locally, it uses a command “PrintJobMgr” to send the data to the actual printer. This takes a PDF and does something to convert it to HP’s format.
If you’re using an HP printer connected to a remote Mac, things are very different - it just sends the PDF, PICT or any image format directly to the other Mac - which will then send it through CUPS again, eventually resulting in a PDF form of the data making its way into PrintJobMgr on that distant Mac.
So what’s this mysterious PrintJobMgr? Unfortunately that’s where it all gets a bit depressing. Back in the early days of MacOS X (before 10.2) it didn’t use CUPS - instead it used a “Printer Manager” framework, codenamed Tioga. It appears that PrintJobMgr is a call from CUPS into that older system. That means, the process of converting from PDF (etc.) to the HP-specific data stream is not embedded in a nice executable which we can call from a cupsFilter before sending the data to the ReadyNAS. There’s simply no way to get the data out of the HP driver so that we can send it across the network.
So the HP driver is out - we’d have to run it on the ReadyNAS which means we’d have to run OS X on the ReadyNAS.
What about using “gutenprint.5.1″ to convert to the HP data format, but using the rest of the HP PPD file to configure the available options? I haven’t tried this, but I think it’s a non-starter. According to the Gutenprint user’s manual, the rest of the PPD is very specific to the Gutenprint driver. It specifically says that rastertogutenprint5.1 will fail if you use a PPD other than the one which comes with Gutenprint.
Unfortunately, it seems that to use a DeskJet attached to the ReadyNAS we have no choice but to use Gutenprint.
Update: it’s just occurred to me that the Airport Express is supposed to support printer sharing. I can’t believe it’s running the HP printer driver internally. So perhaps there’s a way… more on that later!
Eh? What? Adrian’s blogging again?
Yes. Macrobug has not gone away, just been dormant for the past few months whilst I’ve been working on other things.
But, fear not, because I am about to launch a product!
I’ve debugged why I was unable to display the memory relating to Android binder transactions coming from the kernel to user-side. It turns out that Binder is marking the pages as VM_IO, and the kernel code behind strace’s umoven function was unwilling to dump pages marked as such.
To fix this, just alter drivers/binder/binder.c line 585 (on my copy)…
vma->vm_flags |= VM_RESERVED | VM_READ | VM_RAND_READ | VM_IO | VM_DONTCOPY | VM_DONTEXPAND;
Remove the “VM_IO” from that list.
This appears to me to be a valid change, because the pages aren’t really I/O… they’re used for communicating within the Linux device, not outside of it.
Thanks to Motz for the time-saving instructions on how to build the Android kernel.
Anyway, I now have a complete record of the OpenBinder transactions sent and received by the ’service’ command. I want to do some cleanups, especially around the area of offsets into the transaction data, so I can nicely show the object references involved. I will eventually get around to posting the output (and of course the changes) here.
Then I’ll apply the same tools (again) to the normal Android startup, and after that, I’ll try to feed the output into my existing Macrobug tools and come up with a flowchart of the startup process of Android. At least that’s the plan.
First, a random notice. There’s an Android-Internals wiki, which hopefully will eventually start to describe everything we know about Android internals.
Secondly, here’s the most recent version of my openbinder.c code for strace. (Again, combine with tweaks to the makefiles etc. I’ve previously posted).
And thirdly, thanks to Davanum’s discover of the ’service’ command, we can start to craft some nice simple OpenBinder interactions and see exactly how they work behind the scenes. Results are promising! I now understand the messages sent from the service command to the kernel, although I have yet to decode the replies from the kernel. More on that later.
Without further ado, here is a transcript using the above strace code of the following command: service call phone 2 s16 “123″. Captured using simply strace -o /tmp/strace.out service call phone 2 s16 “123″.
Specifically, we create a transaction whose target is handle 0. I’m not sure what handle 0 means in OpenBinder, but presumably it’s some general OpenBinder framework or service. Logically, that makes sense, because OpenBinder handles must either refer to some local object (in which case they’re just a pointer) or some remote object… in which case the handle must already have been supplied by the OpenBinder driver in some previous transaction or reply. Since we’re the first, and since it’s clearly not a pointer, handle 0 must mean something special. The request we make has operation code 0. Again, I assume that’s just some general registration request.
The request body has 44 bytes of data, of which none are a reference to any other object or entity in the Binder world, whether in our process or elsewhere.
The contents of the 44 bytes is the string we see containing the word ‘default’. This is the request body - really, the parameters to the function which is being called across the process boundary. The contents of these do not appear to be self-describing, and presumably therefore rely on a mutual knowledge of the interface description. Since I don’t know what this service is, I can’t speculate what data it’s providing. But, I can reveal a few things. The first four bytes are the process ID (0×000002ef == 751). I speculate that the subsequent four bytes of 0×0 are the UID. After that, we’ve got the string length (0×07 bytes) then the string. The strings always seem to end in a null (0×00 0×00) even though they’re length-prefixed. Then we see 0×85,0×2a,0×62,0×73… which is a mystery but I see it (or similar) in lots of these bodies so it would be nice to figure it out. Finally, there’s 0×08 0×00 0×00 0×00, then another empty word (0×00 0×00 0×00 0×00).
The kernel driver replies to say that the transaction was complete. It doesn’t return any data. The purpose of the brNOOP is explained in the OpenBinder documentation available on Dianne Hackborn’s website.
Meanwhile, though, we can see a few things. The reply is 24 bytes long, of which there is one flat_binder_object at a certain offset into the buffer. That flat_binder_object will contain a handle to some sort of service discovery object in some other process. At a guess, that handle is going to be 0×01.
Where to go next? I want to:
The odds of me getting around to any of the above any time soon are slim!
Frequently appears in strace listings of what Android is up to. Appears to be setting the thread local storage.
I’ve also found that strace’ing Zygote and the things it spawns is (possibly predictably) much more fruitful in terms of IPC messages than strace’ing the runtime. However I have don’t yet have enough time to figure out how I dig down into the structure enough to find out the destination process for a transaction, from the data available. Once I get that far I’d like to think I can knock up a simple flow diagram of the messages passed between all the Android threads and we can start to get a decent picture of how the whole system operates.
One day…
Here are some superficial thoughts on Android (which I don’t know much about) versus Symbian OS (which I do).
| Android | Symbian OS | |
|---|---|---|
| Business model | ||
| Revenue stream for creator | Advertising, fairy dust? | Fixed license fee per phone (with a few quirks, all publicly known |
| Source code availability | Available to nobody yet. Available to all, in theory, one day. | Available only to Symbian OS phone manufacturers, plus partners who pay a lot. |
| Source code modifiability | Freely modifiable (although I seem to remember constraints for Open Handset Alliance members, presumably related to keeping the APIs sufficiently similar between devices to keep a common platform for developers | Modifiable, but with similar restrictions aimed at keeping a common platform. |
| End-user openness | End devices will probably accept Java-language applications. No evidence that they will accept native applications. | All end devices accept Java-language applications, and virtually all also accept native applications. |
| Current market position in smartphone market | Zero, but has Google behind it | Overwhelmingly dominant in EMEA and Japan, reasonable success elsewhere except America. (American technology writers just don’t realise Symbian exists, and that it’s America which is unusual…) |
| Technical Basis | ||
| API language | Java | C++ (J2ME Java also available for a subset of APIs) |
| Standards-compliance | Highly POSIXy for native handset software. Proprietary for high-level Java APIs. | Proprietary. POSIX layer available, but it limits interaction with the rest of the OS so much that it’s only really used for porting software. |
| Kernel | Linux with minor changes | Proprietary |
| Basic user library | BSD-derived libc | Proprietary |
| User interface | Unknown; current is a ‘placeholder’ | None; UIs developed by Nokia (S60), DoCoMo (MOAP) and UIQ/Motorola/Sony Ericsson (UIQ). Other UIs have existed in the past. |
| Typical filesystem | YAFFS2 | Proprietary. |
| Toolchain for native software | Standard GCC, possibly requiring prelinking and other oddities | Proprietary, built on top of GCC |
| Binary format | Standard ELF | Proprietary |
| Toolchain for typical third party software | Ant-based, dex compiler, etc. | Same toolchain for entire device |
| Debugging/profiling/investigation tools | Standard UNIX tools | Not a lot |
| Inter-process communication | Unclear. D-bus for some layers. OpenBinder for others. Possibly something different again for messages flowing between the Dalvik-side processes. | Proprietary; based on client-server. |
| Features | ||
| Pre-emptive multithreading | Yes | Yes |
| Memory protection between processes | Yes | Yes |
| Pre-emptible kernel | Yes | Yes |
| Demand paging | Yes | Yes |
| Virtual memory (page outs) | Probably not according to Dianne Hackborn; they are anticipating devices with 64MB RAM and 128MB flash | Not yet, but it may not be long; Symbian OS has been in use on phones with hard disks for some time |
| Shared libraries | Yes | Yes |
| Copy-on-write | Yes | N/A (’fork’ is not used) |
| Reference counting of kernel objects and proper cleanup if a process dies | Yes | Yes |
| Approaches to problems | ||
| Flash space taken by many copies of symbol names for dynamic linkage | Unknown; maybe just don’t use too many native programs | Link by ordinal not by name |
| Memory allocation | Allocate a big hunk of virtual address space for process. On write, try to free up enough physical RAM pages to give it the memory it needs. | Allocate minimum virtual and physical address space; heap algorithms in user library know how to request more RAM from kernel. |
| Memory full behaviour | Kernel will kill other processes if necessary to relinquish physical RAM. If no physical RAM is available, program is killed. | On some Symbian systems, user-side heap library may request other apps to exit. If no memory really is available, malloc equivalent throws an exception and application should be able to handle it. |
| Memory management within user code | Check for NULLs from malloc if you’re lucky, but you’ve probably got spare virtual address space anyway so it’s unlikely you’ll be told of a memory allocation failure | Incredibly anal rules about memory management |
| Security | Each process runs as its own user. Unclear how this prevents access to certain APIs that could do destructive things. | Processes have capabilities and can only access APIs appropriate to their capabilities. Applications must be certified and are then signed such that they cannot access APIs beyond their capabilities. |
| Overhead of loading binaries | Store application code as dex files which can just be mmap’ped in | Store application code as native code which can just be mmap’ed in (or equivalent) |
| Overhead of interpreter startup | Application interpreter starts once (Zygote) and all other processes fork from that | No interpreter - native applications |
| Componentisation | Software uses ‘intents’ to say what it wishes to do. Other software can fulfil the intent. | Applications typically rigidly defined (but can load plug-ins). |
| XML and text files are inefficient | Compile XML down to a binary representation | Compile resource files down to a binary representation |
| Overhead of multiple threads | Remove need for multiple threads, but allow developers to use them if they wish. Provide event loop which runs in main thread and can respond to most events. Allow events to be posted onto event loop using handler object. | Remove need for multiple threads, but allow developers to use them if they wish. Provide object-oriented event loop in main thread (’active scheduler’), using ‘active objects’ to respond to each type of incoming event. |
| Overhead of making APIs and libraries thread-safe | Insist they are used only from main thread | Run in a different process; use IPC for all UI requests. And indeed pretty much anything else. |
Some of that’s probably rubbish, and may be based on misinterpreting bits of the interesting podcast. I’m happy to accept corrections, but then again, only six people read this blog anyway so this post was mostly for my own interest :-)
In other news, Motz has figured out the correct linker settings for the Android loader!
Notes from the recent Android podcast…
Here’s my latest openbinder.c file (use with the other changes in my previous post).
Much more sensible output this time. I was right to be suspicious about all the nulls - it turned out I was only transferring the first byte of the data. But still no strings yet… next is to decode the replies from the kernel, then to read the kernel module source a little more closely to work out if we can decode any of the stuff in ‘buffer’. Plus, of course, to keep working on it to see if I’ve made any other daft mistakes :-)
Example output:
22:34:57.044758 ioctl(7, BINDER_WRITE_READ, {write_size=56,write_consumed=56,write_buffer=0×153f8,read_size=256, read_consumed=8,read_buffer=0×152e8,write_data= [bcINCREFS(target=0×00000015)bcACQUIRE(target=0×00000015) bcREPLY({cookie=0×00014130,code=0×00000000,flags=0×00000000,priority=80,data_size=12, offsets_size=0,data={buffer=0×00015970,offsets=0×00000000, *buffer=[0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0×00,0×00,0×00,0×00,…………]}})], read_data={…}) = 0
22:34:57.867208 ioctl(7, BINDER_WRITE_READ, {write_size=40,write_consumed=40,write_buffer=0×153f8,read_size=256,read_consumed=8, read_buffer=0×152e8,write_data=[bcREPLY({cookie=0×00014130,code=0×00000000, flags=0×00000000,priority=80,data_size=24,offsets_size=4,data={buffer=0×00015e18, offsets=0×00014130,*buffer= [0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0×85,0×2a,0×68,0×73,0×08,0×00,0×00,0×00,0×0e,0×00,0×00,0×00,0×04,0xfe,0×4f,0×10, ………*hs……….O.]}})],read_data={…}) = 0
So far the only decent string I’ve seen in any of the buffers is /dev/input/event0… which might give another clue about where to head next.