Gerrit Niezen

Maker of open-source software and hardware.

In the previous post I looked at connecting to an Android device over WebUSB and opening an MTP session.

Let's have a look at how to see what objects (directories, files etc.) are available on the device. Before we get started, let's define some of the operation and response codes that we'll need, as well as some possible container types:

const CODE = {
  OPEN_SESSION: { value: 0x1002, name: 'OpenSession' },
  CLOSE_SESSION: { value: 0x1003, name: 'CloseSession' },
  GET_OBJECT_HANDLES: { value: 0x1007, name: 'GetObjectHandles'},
  OK: { value: 0x2001, name: 'OK'},
  INVALID_PARAMETER: { value: 0x201D, name: 'Invalid parameter'},
  INVALID_OBJECTPROP_FORMAT: { value: 0xA802, name: 'Invalid_ObjectProp_Format'},
  GET_OBJECT_PROP_VALUE: { value: 0x9803, name: 'GetObjectPropValue' },
};

onst TYPE = [
  'undefined',
  'Command Block',
  'Data Block',
  'Response Block',
  'Event Block'
];

We also need to be able to parse container packets:

const parseContainerPacket = (bytes, length) => {
  const fields = {
    type : TYPE[bytes.getUint16(4, true)],
    code : getName(CODE, bytes.getUint16(6, true)),
    transactionID : bytes.getUint32(8, true),
    payload : [],
  };

  for (let i = 12; i < length; i += 4) {
    fields.payload.push(bytes.getUint32(i, true));
  }

  return fields;
};

The getName() helper function looks like this:

const getName = (list, idx) => {
  for (let i in list) {
    if (list[i].value === idx) {
      return list[i].name;
    }
  }
  return 'unknown';
};

To get the object handles, we do the following:

  const getObjectHandles = {
    type: 1, // command block
    code: CODE.GET_OBJECT_HANDLES.value,
    payload: [0xFFFFFFFF, 0, 0xFFFFFFFF], // get all
  };
  data = buildContainerPacket(getObjectHandles, 4);
  result = await device.transferOut(0x01, data);
  console.log('result:', result);
  let { payload } = await receiveData();

Now we have an array of object handles. We can use the object handle to retrieve the filename, using the “Object File Name” object property code 0xDC07:

const getFilename = {
    type: 1,
    code: CODE.GET_OBJECT_PROP_VALUE.value,
    payload: [0x2B, 0xDC07], // object handle and object property code
  };
  result = await device.transferOut(0x01, buildContainerPacket(getFilename));
  console.log('result:', result);
  let { payload } = await receiveData();

The payload is a 16-bit unicode string containing the file name. Nice!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day51 #mtp

Comment on this post

Last time I started looking at how to implement a subset of the MTP specification, in order to read files from Android devices. Let's start by writing some code to talk to the device over WebUSB:

const device = await usb.requestDevice({
  filters: [
    {
      vendorId: 3725,
      productId: 8221,
    }
  ]
});

await device.open();

if (device.configuration === null) {
  console.log('selectConfiguration');
  await device.selectConfiguration(1);
}
await device.claimInterface(0);

MTP uses containers to structure what the various commands and responses look like. Here is how to build a container packet (and note that struct.pack() is part of a convenience library):

const buildContainerPacket = (container, payloadLength) => {
  const packetLength = 12 + payloadLength;
  const buf = new ArrayBuffer(packetLength);
  const bytes = new Uint8Array(buf);
  let ctr = struct.pack(bytes, 0, 'issi', packetLength, container.type, container.code, container.transactionID);
  if (payloadLength > 0) {
    ctr += struct.copyBytes(bytes, ctr, container.payload, payloadLength);
  }

  return buf;
};

To open a new session, we fill the container object with the right info and send it off via the bulk pipe:

  const openSession = {
    type: 1, // command block
    code: 0x1002, // open session
    transactionID: 0,
    payload: 1, // session ID
  };
  let data = buildContainerPacket(openSession, 4);
  let result = await device.transferOut(0x01, data);

To read the response, I'm creating a receiveData() function:

  const receiveData = async () => {
    const timeoutID = setTimeout(async() => {
      console.warn('Device not connected');
    }, 5000);

    console.log('Receiving...');

    let incoming = await device.transferIn(0x01, 1024);

    if (incoming.data.byteLength > 0) {
      clearTimeout(timeoutID);

      const [length] = new Uint32Array(incoming.data.buffer, 0, 1);
      console.log('Length:', length);

      // TODO: if length !== incoming.data.byteLength, read more data
    }
  };

To actually receive some data, we call

await receiveData();

and then parse the response (0c 00 00 00 03 00 01 20 01 00 00 00), which is also in the same container format:

  • Length: 0x0C000000 – 12 (in little-endian format)
  • Type: 0x0300 – Response block
  • 0x0120, or 0x2001 in big-endian – response code OK
  • 0x0000000000 – transaction ID

So, we're able to successfully open a session!

PS: Hey, I'm halfway through #100DaysToOffload already! 🎉️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day50 #mtp

Comment on this post

I've done a 5-part series on using MTP on MacOS with Node.js in 2018. I also did a 3-part series on using MTP on Windows with Node.js just last month. The learnings from all these posts have been incorporated into my node-mtp Node.js library.

The node-mtp library is essentially a wrapper around the libmtp library, which means it doesn't work in the browser. Given that I eventually want this to work in the browser, and that I'm struggling to get libmtp built on 32-bit Windows, I decided to write a subset of the MTP protocol myself on top of WebUSB.

The MTP protocol spec itself can be downloaded for free from the USB Implementer's Forum. It's a 282-page document, but doesn't itself describe what the packet structure looks like. For that, you need the earlier PTP protocol that it builds on, which has been standardized as ISO 15740. Luckily the USB Implementer's Forum also has the PTP over USB spec, so you don't have to buy the ISO spec to be able to implement it. Before I discovered the PTP over USB spec, this USB device fuzzing code was pretty helpful to start decoding what's going on.

When you connect to an MTP device over USB, you should look for an interface with three endpoints, one of which will be a bulk transfer endpoint. I captured some packets in Wireshark while running Android File Transfer on Linux, which has a nice CLI tool called aft-mtp-cli.

The first bulk transfer packet sent to the device from the computer looks like this:

10000000010002100000000001000000

According to the PTP over USB spec, the structure is as follows:

  • Container Length (4 bytes)
  • Container Type (2 bytes)
  • Code (2 bytes)
  • Transaction ID (4 bytes)
  • Payload

Now we can see that10000000010002100000000001000000 can be broken down into:

  • Length: 0x10000000 – 16 (in little-endian format)
  • Type: 0x0100 – Command Block
  • 0x0210, or 0x1002 in big-endian – operation code OpenSession
  • 0x00000000 – Transaction ID
  • 0x01000000 – Payload, parameter 1: Session ID

Next up, actually using WebUSB to start a new MTP session on the device and read some data!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day49 #mtp

Comment on this post

I was trying out the new experimental Native File System API in Chromium, that's still hidden behind a feature flag and in an origin trial. Since it's still being worked out, there are sure to be bugs and I've discovered one on my first try of using it.

When you connect an Android device on a Windows machine, if you just try to select a file using window.chooseFileSystemEntries(), it results in a “can't open files in this folder because it contains system files” error.

My guess is that the problem is that the Native File System API does not have proper support for the MTP protocol used by Android, even though that is one of the design goals of the API:

Apps that want to work with “libraries” of certain types of files. I.e. photo managers, music managers/media players, or even drawing/publishing apps that want access to the raw font files for all fonts on the system.

I reported the bug, and quickly realised I need to provide properly detailed steps to enable them to reproduce the issue before a Chromium developer will look into it. For now, this is the best that I got:

  1. Check that the #native-file-system-api flag is enabled in chrome://flags
  2. Connect Android device to Windows laptop
  3. Go to https://googlechromelabs.github.io/text-editor/
  4. Click on File->Open, navigate to the connected Android device and attempt to open a text file
  5. Chrome responds with a “Can't open this file” error message.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day48

Comment on this post

It's getting warmer and sunnier every year during the spring and summer here in Swansea #globalwarming. It's now at the stage where I need something to cool down my small home office when the sun is shining in the afternoon. I would prefer some passive cooling like planting a tree outside, but unfortunately the room is not on the ground floor.

I've been looking at various options for cooling the room, and the state-of-the-art appears to be Dyson's Pure Hot + Cool purifying fan heater. It's an expensive device, but it is essentially three devices in one: A heater, a cooler and a purifier. Unfortunately James Dyson is one of the big Brexiteers, which already make me not want to support him. Then I found out replacement filters for the device is at least £65. It's also probably too big for my little home office.

I then came across the Evapolar evaCHILL. It's a small evaporative air cooler that fits on your desk. It runs on 5V USB power, and only consumes 7.5W when running. The replacement cartridges are a more affordable £25, and should last 3-6 months. I'v ordered one and will report back with a more detailed review once I've tried it out.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day47

Comment on this post

Every now and again, I like to bake a no-knead bread. I found this simple recipe a while back, and usually it's a great success. I used to fret about the exact amounts, until I watched the original New York Times video where the no-knead bread recipe was introduced 13 years ago:

After watching that video, I finally realised how easy and quick this method really is. You're probably spending five minutes in total on this recipe, and the only tricky part is timing it right so that you can wait the required periods.

Basically, I throw together the ingredients the night before and put it somewhere warm. The next morning I:

  • fold it and turn it over on a piece of parchment paper,
  • wait 30 minutes and then turn the oven on,
  • wait another 30 minutes and put it in the oven
  • wait another 30 minutes and take the lid off
  • wait 15 minutes and then put it on a wire rack

Delicious, soft bread with an almost sourdough-like consistency for lunch. Yum! 😋️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day46

Comment on this post

On Linux, getting details about the USB devices connected to your computer is as simple as typing lsusb. If you want even more info, like what kernel driver is being used, you can use usb-devices.

On MacOS, things are a little bit more complicated. You can open System Information and look under USB, or you can type system_profiler SPUSBDataType.

On Windows, it's a bit of a nightmare. Sure, you can open Device Manager if you're just looking for some USB product ID and vendor ID details, but what if you want to list all the details of all the USB devices connected? The official Microsoft advice is to:

  • Download and install the Windows SDK
  • During the installation, select only the Debugging Tools for Windows box and clear all other boxes.
  • Navigate to the right directory (probably C:\Program Files (x86)\Windows Kits\10\Debuggers\x64) and then select USBView.exe to start the utility.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day45

Comment on this post

I saw this toot on Mastodon today and decided to see if there's anything interesting on Bandcamp:

And would you know, within a couple of minutes I discovered an amazing chiptune album created by someone living in my town (!):

Not only that, but this same person also created a Nintendo Game Boy game during lockdown, and their blog is pretty delightful too.

The things you can discover when you don't depend on algorithms to do your discovery for you, which is basically what I've done with Spotify for the last decade.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day44

Comment on this post

I've been growing a Habanero chilli plant in a small hydroponics setup for a while now, and harvested enough chilli peppers a couple of months ago to make this 5-ingredient Habanero hot sauce.

I enjoyed it so much that I'm planning on making it again tonight, as the chilli plant is very heavy with peppers again.

Habanero chilli plant grown hydroponically


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day43

Comment on this post

I'm about halfway through reading Humankind: A Hopeful History by Rutger Bregman. So far it's been a really insightful read, and it's one of the few books that I've read lately where I can say I've learnt a lot. I want to say I wish that this is a book everyone would read, but I'll wait until I've actually finished the book before I actually say that.

It's not a quick read, coming in at about 480 pages, but it covers a lot and is written in very accessible language. So far it has focused on a lot of scientific studies and debunking ones that try to show that humans are selfish by nature. I'm hoping the second half of the book will be looking at some really actionable advice for the future.

I also enjoyed Bregman's 2017 book Utopia for Realists, but it hasn't had the same perspective-shifting impact on me as Humankind has had so far.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day42

Comment on this post

Enter your email to subscribe to updates.