Gerrit Niezen

Maker of open-source software and hardware.

Last time I started looking at how to implement a subset of the MTP specification, in order to read files from Android devices. Let's start by writing some code to talk to the device over WebUSB:

const device = await usb.requestDevice({
  filters: [
    {
      vendorId: 3725,
      productId: 8221,
    }
  ]
});

await device.open();

if (device.configuration === null) {
  console.log('selectConfiguration');
  await device.selectConfiguration(1);
}
await device.claimInterface(0);

MTP uses containers to structure what the various commands and responses look like. Here is how to build a container packet (and note that struct.pack() is part of a convenience library):

const buildContainerPacket = (container, payloadLength) => {
  const packetLength = 12 + payloadLength;
  const buf = new ArrayBuffer(packetLength);
  const bytes = new Uint8Array(buf);
  let ctr = struct.pack(bytes, 0, 'issi', packetLength, container.type, container.code, container.transactionID);
  if (payloadLength > 0) {
    ctr += struct.copyBytes(bytes, ctr, container.payload, payloadLength);
  }

  return buf;
};

To open a new session, we fill the container object with the right info and send it off via the bulk pipe:

  const openSession = {
    type: 1, // command block
    code: 0x1002, // open session
    transactionID: 0,
    payload: 1, // session ID
  };
  let data = buildContainerPacket(openSession, 4);
  let result = await device.transferOut(0x01, data);

To read the response, I'm creating a receiveData() function:

  const receiveData = async () => {
    const timeoutID = setTimeout(async() => {
      console.warn('Device not connected');
    }, 5000);

    console.log('Receiving...');

    let incoming = await device.transferIn(0x01, 1024);

    if (incoming.data.byteLength > 0) {
      clearTimeout(timeoutID);

      const [length] = new Uint32Array(incoming.data.buffer, 0, 1);
      console.log('Length:', length);

      // TODO: if length !== incoming.data.byteLength, read more data
    }
  };

To actually receive some data, we call

await receiveData();

and then parse the response (0c 00 00 00 03 00 01 20 01 00 00 00), which is also in the same container format:

  • Length: 0x0C000000 – 12 (in little-endian format)
  • Type: 0x0300 – Response block
  • 0x0120, or 0x2001 in big-endian – response code OK
  • 0x0000000000 – transaction ID

So, we're able to successfully open a session!

PS: Hey, I'm halfway through #100DaysToOffload already! 🎉️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day50 #mtp

I've done a 5-part series on using MTP on MacOS with Node.js in 2018. I also did a 3-part series on using MTP on Windows with Node.js just last month. The learnings from all these posts have been incorporated into my node-mtp Node.js library.

The node-mtp library is essentially a wrapper around the libmtp library, which means it doesn't work in the browser. Given that I eventually want this to work in the browser, and that I'm struggling to get libmtp built on 32-bit Windows, I decided to write a subset of the MTP protocol myself on top of WebUSB.

The MTP protocol spec itself can be downloaded for free from the USB Implementer's Forum. It's a 282-page document, but doesn't itself describe what the packet structure looks like. For that, you need the earlier PTP protocol that it builds on, which has been standardized as ISO 15740. Luckily the USB Implementer's Forum also has the PTP over USB spec, so you don't have to buy the ISO spec to be able to implement it. Before I discovered the PTP over USB spec, this USB device fuzzing code was pretty helpful to start decoding what's going on.

When you connect to an MTP device over USB, you should look for an interface with three endpoints, one of which will be a bulk transfer endpoint. I captured some packets in Wireshark while running Android File Transfer on Linux, which has a nice CLI tool called aft-mtp-cli.

The first bulk transfer packet sent to the device from the computer looks like this:

10000000010002100000000001000000

According to the PTP over USB spec, the structure is as follows:

  • Container Length (4 bytes)
  • Container Type (2 bytes)
  • Code (2 bytes)
  • Transaction ID (4 bytes)
  • Payload

Now we can see that10000000010002100000000001000000 can be broken down into:

  • Length: 0x10000000 – 16 (in little-endian format)
  • Type: 0x0100 – Command Block
  • 0x0210, or 0x1002 in big-endian – operation code OpenSession
  • 0x00000000 – Transaction ID
  • 0x01000000 – Payload, parameter 1: Session ID

Next up, actually using WebUSB to start a new MTP session on the device and read some data!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day49 #mtp

I was trying out the new experimental Native File System API in Chromium, that's still hidden behind a feature flag and in an origin trial. Since it's still being worked out, there are sure to be bugs and I've discovered one on my first try of using it.

When you connect an Android device on a Windows machine, if you just try to select a file using window.chooseFileSystemEntries(), it results in a “can't open files in this folder because it contains system files” error.

My guess is that the problem is that the Native File System API does not have proper support for the MTP protocol used by Android, even though that is one of the design goals of the API:

Apps that want to work with “libraries” of certain types of files. I.e. photo managers, music managers/media players, or even drawing/publishing apps that want access to the raw font files for all fonts on the system.

I reported the bug, and quickly realised I need to provide properly detailed steps to enable them to reproduce the issue before a Chromium developer will look into it. For now, this is the best that I got:

  1. Check that the #native-file-system-api flag is enabled in chrome://flags
  2. Connect Android device to Windows laptop
  3. Go to https://googlechromelabs.github.io/text-editor/
  4. Click on File->Open, navigate to the connected Android device and attempt to open a text file
  5. Chrome responds with a “Can't open this file” error message.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day48

It's getting warmer and sunnier every year during the spring and summer here in Swansea #globalwarming. It's now at the stage where I need something to cool down my small home office when the sun is shining in the afternoon. I would prefer some passive cooling like planting a tree outside, but unfortunately the room is not on the ground floor.

I've been looking at various options for cooling the room, and the state-of-the-art appears to be Dyson's Pure Hot + Cool purifying fan heater. It's an expensive device, but it is essentially three devices in one: A heater, a cooler and a purifier. Unfortunately James Dyson is one of the big Brexiteers, which already make me not want to support him. Then I found out replacement filters for the device is at least £65. It's also probably too big for my little home office.

I then came across the Evapolar evaCHILL. It's a small evaporative air cooler that fits on your desk. It runs on 5V USB power, and only consumes 7.5W when running. The replacement cartridges are a more affordable £25, and should last 3-6 months. I'v ordered one and will report back with a more detailed review once I've tried it out.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day47

Every now and again, I like to bake a no-knead bread. I found this simple recipe a while back, and usually it's a great success. I used to fret about the exact amounts, until I watched the original New York Times video where the no-knead bread recipe was introduced 13 years ago:

After watching that video, I finally realised how easy and quick this method really is. You're probably spending five minutes in total on this recipe, and the only tricky part is timing it right so that you can wait the required periods.

Basically, I throw together the ingredients the night before and put it somewhere warm. The next morning I:

  • fold it and turn it over on a piece of parchment paper,
  • wait 30 minutes and then turn the oven on,
  • wait another 30 minutes and put it in the oven
  • wait another 30 minutes and take the lid off
  • wait 15 minutes and then put it on a wire rack

Delicious, soft bread with an almost sourdough-like consistency for lunch. Yum! 😋️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day46

On Linux, getting details about the USB devices connected to your computer is as simple as typing lsusb. If you want even more info, like what kernel driver is being used, you can use usb-devices.

On MacOS, things are a little bit more complicated. You can open System Information and look under USB, or you can type system_profiler SPUSBDataType.

On Windows, it's a bit of a nightmare. Sure, you can open Device Manager if you're just looking for some USB product ID and vendor ID details, but what if you want to list all the details of all the USB devices connected? The official Microsoft advice is to:

  • Download and install the Windows SDK
  • During the installation, select only the Debugging Tools for Windows box and clear all other boxes.
  • Navigate to the right directory (probably C:\Program Files (x86)\Windows Kits\10\Debuggers\x64) and then select USBView.exe to start the utility.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day45

I saw this toot on Mastodon today and decided to see if there's anything interesting on Bandcamp:

And would you know, within a couple of minutes I discovered an amazing chiptune album created by someone living in my town (!):

Not only that, but this same person also created a Nintendo Game Boy game during lockdown, and their blog is pretty delightful too.

The things you can discover when you don't depend on algorithms to do your discovery for you, which is basically what I've done with Spotify for the last decade.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day44

I've been growing a Habanero chilli plant in a small hydroponics setup for a while now, and harvested enough chilli peppers a couple of months ago to make this 5-ingredient Habanero hot sauce.

I enjoyed it so much that I'm planning on making it again tonight, as the chilli plant is very heavy with peppers again.

Habanero chilli plant grown hydroponically


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day43

I'm about halfway through reading Humankind: A Hopeful History by Rutger Bregman. So far it's been a really insightful read, and it's one of the few books that I've read lately where I can say I've learnt a lot. I want to say I wish that this is a book everyone would read, but I'll wait until I've actually finished the book before I actually say that.

It's not a quick read, coming in at about 480 pages, but it covers a lot and is written in very accessible language. So far it has focused on a lot of scientific studies and debunking ones that try to show that humans are selfish by nature. I'm hoping the second half of the book will be looking at some really actionable advice for the future.

I also enjoyed Bregman's 2017 book Utopia for Realists, but it hasn't had the same perspective-shifting impact on me as Humankind has had so far.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day42

I'm pretty excited about a new project here in my neighbourhood that's getting off the ground called Room to Grow. They basically want to convert people's front gardens into places to grow food.

I recently came across a book called Eat Your Front Garden by Mat Coward. It describes various plants that may look ornamental, but are actually edible. I was hoping to plant some of them in our own front garden, so it's great to see that there are other people in my local community thinking the same thing. Hopefully I can get more involved with the project, instead of just doing it on my own.

If you're based in Swansea, maybe have a look at their crowdfunding page! The people behind this are the same ones behind a lovely local cafe called Hoogah.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day41

Enter your email to subscribe to updates.