Gerrit Niezen

mtp

This is the final instalment of my second series of blog posts on MTP. I now have my code working in both the browser and Node.js, where it connects to an Android device, reads the latest file and saves it locally.

First off, I had to write some code to figure out whether we're in the browser, in Node.js or in Electron:

let isBrowser = null;

if (typeof navigator !== 'undefined') {
  const userAgent = navigator.userAgent.toLowerCase();
  isBrowser = userAgent.indexOf(' electron/') === -1 && typeof window !== 'undefined';
} else {
  // Node.js process
  isBrowser = false;
}

I'm sure there are modules out there that do this better, but it works for now. Now we have to load some modules for Node.js when we're not in the browser:

let usb, fs = null;

if (!isBrowser) {
  // For Node.js and Electron
  usb = require('webusb').usb;
  EventTarget = require('events');
  fs = require('fs');
} else {
  usb = navigator.usb; // Yay, we're using WebUSB!
}

We also need to be able to save the file to disk in both the browser and Node.js:

const array = new Uint8Array(data.payload);

if (isBrowser) {
  const file = new Blob([array]);
  const a = document.createElement('a'),
  url = URL.createObjectURL(file);
  a.href = url;
  a.download = this.filename;
  document.body.appendChild(a);
  a.click();
  setTimeout(function() {
    document.body.removeChild(a);
    window.URL.revokeObjectURL(url);
  }, 0);
} else {
  fs.writeFileSync(this.filename, array);
}

We also need to deal with event listeners differently in the browser and Node.js:

const initMTP = () => {
  const mtp = new Mtp(0x0e8d, 0x201d);

  if (isBrowser) {
    mtp.addEventListener('error', err => console.log('Error', err));

    mtp.addEventListener('ready', async () => {
      mtp.addEventListener('data', (e) => mtp.dataHandler(e.detail));
      await mtp.openSession();
    } );
  } else {
    mtp.on('error', err => console.log('Error', err));
    mtp.on('ready', async () => {
      mtp.on('data', (data) => mtp.dataHandler(data));
      await mtp.openSession();
    });
  }
};

And finally, WebUSB requires a user interaction before it will show the permission prompt, so we need:

if (isBrowser) {
  document.addEventListener('DOMContentLoaded', event => {
    let button = document.getElementById('connect');

    button.addEventListener('click', async() => {
      initMTP();
    });
 });
} else {
  initMTP();
}

So, that's it! Over on GitHub I have the full implementation. If you run it in Node.js, it will save the newest file on your Android device to disk. If you run it in the browser, it will do the same after you click the Connect button:

<html>
  <head>
    <title>MTP over WebUSB</title>
    <script src="mtp.js"></script>
  </head>
  <body>
    <button id="connect">Connect</button>
  </body>
</html>

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day54 #mtp

In the previous post on MTP we looked at how to retrieve the name of a file on an Android device using MTP. Let's see how to decode that filename:

const array = new Uint8Array(data.payload);
const decoder = new TextDecoder('utf-16le');
filename = decoder.decode(array.subarray(1, array.byteLength - 2));
console.log('Filename:', filename);

In short, we need to remove the length byte at the beginning and the zero terminator bytes at the end, and then decode it as UTF-16LE. To retrieve the file itself, we use the following (where CODE.GET_OBJECT.value is 0x1009):

const getFile = {
  type: 1,
  code: CODE.GET_OBJECT.value,
  payload: [objectHandle],
};
await device.transferOut(0x01, buildContainerPacket(getFile));

All that's left to do is to save the data that is returned to a file! (OK, I also need to write some code that handle files that are larger than the buffer we use for transferIn.)

All the code is currently on a branch on GitHub: https://github.com/tidepool-org/node-mtp/blob/pure-js/mtp.js

Once I have it all working, I hope to replace the Node.js-specific bits so that it can run in the browser using WebUSB, and ideally have a proof-of-concept available on the web.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day53 #mtp

In the previous post I looked at connecting to an Android device over WebUSB and opening an MTP session.

Let's have a look at how to see what objects (directories, files etc.) are available on the device. Before we get started, let's define some of the operation and response codes that we'll need, as well as some possible container types:

const CODE = {
  OPEN_SESSION: { value: 0x1002, name: 'OpenSession' },
  CLOSE_SESSION: { value: 0x1003, name: 'CloseSession' },
  GET_OBJECT_HANDLES: { value: 0x1007, name: 'GetObjectHandles'},
  OK: { value: 0x2001, name: 'OK'},
  INVALID_PARAMETER: { value: 0x201D, name: 'Invalid parameter'},
  INVALID_OBJECTPROP_FORMAT: { value: 0xA802, name: 'Invalid_ObjectProp_Format'},
  GET_OBJECT_PROP_VALUE: { value: 0x9803, name: 'GetObjectPropValue' },
};

onst TYPE = [
  'undefined',
  'Command Block',
  'Data Block',
  'Response Block',
  'Event Block'
];

We also need to be able to parse container packets:

const parseContainerPacket = (bytes, length) => {
  const fields = {
    type : TYPE[bytes.getUint16(4, true)],
    code : getName(CODE, bytes.getUint16(6, true)),
    transactionID : bytes.getUint32(8, true),
    payload : [],
  };

  for (let i = 12; i < length; i += 4) {
    fields.payload.push(bytes.getUint32(i, true));
  }

  return fields;
};

The getName() helper function looks like this:

const getName = (list, idx) => {
  for (let i in list) {
    if (list[i].value === idx) {
      return list[i].name;
    }
  }
  return 'unknown';
};

To get the object handles, we do the following:

  const getObjectHandles = {
    type: 1, // command block
    code: CODE.GET_OBJECT_HANDLES.value,
    payload: [0xFFFFFFFF, 0, 0xFFFFFFFF], // get all
  };
  data = buildContainerPacket(getObjectHandles, 4);
  result = await device.transferOut(0x01, data);
  console.log('result:', result);
  let { payload } = await receiveData();

Now we have an array of object handles. We can use the object handle to retrieve the filename, using the “Object File Name” object property code 0xDC07:

const getFilename = {
    type: 1,
    code: CODE.GET_OBJECT_PROP_VALUE.value,
    payload: [0x2B, 0xDC07], // object handle and object property code
  };
  result = await device.transferOut(0x01, buildContainerPacket(getFilename));
  console.log('result:', result);
  let { payload } = await receiveData();

The payload is a 16-bit unicode string containing the file name. Nice!

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day51 #mtp

Last time I started looking at how to implement a subset of the MTP specification, in order to read files from Android devices. Let's start by writing some code to talk to the device over WebUSB:

const device = await usb.requestDevice({
  filters: [
    {
      vendorId: 3725,
      productId: 8221,
    }
  ]
});

await device.open();

if (device.configuration === null) {
  console.log('selectConfiguration');
  await device.selectConfiguration(1);
}
await device.claimInterface(0);

MTP uses containers to structure what the various commands and responses look like. Here is how to build a container packet (and note that struct.pack() is part of a convenience library):

const buildContainerPacket = (container, payloadLength) => {
  const packetLength = 12 + payloadLength;
  const buf = new ArrayBuffer(packetLength);
  const bytes = new Uint8Array(buf);
  let ctr = struct.pack(bytes, 0, 'issi', packetLength, container.type, container.code, container.transactionID);
  if (payloadLength > 0) {
    ctr += struct.copyBytes(bytes, ctr, container.payload, payloadLength);
  }

  return buf;
};

To open a new session, we fill the container object with the right info and send it off via the bulk pipe:

  const openSession = {
    type: 1, // command block
    code: 0x1002, // open session
    transactionID: 0,
    payload: 1, // session ID
  };
  let data = buildContainerPacket(openSession, 4);
  let result = await device.transferOut(0x01, data);

To read the response, I'm creating a receiveData() function:

  const receiveData = async () => {
    const timeoutID = setTimeout(async() => {
      console.warn('Device not connected');
    }, 5000);

    console.log('Receiving...');

    let incoming = await device.transferIn(0x01, 1024);

    if (incoming.data.byteLength > 0) {
      clearTimeout(timeoutID);

      const [length] = new Uint32Array(incoming.data.buffer, 0, 1);
      console.log('Length:', length);

      // TODO: if length !== incoming.data.byteLength, read more data
    }
  };

To actually receive some data, we call

await receiveData();

and then parse the response (0c 00 00 00 03 00 01 20 01 00 00 00), which is also in the same container format:

  • Length: 0x0C000000 – 12 (in little-endian format)
  • Type: 0x0300 – Response block
  • 0x0120, or 0x2001 in big-endian – response code OK
  • 0x0000000000 – transaction ID

So, we're able to successfully open a session!

PS: Hey, I'm halfway through #100DaysToOffload already! 🎉️

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day50 #mtp

I've done a 5-part series on using MTP on MacOS with Node.js in 2018. I also did a 3-part series on using MTP on Windows with Node.js just last month. The learnings from all these posts have been incorporated into my node-mtp Node.js library.

The node-mtp library is essentially a wrapper around the libmtp library, which means it doesn't work in the browser. Given that I eventually want this to work in the browser, and that I'm struggling to get libmtp built on 32-bit Windows, I decided to write a subset of the MTP protocol myself on top of WebUSB.

The MTP protocol spec itself can be downloaded for free from the USB Implementer's Forum. It's a 282-page document, but doesn't itself describe what the packet structure looks like. For that, you need the earlier PTP protocol that it builds on, which has been standardized as ISO 15740. Luckily the USB Implementer's Forum also has the PTP over USB spec, so you don't have to buy the ISO spec to be able to implement it. Before I discovered the PTP over USB spec, this USB device fuzzing code was pretty helpful to start decoding what's going on.

When you connect to an MTP device over USB, you should look for an interface with three endpoints, one of which will be a bulk transfer endpoint. I captured some packets in Wireshark while running Android File Transfer on Linux, which has a nice CLI tool called aft-mtp-cli.

The first bulk transfer packet sent to the device from the computer looks like this:

10000000010002100000000001000000

According to the PTP over USB spec, the structure is as follows:

  • Container Length (4 bytes)
  • Container Type (2 bytes)
  • Code (2 bytes)
  • Transaction ID (4 bytes)
  • Payload

Now we can see that10000000010002100000000001000000 can be broken down into:

  • Length: 0x10000000 – 16 (in little-endian format)
  • Type: 0x0100 – Command Block
  • 0x0210, or 0x1002 in big-endian – operation code OpenSession
  • 0x00000000 – Transaction ID
  • 0x01000000 – Payload, parameter 1: Session ID

Next up, actually using WebUSB to start a new MTP session on the device and read some data!

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day49 #mtp