Gerrit Niezen

100DaysToOffload

A couple of weeks ago I mentioned that I was making a simple hydroponics planter on my 3D printer. Well, I just realised that I haven't even posted a picture of the planter, so here we go:

Hydroponics planter with first basil leaves showing

You may notice that the first leaves of the basil seed that I planted just appeared! 🎉️ I started off the seed in a Root Riot cube, surrounded by some clay pellets. I haven't yet put any nutrient solution in the reservoir, but will as soon as the first roots start appearing.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day55 #hydroponics

This is the final instalment of my second series of blog posts on MTP. I now have my code working in both the browser and Node.js, where it connects to an Android device, reads the latest file and saves it locally.

First off, I had to write some code to figure out whether we're in the browser, in Node.js or in Electron:

let isBrowser = null;

if (typeof navigator !== 'undefined') {
  const userAgent = navigator.userAgent.toLowerCase();
  isBrowser = userAgent.indexOf(' electron/') === -1 && typeof window !== 'undefined';
} else {
  // Node.js process
  isBrowser = false;
}

I'm sure there are modules out there that do this better, but it works for now. Now we have to load some modules for Node.js when we're not in the browser:

let usb, fs = null;

if (!isBrowser) {
  // For Node.js and Electron
  usb = require('webusb').usb;
  EventTarget = require('events');
  fs = require('fs');
} else {
  usb = navigator.usb; // Yay, we're using WebUSB!
}

We also need to be able to save the file to disk in both the browser and Node.js:

const array = new Uint8Array(data.payload);

if (isBrowser) {
  const file = new Blob([array]);
  const a = document.createElement('a'),
  url = URL.createObjectURL(file);
  a.href = url;
  a.download = this.filename;
  document.body.appendChild(a);
  a.click();
  setTimeout(function() {
    document.body.removeChild(a);
    window.URL.revokeObjectURL(url);
  }, 0);
} else {
  fs.writeFileSync(this.filename, array);
}

We also need to deal with event listeners differently in the browser and Node.js:

const initMTP = () => {
  const mtp = new Mtp(0x0e8d, 0x201d);

  if (isBrowser) {
    mtp.addEventListener('error', err => console.log('Error', err));

    mtp.addEventListener('ready', async () => {
      mtp.addEventListener('data', (e) => mtp.dataHandler(e.detail));
      await mtp.openSession();
    } );
  } else {
    mtp.on('error', err => console.log('Error', err));
    mtp.on('ready', async () => {
      mtp.on('data', (data) => mtp.dataHandler(data));
      await mtp.openSession();
    });
  }
};

And finally, WebUSB requires a user interaction before it will show the permission prompt, so we need:

if (isBrowser) {
  document.addEventListener('DOMContentLoaded', event => {
    let button = document.getElementById('connect');

    button.addEventListener('click', async() => {
      initMTP();
    });
 });
} else {
  initMTP();
}

So, that's it! Over on GitHub I have the full implementation. If you run it in Node.js, it will save the newest file on your Android device to disk. If you run it in the browser, it will do the same after you click the Connect button:

<html>
  <head>
    <title>MTP over WebUSB</title>
    <script src="mtp.js"></script>
  </head>
  <body>
    <button id="connect">Connect</button>
  </body>
</html>

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day54 #mtp

In the previous post on MTP we looked at how to retrieve the name of a file on an Android device using MTP. Let's see how to decode that filename:

const array = new Uint8Array(data.payload);
const decoder = new TextDecoder('utf-16le');
filename = decoder.decode(array.subarray(1, array.byteLength - 2));
console.log('Filename:', filename);

In short, we need to remove the length byte at the beginning and the zero terminator bytes at the end, and then decode it as UTF-16LE. To retrieve the file itself, we use the following (where CODE.GET_OBJECT.value is 0x1009):

const getFile = {
  type: 1,
  code: CODE.GET_OBJECT.value,
  payload: [objectHandle],
};
await device.transferOut(0x01, buildContainerPacket(getFile));

All that's left to do is to save the data that is returned to a file! (OK, I also need to write some code that handle files that are larger than the buffer we use for transferIn.)

All the code is currently on a branch on GitHub: https://github.com/tidepool-org/node-mtp/blob/pure-js/mtp.js

Once I have it all working, I hope to replace the Node.js-specific bits so that it can run in the browser using WebUSB, and ideally have a proof-of-concept available on the web.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day53 #mtp

I've been wanting to build my own hydroponics setup for a while, but up until now I've just been using my commercial grow pod for growing stuff. I actually bought some expanded clay pellets, nutrient solution and grow plugs ages ago, but just never actually built the setup.

I think it's due to a number of reasons, the main one being that the original design I was going to build was to be outside, but I don't have a suitable greenhouse to put it in. So I decided to look around at some other designs, specifically ones that:

  • are for indoor use
  • can be completely 3D-printed (as I can't just drive to IKEA in Cardiff due to lockdown)
  • modular, and
  • are very simple to build.

I've been interested in this modular vertical setup for a while, but based on the comments it is prone to leaking and hasn't been updated in a while.

I've been wanting to design a voronoi net cup for fun, and noticed while searching Thingiverse that someone else had the same idea. Not only that, but they have a simple, modular and indoor system to go with it. It doesn't look like the build has been replicated by any other people, but I decided to give it a try and the base is printing right now.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day52 #hydroponics

In the previous post I looked at connecting to an Android device over WebUSB and opening an MTP session.

Let's have a look at how to see what objects (directories, files etc.) are available on the device. Before we get started, let's define some of the operation and response codes that we'll need, as well as some possible container types:

const CODE = {
  OPEN_SESSION: { value: 0x1002, name: 'OpenSession' },
  CLOSE_SESSION: { value: 0x1003, name: 'CloseSession' },
  GET_OBJECT_HANDLES: { value: 0x1007, name: 'GetObjectHandles'},
  OK: { value: 0x2001, name: 'OK'},
  INVALID_PARAMETER: { value: 0x201D, name: 'Invalid parameter'},
  INVALID_OBJECTPROP_FORMAT: { value: 0xA802, name: 'Invalid_ObjectProp_Format'},
  GET_OBJECT_PROP_VALUE: { value: 0x9803, name: 'GetObjectPropValue' },
};

onst TYPE = [
  'undefined',
  'Command Block',
  'Data Block',
  'Response Block',
  'Event Block'
];

We also need to be able to parse container packets:

const parseContainerPacket = (bytes, length) => {
  const fields = {
    type : TYPE[bytes.getUint16(4, true)],
    code : getName(CODE, bytes.getUint16(6, true)),
    transactionID : bytes.getUint32(8, true),
    payload : [],
  };

  for (let i = 12; i < length; i += 4) {
    fields.payload.push(bytes.getUint32(i, true));
  }

  return fields;
};

The getName() helper function looks like this:

const getName = (list, idx) => {
  for (let i in list) {
    if (list[i].value === idx) {
      return list[i].name;
    }
  }
  return 'unknown';
};

To get the object handles, we do the following:

  const getObjectHandles = {
    type: 1, // command block
    code: CODE.GET_OBJECT_HANDLES.value,
    payload: [0xFFFFFFFF, 0, 0xFFFFFFFF], // get all
  };
  data = buildContainerPacket(getObjectHandles, 4);
  result = await device.transferOut(0x01, data);
  console.log('result:', result);
  let { payload } = await receiveData();

Now we have an array of object handles. We can use the object handle to retrieve the filename, using the “Object File Name” object property code 0xDC07:

const getFilename = {
    type: 1,
    code: CODE.GET_OBJECT_PROP_VALUE.value,
    payload: [0x2B, 0xDC07], // object handle and object property code
  };
  result = await device.transferOut(0x01, buildContainerPacket(getFilename));
  console.log('result:', result);
  let { payload } = await receiveData();

The payload is a 16-bit unicode string containing the file name. Nice!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day51 #mtp

Last time I started looking at how to implement a subset of the MTP specification, in order to read files from Android devices. Let's start by writing some code to talk to the device over WebUSB:

const device = await usb.requestDevice({
  filters: [
    {
      vendorId: 3725,
      productId: 8221,
    }
  ]
});

await device.open();

if (device.configuration === null) {
  console.log('selectConfiguration');
  await device.selectConfiguration(1);
}
await device.claimInterface(0);

MTP uses containers to structure what the various commands and responses look like. Here is how to build a container packet (and note that struct.pack() is part of a convenience library):

const buildContainerPacket = (container, payloadLength) => {
  const packetLength = 12 + payloadLength;
  const buf = new ArrayBuffer(packetLength);
  const bytes = new Uint8Array(buf);
  let ctr = struct.pack(bytes, 0, 'issi', packetLength, container.type, container.code, container.transactionID);
  if (payloadLength > 0) {
    ctr += struct.copyBytes(bytes, ctr, container.payload, payloadLength);
  }

  return buf;
};

To open a new session, we fill the container object with the right info and send it off via the bulk pipe:

  const openSession = {
    type: 1, // command block
    code: 0x1002, // open session
    transactionID: 0,
    payload: 1, // session ID
  };
  let data = buildContainerPacket(openSession, 4);
  let result = await device.transferOut(0x01, data);

To read the response, I'm creating a receiveData() function:

  const receiveData = async () => {
    const timeoutID = setTimeout(async() => {
      console.warn('Device not connected');
    }, 5000);

    console.log('Receiving...');

    let incoming = await device.transferIn(0x01, 1024);

    if (incoming.data.byteLength > 0) {
      clearTimeout(timeoutID);

      const [length] = new Uint32Array(incoming.data.buffer, 0, 1);
      console.log('Length:', length);

      // TODO: if length !== incoming.data.byteLength, read more data
    }
  };

To actually receive some data, we call

await receiveData();

and then parse the response (0c 00 00 00 03 00 01 20 01 00 00 00), which is also in the same container format:

  • Length: 0x0C000000 – 12 (in little-endian format)
  • Type: 0x0300 – Response block
  • 0x0120, or 0x2001 in big-endian – response code OK
  • 0x0000000000 – transaction ID

So, we're able to successfully open a session!

PS: Hey, I'm halfway through #100DaysToOffload already! 🎉️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day50 #mtp

I've done a 5-part series on using MTP on MacOS with Node.js in 2018. I also did a 3-part series on using MTP on Windows with Node.js just last month. The learnings from all these posts have been incorporated into my node-mtp Node.js library.

The node-mtp library is essentially a wrapper around the libmtp library, which means it doesn't work in the browser. Given that I eventually want this to work in the browser, and that I'm struggling to get libmtp built on 32-bit Windows, I decided to write a subset of the MTP protocol myself on top of WebUSB.

The MTP protocol spec itself can be downloaded for free from the USB Implementer's Forum. It's a 282-page document, but doesn't itself describe what the packet structure looks like. For that, you need the earlier PTP protocol that it builds on, which has been standardized as ISO 15740. Luckily the USB Implementer's Forum also has the PTP over USB spec, so you don't have to buy the ISO spec to be able to implement it. Before I discovered the PTP over USB spec, this USB device fuzzing code was pretty helpful to start decoding what's going on.

When you connect to an MTP device over USB, you should look for an interface with three endpoints, one of which will be a bulk transfer endpoint. I captured some packets in Wireshark while running Android File Transfer on Linux, which has a nice CLI tool called aft-mtp-cli.

The first bulk transfer packet sent to the device from the computer looks like this:

10000000010002100000000001000000

According to the PTP over USB spec, the structure is as follows:

  • Container Length (4 bytes)
  • Container Type (2 bytes)
  • Code (2 bytes)
  • Transaction ID (4 bytes)
  • Payload

Now we can see that10000000010002100000000001000000 can be broken down into:

  • Length: 0x10000000 – 16 (in little-endian format)
  • Type: 0x0100 – Command Block
  • 0x0210, or 0x1002 in big-endian – operation code OpenSession
  • 0x00000000 – Transaction ID
  • 0x01000000 – Payload, parameter 1: Session ID

Next up, actually using WebUSB to start a new MTP session on the device and read some data!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day49 #mtp

I was trying out the new experimental Native File System API in Chromium, that's still hidden behind a feature flag and in an origin trial. Since it's still being worked out, there are sure to be bugs and I've discovered one on my first try of using it.

When you connect an Android device on a Windows machine, if you just try to select a file using window.chooseFileSystemEntries(), it results in a “can't open files in this folder because it contains system files” error.

My guess is that the problem is that the Native File System API does not have proper support for the MTP protocol used by Android, even though that is one of the design goals of the API:

Apps that want to work with “libraries” of certain types of files. I.e. photo managers, music managers/media players, or even drawing/publishing apps that want access to the raw font files for all fonts on the system.

I reported the bug, and quickly realised I need to provide properly detailed steps to enable them to reproduce the issue before a Chromium developer will look into it. For now, this is the best that I got:

  1. Check that the #native-file-system-api flag is enabled in chrome://flags
  2. Connect Android device to Windows laptop
  3. Go to https://googlechromelabs.github.io/text-editor/
  4. Click on File->Open, navigate to the connected Android device and attempt to open a text file
  5. Chrome responds with a “Can't open this file” error message.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day48

It's getting warmer and sunnier every year during the spring and summer here in Swansea #globalwarming. It's now at the stage where I need something to cool down my small home office when the sun is shining in the afternoon. I would prefer some passive cooling like planting a tree outside, but unfortunately the room is not on the ground floor.

I've been looking at various options for cooling the room, and the state-of-the-art appears to be Dyson's Pure Hot + Cool purifying fan heater. It's an expensive device, but it is essentially three devices in one: A heater, a cooler and a purifier. Unfortunately James Dyson is one of the big Brexiteers, which already make me not want to support him. Then I found out replacement filters for the device is at least £65. It's also probably too big for my little home office.

I then came across the Evapolar evaCHILL. It's a small evaporative air cooler that fits on your desk. It runs on 5V USB power, and only consumes 7.5W when running. The replacement cartridges are a more affordable £25, and should last 3-6 months. I'v ordered one and will report back with a more detailed review once I've tried it out.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day47

Every now and again, I like to bake a no-knead bread. I found this simple recipe a while back, and usually it's a great success. I used to fret about the exact amounts, until I watched the original New York Times video where the no-knead bread recipe was introduced 13 years ago:

After watching that video, I finally realised how easy and quick this method really is. You're probably spending five minutes in total on this recipe, and the only tricky part is timing it right so that you can wait the required periods.

Basically, I throw together the ingredients the night before and put it somewhere warm. The next morning I:

  • fold it and turn it over on a piece of parchment paper,
  • wait 30 minutes and then turn the oven on,
  • wait another 30 minutes and put it in the oven
  • wait another 30 minutes and take the lid off
  • wait 15 minutes and then put it on a wire rack

Delicious, soft bread with an almost sourdough-like consistency for lunch. Yum! 😋️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day46