Gerrit Niezen

100DaysToOffload

I'm putting together a spreadsheet of the various requirements to grow different vegetables in hydroponics, e.g.:

  • days to first harvest
  • pH range
  • EC (Electrical Conductivity) range
  • Light range
  • Temperature

I'm thinking that having something that people could collaboratively edit could be useful. For example, also stating the variety, nutrients, method or grow light that was used. That could lead to creating common “recipes” for specific types of crops. You could even have recipes that change during the growing the period, for example increasing night time temperatures and decreasing daytime temperatures for fruit production in pepper plants after they reach their mature height.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day57 #hydroponics

I somehow got stuck on day 55 of #100DaysToOffload in July and suddenly it's September already! Oh well, I guess it's now day 56.

For the month of September I'm taking a sabbatical from work, to work on something different than what I'm usually working on. I decided to work on DIY hydroponics, which I've mentioned a couple of times on this blog in the past. I bought the book DIY Hydroponic Gardens by Tyler Baras, and started to build one of the designs in the book. The one I'm starting with is called a media bed, using a simple “flood and drain” (also called “ebb and flow” or “ebb and flood”) system. It consists of two containers on top of each other, with water pumped up from the bottom container and draining back into it using a pump and a timer.

I've already spray painted one of the containers, and most of the equipment I need have been ordered. I'll post some photos soon.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day56 #hydroponics

A couple of weeks ago I mentioned that I was making a simple hydroponics planter on my 3D printer. Well, I just realised that I haven't even posted a picture of the planter, so here we go:

Hydroponics planter with first basil leaves showing

You may notice that the first leaves of the basil seed that I planted just appeared! 🎉️ I started off the seed in a Root Riot cube, surrounded by some clay pellets. I haven't yet put any nutrient solution in the reservoir, but will as soon as the first roots start appearing.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day55 #hydroponics

This is the final instalment of my second series of blog posts on MTP. I now have my code working in both the browser and Node.js, where it connects to an Android device, reads the latest file and saves it locally.

First off, I had to write some code to figure out whether we're in the browser, in Node.js or in Electron:

let isBrowser = null;

if (typeof navigator !== 'undefined') {
  const userAgent = navigator.userAgent.toLowerCase();
  isBrowser = userAgent.indexOf(' electron/') === -1 && typeof window !== 'undefined';
} else {
  // Node.js process
  isBrowser = false;
}

I'm sure there are modules out there that do this better, but it works for now. Now we have to load some modules for Node.js when we're not in the browser:

let usb, fs = null;

if (!isBrowser) {
  // For Node.js and Electron
  usb = require('webusb').usb;
  EventTarget = require('events');
  fs = require('fs');
} else {
  usb = navigator.usb; // Yay, we're using WebUSB!
}

We also need to be able to save the file to disk in both the browser and Node.js:

const array = new Uint8Array(data.payload);

if (isBrowser) {
  const file = new Blob([array]);
  const a = document.createElement('a'),
  url = URL.createObjectURL(file);
  a.href = url;
  a.download = this.filename;
  document.body.appendChild(a);
  a.click();
  setTimeout(function() {
    document.body.removeChild(a);
    window.URL.revokeObjectURL(url);
  }, 0);
} else {
  fs.writeFileSync(this.filename, array);
}

We also need to deal with event listeners differently in the browser and Node.js:

const initMTP = () => {
  const mtp = new Mtp(0x0e8d, 0x201d);

  if (isBrowser) {
    mtp.addEventListener('error', err => console.log('Error', err));

    mtp.addEventListener('ready', async () => {
      mtp.addEventListener('data', (e) => mtp.dataHandler(e.detail));
      await mtp.openSession();
    } );
  } else {
    mtp.on('error', err => console.log('Error', err));
    mtp.on('ready', async () => {
      mtp.on('data', (data) => mtp.dataHandler(data));
      await mtp.openSession();
    });
  }
};

And finally, WebUSB requires a user interaction before it will show the permission prompt, so we need:

if (isBrowser) {
  document.addEventListener('DOMContentLoaded', event => {
    let button = document.getElementById('connect');

    button.addEventListener('click', async() => {
      initMTP();
    });
 });
} else {
  initMTP();
}

So, that's it! Over on GitHub I have the full implementation. If you run it in Node.js, it will save the newest file on your Android device to disk. If you run it in the browser, it will do the same after you click the Connect button:

<html>
  <head>
    <title>MTP over WebUSB</title>
    <script src="mtp.js"></script>
  </head>
  <body>
    <button id="connect">Connect</button>
  </body>
</html>

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day54 #mtp

In the previous post on MTP we looked at how to retrieve the name of a file on an Android device using MTP. Let's see how to decode that filename:

const array = new Uint8Array(data.payload);
const decoder = new TextDecoder('utf-16le');
filename = decoder.decode(array.subarray(1, array.byteLength - 2));
console.log('Filename:', filename);

In short, we need to remove the length byte at the beginning and the zero terminator bytes at the end, and then decode it as UTF-16LE. To retrieve the file itself, we use the following (where CODE.GET_OBJECT.value is 0x1009):

const getFile = {
  type: 1,
  code: CODE.GET_OBJECT.value,
  payload: [objectHandle],
};
await device.transferOut(0x01, buildContainerPacket(getFile));

All that's left to do is to save the data that is returned to a file! (OK, I also need to write some code that handle files that are larger than the buffer we use for transferIn.)

All the code is currently on a branch on GitHub: https://github.com/tidepool-org/node-mtp/blob/pure-js/mtp.js

Once I have it all working, I hope to replace the Node.js-specific bits so that it can run in the browser using WebUSB, and ideally have a proof-of-concept available on the web.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day53 #mtp

I've been wanting to build my own hydroponics setup for a while, but up until now I've just been using my commercial grow pod for growing stuff. I actually bought some expanded clay pellets, nutrient solution and grow plugs ages ago, but just never actually built the setup.

I think it's due to a number of reasons, the main one being that the original design I was going to build was to be outside, but I don't have a suitable greenhouse to put it in. So I decided to look around at some other designs, specifically ones that:

  • are for indoor use
  • can be completely 3D-printed (as I can't just drive to IKEA in Cardiff due to lockdown)
  • modular, and
  • are very simple to build.

I've been interested in this modular vertical setup for a while, but based on the comments it is prone to leaking and hasn't been updated in a while.

I've been wanting to design a voronoi net cup for fun, and noticed while searching Thingiverse that someone else had the same idea. Not only that, but they have a simple, modular and indoor system to go with it. It doesn't look like the build has been replicated by any other people, but I decided to give it a try and the base is printing right now.


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day52 #hydroponics

In the previous post I looked at connecting to an Android device over WebUSB and opening an MTP session.

Let's have a look at how to see what objects (directories, files etc.) are available on the device. Before we get started, let's define some of the operation and response codes that we'll need, as well as some possible container types:

const CODE = {
  OPEN_SESSION: { value: 0x1002, name: 'OpenSession' },
  CLOSE_SESSION: { value: 0x1003, name: 'CloseSession' },
  GET_OBJECT_HANDLES: { value: 0x1007, name: 'GetObjectHandles'},
  OK: { value: 0x2001, name: 'OK'},
  INVALID_PARAMETER: { value: 0x201D, name: 'Invalid parameter'},
  INVALID_OBJECTPROP_FORMAT: { value: 0xA802, name: 'Invalid_ObjectProp_Format'},
  GET_OBJECT_PROP_VALUE: { value: 0x9803, name: 'GetObjectPropValue' },
};

onst TYPE = [
  'undefined',
  'Command Block',
  'Data Block',
  'Response Block',
  'Event Block'
];

We also need to be able to parse container packets:

const parseContainerPacket = (bytes, length) => {
  const fields = {
    type : TYPE[bytes.getUint16(4, true)],
    code : getName(CODE, bytes.getUint16(6, true)),
    transactionID : bytes.getUint32(8, true),
    payload : [],
  };

  for (let i = 12; i < length; i += 4) {
    fields.payload.push(bytes.getUint32(i, true));
  }

  return fields;
};

The getName() helper function looks like this:

const getName = (list, idx) => {
  for (let i in list) {
    if (list[i].value === idx) {
      return list[i].name;
    }
  }
  return 'unknown';
};

To get the object handles, we do the following:

  const getObjectHandles = {
    type: 1, // command block
    code: CODE.GET_OBJECT_HANDLES.value,
    payload: [0xFFFFFFFF, 0, 0xFFFFFFFF], // get all
  };
  data = buildContainerPacket(getObjectHandles, 4);
  result = await device.transferOut(0x01, data);
  console.log('result:', result);
  let { payload } = await receiveData();

Now we have an array of object handles. We can use the object handle to retrieve the filename, using the “Object File Name” object property code 0xDC07:

const getFilename = {
    type: 1,
    code: CODE.GET_OBJECT_PROP_VALUE.value,
    payload: [0x2B, 0xDC07], // object handle and object property code
  };
  result = await device.transferOut(0x01, buildContainerPacket(getFilename));
  console.log('result:', result);
  let { payload } = await receiveData();

The payload is a 16-bit unicode string containing the file name. Nice!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day51 #mtp

Last time I started looking at how to implement a subset of the MTP specification, in order to read files from Android devices. Let's start by writing some code to talk to the device over WebUSB:

const device = await usb.requestDevice({
  filters: [
    {
      vendorId: 3725,
      productId: 8221,
    }
  ]
});

await device.open();

if (device.configuration === null) {
  console.log('selectConfiguration');
  await device.selectConfiguration(1);
}
await device.claimInterface(0);

MTP uses containers to structure what the various commands and responses look like. Here is how to build a container packet (and note that struct.pack() is part of a convenience library):

const buildContainerPacket = (container, payloadLength) => {
  const packetLength = 12 + payloadLength;
  const buf = new ArrayBuffer(packetLength);
  const bytes = new Uint8Array(buf);
  let ctr = struct.pack(bytes, 0, 'issi', packetLength, container.type, container.code, container.transactionID);
  if (payloadLength > 0) {
    ctr += struct.copyBytes(bytes, ctr, container.payload, payloadLength);
  }

  return buf;
};

To open a new session, we fill the container object with the right info and send it off via the bulk pipe:

  const openSession = {
    type: 1, // command block
    code: 0x1002, // open session
    transactionID: 0,
    payload: 1, // session ID
  };
  let data = buildContainerPacket(openSession, 4);
  let result = await device.transferOut(0x01, data);

To read the response, I'm creating a receiveData() function:

  const receiveData = async () => {
    const timeoutID = setTimeout(async() => {
      console.warn('Device not connected');
    }, 5000);

    console.log('Receiving...');

    let incoming = await device.transferIn(0x01, 1024);

    if (incoming.data.byteLength > 0) {
      clearTimeout(timeoutID);

      const [length] = new Uint32Array(incoming.data.buffer, 0, 1);
      console.log('Length:', length);

      // TODO: if length !== incoming.data.byteLength, read more data
    }
  };

To actually receive some data, we call

await receiveData();

and then parse the response (0c 00 00 00 03 00 01 20 01 00 00 00), which is also in the same container format:

  • Length: 0x0C000000 – 12 (in little-endian format)
  • Type: 0x0300 – Response block
  • 0x0120, or 0x2001 in big-endian – response code OK
  • 0x0000000000 – transaction ID

So, we're able to successfully open a session!

PS: Hey, I'm halfway through #100DaysToOffload already! 🎉️


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day50 #mtp

I've done a 5-part series on using MTP on MacOS with Node.js in 2018. I also did a 3-part series on using MTP on Windows with Node.js just last month. The learnings from all these posts have been incorporated into my node-mtp Node.js library.

The node-mtp library is essentially a wrapper around the libmtp library, which means it doesn't work in the browser. Given that I eventually want this to work in the browser, and that I'm struggling to get libmtp built on 32-bit Windows, I decided to write a subset of the MTP protocol myself on top of WebUSB.

The MTP protocol spec itself can be downloaded for free from the USB Implementer's Forum. It's a 282-page document, but doesn't itself describe what the packet structure looks like. For that, you need the earlier PTP protocol that it builds on, which has been standardized as ISO 15740. Luckily the USB Implementer's Forum also has the PTP over USB spec, so you don't have to buy the ISO spec to be able to implement it. Before I discovered the PTP over USB spec, this USB device fuzzing code was pretty helpful to start decoding what's going on.

When you connect to an MTP device over USB, you should look for an interface with three endpoints, one of which will be a bulk transfer endpoint. I captured some packets in Wireshark while running Android File Transfer on Linux, which has a nice CLI tool called aft-mtp-cli.

The first bulk transfer packet sent to the device from the computer looks like this:

10000000010002100000000001000000

According to the PTP over USB spec, the structure is as follows:

  • Container Length (4 bytes)
  • Container Type (2 bytes)
  • Code (2 bytes)
  • Transaction ID (4 bytes)
  • Payload

Now we can see that10000000010002100000000001000000 can be broken down into:

  • Length: 0x10000000 – 16 (in little-endian format)
  • Type: 0x0100 – Command Block
  • 0x0210, or 0x1002 in big-endian – operation code OpenSession
  • 0x00000000 – Transaction ID
  • 0x01000000 – Payload, parameter 1: Session ID

Next up, actually using WebUSB to start a new MTP session on the device and read some data!


I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day49 #mtp

I was trying out the new experimental Native File System API in Chromium, that's still hidden behind a feature flag and in an origin trial. Since it's still being worked out, there are sure to be bugs and I've discovered one on my first try of using it.

When you connect an Android device on a Windows machine, if you just try to select a file using window.chooseFileSystemEntries(), it results in a “can't open files in this folder because it contains system files” error.

My guess is that the problem is that the Native File System API does not have proper support for the MTP protocol used by Android, even though that is one of the design goals of the API:

Apps that want to work with “libraries” of certain types of files. I.e. photo managers, music managers/media players, or even drawing/publishing apps that want access to the raw font files for all fonts on the system.

I reported the bug, and quickly realised I need to provide properly detailed steps to enable them to reproduce the issue before a Chromium developer will look into it. For now, this is the best that I got:

  1. Check that the #native-file-system-api flag is enabled in chrome://flags
  2. Connect Android device to Windows laptop
  3. Go to https://googlechromelabs.github.io/text-editor/
  4. Click on File->Open, navigate to the connected Android device and attempt to open a text file
  5. Chrome responds with a “Can't open this file” error message.

I’m publishing this as part of 100 Days To Offload. You can join in yourself by visiting https://100daystooffload.com.

#100DaysToOffload #day48