Today I managed to get my new MTP module for Node.js working in Electron. There are still a couple of rough edges to smooth over, but I can retrieve a file from the device and then load and read it using Node's fs module:

var list = mtp.getFileListing();
console.log('Files:', list);
mtp.getFile(_.maxBy(list, 'id').id, 'test.ibf');
fs.readFile('test.ibf', (err, data) => {
  return cb(err, data);
});

In this specific instance, there are a bunch of files on the device, and I want to retrieve the one with the highest id, so I use _.maxBy() from lodash. mtp.getFile() gets the file from the device and saves it locally. I then use fs.readFile to read the file from the local disk.

Since I'm using version 3 of N-API and it's only available on Node v10 and higher, I needed to use the latest Electron v3.0.0-beta.4, which has Node v10.2.0 inside. Electron v2 has Node v8 inside, but for v8 N-API is still experimental, and also only supports N-API v1 and v2. Hopefully a stable version of Electron v3 will be released soon, so that I can get this code into the Tidepool Uploader.

#nodemtp

Comment on this post

In Part 3 I described how I started wrapping libmtp in a Node.js module. Today I discovered napi-macros, which makes writing N-API modules much easier. Recall the getFile function we had from last time:

napi_value getFile(napi_env env, napi_callback_info info) {
    napi_status status;

    size_t argc = 2;
    napi_value argv[2];
    status = napi_get_cb_info(env, info, &argc;, argv, NULL, NULL);
    if (status != napi_ok) {
        napi_throw_error(env, NULL, "Failed to parse arguments");
    }

    int id = 0;
    char path[20];
    napi_get_value_int32(env, argv[0], &id;);
    napi_get_value_string_utf8(env, argv[1], path, 20, NULL);

    int ret = LIBMTP_Get_File_To_File(device, id, path, NULL, NULL);

    napi_value retVal;
    status = napi_create_int32(env, ret, &retVal;);
    if (status != napi_ok) {
        napi_throw_error(env, NULL, "Unable to create return value");
    }

    return retVal;
}

I remember when I was writing that it felt very verbose. Now, using napi-macros this collapses into:

NAPI_METHOD(getFile) {
  NAPI_ARGV(2)
  NAPI_ARGV_INT32(id, 0)
  NAPI_ARGV_UTF8_MALLOC(path, 1)

  int ret = LIBMTP_Get_File_To_File(device, id, path, NULL, NULL);

  NAPI_RETURN_INT32(ret)
}

Wow, that's quite the difference – succinct and easier to read. I like it!

#nodemtp

Comment on this post

Today I started on wrapping the libmtp C library in a Node.js module. So far, I've managed to load the list of files on the device and successfully retrieve a file.

Let's start with the bindings.gyp file:

{
  "targets": [{
      "target_name": "module",
      "sources": [ "./src/module.c" ],
      "libraries": [
          "<!@(pkg-config --libs libmtp)"
      ],
      "cflags": [
          "<!@(pkg-config --cflags libmtp)"
      ]
  }],
}

Here I'm specifying that the file containing the C code is in module.c, and that we're using the libmtp shared library. On macOS, use brew install libmtp. On Linux, use sudo apt-get install libmtp-dev. Oh, you'll also need pkg-config if it's not installed on your system yet[1].

Here is the NAPI function to run the code in the C library and return the result to JS:

napi_value getFile(napi_env env, napi_callback_info info) {
  napi_status status;

  size_t argc = 2;
  napi_value argv[2];
  status = napi_get_cb_info(env, info, &argc;, argv, NULL, NULL);
  if (status != napi_ok) {
      napi_throw_error(env, NULL, "Failed to parse arguments");
  }

  int id = 0;
  char path[20];
  napi_get_value_int32(env, argv[0], &id;);
  napi_get_value_string_utf8(env, argv[1], path, 20, NULL);

  int ret = LIBMTP_Get_File_To_File(device, id, path, NULL, NULL);

  napi_value retVal;
  status = napi_create_int32(env, ret, &retVal;);
  if (status != napi_ok) {
      napi_throw_error(env, NULL, "Unable to create return value");
  }

  return retVal;
}

For details on how we get the arguments converted from JS to C, and the return value back to JS, see this tutorial. The line to focus on here is the one with LIBMTP_Get_File_To_File, where we pass the file ID to get from the device, and a path to save that file to.

As we need to connect to the device before we read the file, I added the following code to the NAPI Init function:

napi_value Init(napi_env env, napi_value exports) {
  napi_status status;
  napi_value fn;

  LIBMTP_Init();

  fprintf(stdout, "libmtp version: " LIBMTP_VERSION_STRING "\n\n");

  device = LIBMTP_Get_First_Device();
  if (device == NULL) {
    printf("No devices.\n");
    return 0;
  }

  status = napi_create_function(env, NULL, 0, getFile, NULL, &fn;);
  if (status != napi_ok) {
      napi_throw_error(env, NULL, "Unable to wrap native function");
  }

  status = napi_set_named_property(env, exports, "get_file_to_file", fn);
  if (status != napi_ok) {
      napi_throw_error(env, NULL, "Unable to populate exports");
  }

  return exports;
}

NAPI_MODULE(NODE_GYP_MODULE_NAME, Init)

The line with napi_set_named_property specifies what the function name is called in JS, while napi_create_function creates a JS function from the C function called getFile.

Now it's as easy as just calling the function from JS:

const binding = require('node-gyp-build')(__dirname);
    
console.log(`Status:`, binding.get_file_to_file(1693, 'test.jpg'));

We pass the file ID (1693 in this example) and the path to save file into (test.jpg in the same folder). And that's it, we can read files from an Android phone over MTP on macOS! I'll be posting all the code on GitHub soon once I've cleaned it up a bit.

1. brew install pkg-config on macOS. ↩︎

#nodemtp

Comment on this post

Following up from yesterday, I started using libmtp to see if I can connect to an Android phone and read a file from it. Once the library is successfully compiled, you can run some of the example programs in the examples/ folder. To list all the files on your device, run ./mtp-files. Then, to retrieve a file, you can run

./mtp-connect --getfile <FILE_NUMBER> <DESTINATION>

It seems to work just fine, so the next step is to wrap it as a Node.js library. There's a great article by Mathias Buus on the past and present of handling native (C/C++) modules in Node.js. Basically, N-API is the way to go. It's already stable in Node v10, and works in Node v8 as well.

I followed this tutorial to build a basic native module with N-API, and it works great. The best part was building it and running it in Node v10, and then running the same code in Node v8 without any issues. Nice!

#nodemtp

Comment on this post

Media Transfer Protocol (MTP) has replaced USB Mass Storage as the dominant protocol for transferring digital media between an Android phone and a computer. USB Mass Storage allows devices to be mounted as drives on the host computer, and while MTP also allows drives to be mounted, it happens at a level where it's easier to restrict which files are made available to the host system. If that sounds to you like something Microsoft would design, you'd be right.

MTP works great on Windows machines of course, and surprisingly well on Linux. However, while there's apparently some basic MTP support on macOS, it doesn't even allow you to mount a device without external software, like Android File Transfer.

I spent some time today looking at options for connecting to an Android phone on macOS with MTP using Node.js. Here are some possible options:

1. Use FUSE (Filesystem in Userspace) with Node bindings. This will requires installation of FUSE for macOS first, which I think also involves installing a kernel extension. So it's basically a no-go for macOS High Sierra and above.
2. Write a wrapper for libmtp, using this unmaintained repo as a base.
3. Port parts of this application over to Node.js, as its only dependency seems to be libusb. This means essentially writing your own MTP implementation on top of node-usb.

Since MTP has been standardised as part of the USB Device Class specifications, writing an MTP implementation on node-usb does make sense, but will probably be a lot of work. I think I will opt for (2) first as it's easier, with (3) as a fallback.

#nodemtp

Comment on this post