Cloud Tutorial: API with Node.js and Datastore

You should follow this tutorial on a Compute Engine VM with HTTP networking and full access to the cloud APIs (these are parameters you can set when creating the VM).

You will need Node.js version 10 or later, git, and the gcloud tools. On a Compute Engine VM, gcloud tools are installed by default; so we only need to install the other two:

curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt-get install -y nodejs git

The tutorial assumes that you are starting in an empty directory but have a copy of this repository at hand.

Because this tutorial doesn't look at how the client web page is done, please just copy the directory static/ and its contents from this repository.

You should now have a directory with static/ inside it.

Run npm init, answer the prompts, the defaults are generally OK because we won't be publishing this package.

For that, we will need the express package, so run npm install express.

Now create the main file for your app, which we can name app.js. The following is its basic content:

const express = require('express');

const app = express();

app.use(express.static('static', { extensions: ['html'] }));

const port = process.env.PORT || 8080;

app.listen(port, (err) => {
  if (err) console.log('error', err);
  else console.log(`app listening on port ${port}`);
});

You can test this app on your VM by running sudo PORT=80 node app and connecting to your machine's external IP address. It will serve the static pages only, so you can see that the app can retrieve /style.css and /script.js. The file drop-down box will say "error" because we don't have an API yet.

For example, if your VM has the IP address 10.11.12.13, then you should reach style.css at http://10.11.12.13/style.css. Note that it has to be http: and not https: – the server code does not support HTTPS.

Our app first needs to be able to load the list of available files. In an API that only has a simple list of files, we can return the list of available files at the root of the API:

GET /api/ -> returns ["message", "pets"]

It is good practice to split the API routes away from the main server file, therefore we will put the route in a separate module. Create a folder called api and copy the following code into api/index.js:

const express = require('express');
const api = express.Router();
module.exports = api;

api.get('/', (req, res) => {
  res.json(["message", "pets"]);
});

The code above gives us a Javascript module that exports an Express router, which is a collection of routes. The only route in there simply returns a hard-coded list of files: "message" and "pets".

Now add the following line into app.js before the line that mentions static:

app.use('/api', require('./api'));

(It needs to go before the line that mentions static to give preference to the API: if the static/ directory contains – probably by mistake – a directory called api with some files in it, we don't want them to override our API.)

Now the server is ready to return a list of files. Run the app like above with node app (or restart the app if it was still running) and check that the drop-down box of file names gets loaded correctly. It should contain "message" and "pets", the hard-coded list we have in the code.

A hard-coded list of files would not be useful. For now, we will use an in-memory database for our files. Put the following in the file api/db-inmemory.js:

const data = {
  pets: 'kitten, doggie, tortoise',
  message: 'Hello World!',
  shoppinglist: '1. milk\n2. cookies',
};

module.exports.list = () => {
  return Object.keys(data);
};

This keeps the files in an object, keyed by the file name. Object.keys(data) returns an array of the keys, exactly what we need to return to the client.

We have put the database in a new file (api/db-inmemory.js) to separate the implementation of the API from the implementation of the data model.

Now we can use this file in api/index.js by replacing it with:

const express = require('express');
const api = express.Router();
module.exports = api;

const db = require('./db-inmemory');

api.get('/', (req, res) => {
  res.json(db.list());
});

Restart the app and reload the page and the list of files should now contain "shoppinglist" as well as the other files we had in the hard-coded list before.

Listing the available files above is only the first step. To add manipulation of the content of files, first, we will add the following to api/db-inmemory.js:

module.exports.get = (id) => {
  if (data[id] == null) return '';
  return data[id];
};

module.exports.put = (id, val) => {
  data[id] = val;
};

These two new functions need to be used in the API, so we will add this in api/index.js:

api.get('/:id(\\w+)', (req, res) => {
  res.send(db.get(req.params.id));
});

const bodyParser = require('body-parser');
api.put('/:id(\\w+)', bodyParser.text(), (req, res) => {
  db.put(req.params.id, req.body);
  res.sendStatus(204);
});

The first route serves GET requests – retrieving file contents. If a file isn't in the database, we will return an empty string as if the file was there.

The second route serves PUT requests – saving file contents. The route need not return any data, so it returns status 204 (HTTP OK, no content coming back).

The second route uses the body-parser package to get the body of the request. To install it, run npm install body-parser

In the routes, the path '/:id(\\w+)' specifies that the content of the URL becomes a request parameter called id (so we can get it as req.params.id), and that it must match the regex \w+ – only alphanumeric characters.

Restarting the app now, we should see that the Web app works: lists all the files, loads their content, allows saving, and also allows us to create new files.

Because we only have an in-memory database, the app will only remember any saved changes as long as it is not restarted. You can try it by saving a file, restarting the app, and loading the file again – the changes will be lost. The next step addresses this.

Now we can use a persistent database for our files. We will use Google Datastore. If you haven't done so already, you will need to enable Datastore in the Google cloud console under Datastore. Make sure for it to run in Datastore mode and not native.

To be able to access Datastore from our app, we need to install the Node.js package: run npm install @google-cloud/datastore

We can put the database code in a new file, api/db-datastore.js, like this:

const {Datastore} = require('@google-cloud/datastore');
const ds = new Datastore({ namespace: 'tutorial' });

const kind = 'files';

function key(id) {
  return ds.key([kind, id]);
}

module.exports.list = async () => {
  // asynchronously get a list of entities with names
  let [data] = await ds.createQuery(kind).select('name').order('name').run();
  // extract only the names
  data = data.map((val) => val.name);
  return data;
};

module.exports.get = async (id) => {
  // asynchronously get the entity
  const [data] = await ds.get(key(id));
  if (data && data.val) return data.val;
  return '';
};

module.exports.put = async (id, val) => {
  const entity = {
    key: key(id),
    data: { name: id, val },
  }
  await ds.save(entity);
};

Data stored in Datastore is seen as entities. We use one entity per file; each entity has two properties: name and val. Every entity is identified by a key which has a kind and an ID; we named the kind "files" and we use the file name as its ID.

The file uses a namespace (by default "tutorial") that you can rename such that the entities do not collide with any other datastore entities used in your project.

Because the Datastore is remote, all the invocations are asynchronous. We use async/await syntax that is only available from Node.js version 8.

The list() function uses a query to retrieve only the names of all the stored files; and sorts them alphabetically.

The get() function is a straightforward use of the datastore.get() function; similarly, the put() function is a simple use of datastore.save().

When this app runs on a Compute Engine VM, or in App Engine, it automatically connects to the Datastore of the same cloud project, otherwise the first line of the code above would need to specify more about the datastore it uses.

In order to use this Datastore-backed database, we need to change api/index.js in two ways: first, we need to require the right module, and then we need to deal with the asynchronous nature of the datastore (together with any possible errors). Therefore, replace api/index.js with the following:

const express = require('express');
const bodyParser = require('body-parser');
const api = express.Router();
module.exports = api;

const db = require(`./db-datastore`);

api.get('/', async (req, res) => {
  try {
    res.json(await db.list());
  } catch (e) {
    console.error(e);
    res.sendStatus(500);
  }
});

api.get('/:id(\\w+)', async (req, res) => {
  try {
    res.send(await db.get(req.params.id));
  } catch (e) {
    console.error(e);
    res.sendStatus(500);
  }
});

api.put('/:id(\\w+)', bodyParser.text(), async (req, res) => {
  try {
    await db.put(req.params.id, req.body);
    res.sendStatus(204);
  } catch (e) {
    console.error(e);
    res.sendStatus(500);
  }
});

Beside the require statement, the asynchrony, and error handling, not much has changed in that file.

If we now restart the app, we may see that there are no files available yet; when we create and save a file, we can test by restarting the app that the data persists.

You can see the data stored as entities in the Google cloud console under Datastore -> Entities.

We have been testing the app by running it locally on a Compute Engine VM. Now we can deploy it in App Engine so that Google manages the runtime environment, scaling, HTTPS, and so on, for us.

First, we need to prepare an application descriptor file app.yaml:

runtime: nodejs10
service: tutorial

In the file, you can change the service name from "tutorial" to avoid any service name conflicts. If you don't have a service in App Engine yet, the service name must be "default".

Second, we need to make sure App Engine knows how to start the app. Therefore, in the package.json file, replace the unnecessary "test" script line which says "Error: no test specified", with the following line:

"start": "node app.js"

To deploy, run gcloud app deploy – this will take a while.

After the deploy command finishes, your app will be ready and running. You can find out the URL for your app with gcloud app browse -s tutorial or by finding the service in the cloud console App Engine section. If you changed your service name in app.yaml, change it in the browse command above, too.