Interact with your Elasticsearch data

Creating the Smart Collection

Let's take a simple example from Kibana, we will use a set of fictitious accounts with randomly generated data. You can easily import the data using Kibana Home page section Ingest your data.

When it's done we can start looking at how to play with those data in Forest Admin.

forest-express-sequelize

First, we declare the bank-accounts collection in the forest/ directory. In this Smart Collection, all fields are related to document mapping attributes except the field id that is computed using the document _id.

You can check out the list of available field options if you need them.

You MUST declare an id field when creating a Smart Collection. The value of this field for each record MUST be unique. On the following example, we simply use the UUID provided on every Elasticsearch documents.

/forest/bank-accounts.js

const { collection } = require('forest-express-sequelize');

collection('bank-accounts', {
  isSearchable: false,
  fields: [
  {
    field: 'id',
    type: 'string',
  }, {
    field: 'account_number',
    type: 'Number',
    isFilterable: true,
  }, {
    field: 'address',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'firstname',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'lastname',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'age',
    type: 'Number',
    isFilterable: true,
  }, {
    field: 'balance',
    type: 'Number',
    isFilterable: true,
  }, {
    field: 'city',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'employer',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'email',
    type: 'String',
    isFilterable: true,
  }, {
    field: 'gender',
    type: 'Enum',
    isFilterable: true,
    enums: [
      'M',
      'F',
    ],
  }, {
    field: 'state',
    type: 'String',
    isFilterable: true,
  }],
});

You can add the option isSearchable: true to your collection to display the search bar. Note that you will have to implement the search yourself by including it into your own GET logic.

Implementing the routes

It's not an easy job to connect several data sources in the same structure. To accommodate you in this journey we already provide you a simple service ElasticsearchHelper that handles all the logic to connect with your Elasticseearch data.

Before getting further, in order to search your data using filters, we need to define the Elasticsearch configuration.

Name	Type	Description
index	string	The name of your Elasticsearch index.
filterDefinition	string	Type of your Elasticsearch fields. Can be number, date, text,keyword
sort	array of objects	(optional) Required only to sort your data. Elasticsearch documentation Example: [ { createdAt: { order: 'desc' } }]
mappingFunction	function	(optional) Required only to modify the data retrieved from Elasticsearch. Example: (id, source) => { id, ...source}

/routes/bank-accounts.js

const express = require('express');
const router = express.Router();

const {
  RecordSerializer,
  RecordCreator,
  RecordsGetter,
  RecordUpdater,
  PermissionMiddlewareCreator
} = require('forest-express-sequelize');

const ElasticsearchHelper = require('../service/elasticsearch-helper');
const { FIELD_DEFINITIONS } = require('../utils/filter-translator');

const permissionMiddlewareCreator = new PermissionMiddlewareCreator('bank-accounts');

const configuration = {
  index: 'bank-accounts',
  filterDefinition: {
    account_number: FIELD_DEFINITIONS.number,
    address: FIELD_DEFINITIONS.keyword,
    age: FIELD_DEFINITIONS.number,
    balance: FIELD_DEFINITIONS.number,
    city: FIELD_DEFINITIONS.keyword,
    email: FIELD_DEFINITIONS.keyword,
    employer: FIELD_DEFINITIONS.keyword,
    firstname: FIELD_DEFINITIONS.keyword,
    lastname: FIELD_DEFINITIONS.keyword,
    employer: FIELD_DEFINITIONS.keyword,
    state: FIELD_DEFINITIONS.keyword,
    gender: FIELD_DEFINITIONS.keyword,
  },
};

const elasticsearchHelper = new ElasticsearchHelper(configuration);

// Routes implementation

module.exports = router;

Our custom filter translator only support number, keyword, text, date data types. Nonetheless, you can implement more filter mapper type in theutils/filter-translator.js

Implementing the GET (all records)

In the file routes/bank-accounts.js, we’ve created a new route to implement the API behind the Smart Collection.

The logic here is to list all the BankAccount records. We use a custom service service/elasticsearch-helper.js for this example. The implementation code of this service is available here.

Finally, the last step is to serialize the response data in the expected format which is simply a standard JSON API document. You are lucky forest-express-sequelize already does this for you using the RecordSerializer.

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.get('/bank-accounts', async (request, response, next) => {
  try {
    const pageSize = Number(request?.query?.page?.size) || 20;
    const page = Number(request?.query?.page?.number) || 1;
    // search is not handle in this example
    // const search = request.query?.search;
    const options = { timezone: request.query?.timezone };

    let filters;
    try {
      filters = request.query?.filters && JSON.parse(request.query.filters);
    } catch (e) {
      filters = undefined;
    }

    const result = await elasticsearchHelper.functionSearch({
      page,
      pageSize,
      filters: filters || undefined,
      options,
    });

    const serializer = new RecordSerializer({ name: 'bank-accounts' });

    response.send({
      ...await serializer.serialize(result.results),
      meta: {
        count: result.count,
      }
    });
  } catch (e) {
    next(e);
  }
});

module.exports = router;

BankAccount Smart collection with Elasticsearch as a datasource

Our utils provide a variety of filters integration

Implementing the GET (a specific record)

To access the details view of a Smart Collection record, you have to catch the GET API call on a specific record. One more time, we use a custom service that encapsulates the Elasticsearch business logic for this example.

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.get('/bank-accounts/:id', async (request, response, next) => {
  try {
    const bankAccount = await elasticsearchHelper.getRecord(request.params.id);
    const serializer = new RecordSerializer({ name: 'bank-accounts' });

    response.send(await serializer.serialize(bankAccount));
  } catch (e) {
    next(e);
  }
});

module.exports = router;

Implementing the PUT

To handle the update of a record we have to catch the PUT API call.

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.put('/bank-accounts/:id', permissionMiddlewareCreator.update(), (request, response, next) => {
  const updater = new RecordUpdater({ name: 'bank-accounts' }, req.user, req.query)
  
  updater.deserialize(request.body)
    .then(recordToUpdate => elasticsearchHelper.updateRecord(recordToUpdate))
    .then(record => updater.serialize(record))
    .then(recordSerialized => response.send(recordSerialized))
    .catch(next);
});

module.exports = router;

Implementing the DELETE

Now we are able to see all the bank accounts on Forest Admin, it’s time to implement the DELETE HTTP method in order to remove the documents on Elasticsearch when the authorized user needs it.

Delete a list a single record

forest-express-sequelize

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.delete('/bank-accounts/:id', permissionMiddlewareCreator.delete(), async (request, response, next) => {
  try {
    await elasticsearchHelper.removeRecord(request.params.id);
    response.status(204).send();
  } catch (e) {
    next(e);
  }
});

module.exports = router;

Delete a list of records

forest-express-sequelize

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.delete('/bank-accounts', permissionMiddlewareCreator.delete(), async (request, response, next) => {
  const getter = new RecordsGetter({ name: 'bank-accounts' }, request.user, request.query)
  const ids = await getter.getIdsFromRequest(request);
  
  try {
    await elasticsearchHelper.removeRecords(ids);
    response.status(204).send();
  } catch (e) {
    next(e);
  }
});

module.exports = router;

Implementing the POST

To create a record we have to catch the POST API call.

forest-express-sequelize

/routes/bank-accounts.js

// Imports and ElasticsearchHelper base definition ...

router.post('/bank-accounts', permissionMiddlewareCreator.create(), (request, response, next) => {
  const recordCreator = new RecordCreator({ name: 'bank-accounts' }, request.user, request.query);

  recordCreator.deserialize(request.body)
    .then(recordToCreate => elasticsearchHelper.createRecord(recordToCreate))
    .then(record => recordCreator.serialize(record))
    .then(recordSerialized => response.send(recordSerialized))
    .catch(next);
});

module.exports = router;