Add fields

This is the official documentation of the forestadmin-agent-django and forestadmin-agent-flask Python agents.

Forest Admin allows creating new Fields on any Collection, either computationally, by fetching data on an external API or based on other data that is available on the connected data sources.

How does it work?

When creating a new field you will need to provide:

Field	Description
column_type	Type of the new field which can be any primitive or composite type
dependencies	List of fields that you need from the source records and linked records in order to run the handler
get_values	Handler which computes the new value for a batch of records
enum_values (optional)	When columnType is Enum, you must specify the values that the field will support

Examples

Adding a field by concatenating other fields

This example adds a user.displayName field, which is computed by concatenating the first and last names.

from typing import Any, List

from forestadmin.datasource_toolkit.context.collection_context import (
    CollectionCustomizationContext
)
from forestadmin.datasource_toolkit.interfaces.records import RecordsDataAlias

def get_display_name(
    records:List[RecordsDataAlias], context: CollectionCustomizationContext
) -> List[Any]:
    return [f"{record['firstName']} {record['lastName']}" for record in records]

# "User" Collection has the following structure: { id, firstName, lastName }
agent.customize_collection("user").add_field(
    "displayName",
    {
        "column_type": "String",
        "dependencies": ["firstName", "lastName"],
        "get_values": get_display_name,
    },
)

Adding a field that depends on another computed field

This example adds a user.displayName field, which is computed by concatenating the first and last names, and then another which capitalize it.

from typing import Any, List

from forestadmin.datasource_toolkit.context.collection_context import (
    CollectionCustomizationContext
)
from forestadmin.datasource_toolkit.interfaces.records import RecordsDataAlias

def get_display_name(
    records:List[RecordsDataAlias], context: CollectionCustomizationContext
) -> List[Any]:
    return [f"{record['firstName']} {record['lastName']}" for record in records]

# "User" Collection has the following structure: { id, firstName, lastName }
agent.customize_collection("user").add_field(
  # Create a first field which is computed by concatenating the first and last names
    "displayName",
    {
        "column_type": "String",
        "dependencies": ["firstName", "lastName"],
        "get_values": get_display_name,
    },
).add_field(
    "displayNameCaps",
    {
        "column_type": "String",
        "dependencies": ["displayName"],
        "get_values": lambda records, context: [
            record["displayName"].upper() for record in records
        ],
    },
)

Adding a field that depends on a many-to-one relationship

We can improve the previous example by adding the city of the user to the display name.

# Structure:
# User    { id, addressId, firstName, lastName }
# Address { id, city }
agent.customize_collection("user").add_field(
    "displayName",
    {
        "column_type": "String",
        # We added 'address:city' in the list of dependencies,
        # which tells forest to fetch the related record
        "dependencies":["firstName", "lastName", "address:city"],
        "get_values": lambda records, context: [
            f"{record['firstName']} {record['lastName']} (from {record['customer']['city']})"
            for record in records
      ],
    },
)

Adding a field that depends on a one-to-many relationship

Let's now add a user.totalSpending field by summing the amount of all orders.

from typing import Any, List
from forestadmin.datasource_toolkit.context.collection_context import CollectionCustomizationContext
from forestadmin.datasource_toolkit.interfaces.records import RecordsDataAlias
from forestadmin.datasource_toolkit.interfaces.query.aggregation import Aggregation,
from forestadmin.datasource_toolkit.interfaces.query.condition_tree.nodes.leaf import ConditionTreeLeaf

# Structure
# Customer  { id }
# Order { id, customerId, amount }
async def get_user_spending_values(
    records: List[RecordsDataAlias], context: CollectionCustomizationContext
) -> List[Any]:
    record_ids = [record["id"] for record in records]
    condition = {
        "conditionTree": [ConditionTreeLeaf("customer_id", "in", record_ids)]
    }
    aggregation = Aggregation(
        {"operation": "Sum", "field": "amount", "groups": [{"field": "customer_id"}]},
    )

    # We're using Forest Admin's query interface (you can use an ORM or a plain SQL query)
    rows = await context.datasource.get_collection("order").aggregate(
        condition, aggregation
    )
    ret = []
    for record in records:
        filtered = list(
            filter(
                lambda row: row["group"]["customer_id"] == record["id"], rows
            )
        )
        row = filtered[0] if len(filtered) == 1 else {}
        ret.append(row.get("value", 0))
    return ret

agent.customize_collection("Customer").add_field(
    "TotalSpending",
    {
        "column_type": "Number",
        "dependencies": ["id"],
        "get_values": get_user_spending_values
    },
)

Adding a field fetching data from an API

Let's imagine that we want to check if the email address of our users is deliverable. We can use a verification API to perform that work.

The API we're using is fictional, and the structure of the response is:

{
  "[email protected]": {
    "usernameChecked": false,
    "usernameValid": null,
    "domainValid": true
  },
  "[email protected]": {
    "usernameChecked": false,
    "usernameValid": null,
    "domainValid": true
  }
}

from typing import Any, List

from forestadmin.datasource_toolkit.context.collection_context import (
    CollectionCustomizationContext
)
from forestadmin.datasource_toolkit.interfaces.records import RecordsDataAlias
from fake import EmailVerificationClient

client = EmailVerificationClient()
client.set_api_key("MY_FAKE_API_KEY")

# "User" Collection has the following structure: { id, email }
async def get_user_spending_values(
    records: List[RecordsDataAlias], context: CollectionCustomizationContext
) -> List[Any]:
    emails = [record["email"] for record in records]
    response = client.verifyEmails(emails)
    # Always return values in the same order than the source records
    ret = []
    for record in records:
        check = response[record["email"]]
        ret.append(
            check["domainValid"]
            and (not check["usernameChecked"] or check["usernameValid"])
        )
    return ret

agent.customize_collection("User").add_field(
    "TotalSpending",
    {
        "column_type":"Boolean",
        "dependencies":["email"],
        "get_values":get_user_spending_values
    },
)

Performance

When adding many fields, keep in mind that:

• You should refrain from making queries to external services

  • Use relationships in the dependencies array when that is possible

  • Use batch APIs calls instead of performing requests one by one inside of the records.map handler.

• Only add fields you need in the dependencies list

  • This will reduce the pressure on your data sources (fewer columns to fetch)

  • And increase the probability of reducing the number of records that will be passed to your handler (records are deduplicated).

• Do not duplicate code between handlers of different fields: fields can depend on each other (no cycles allowed).