Skip to main content

Handling file uploads

Some blocks need file upload support, and without an external service for handling these uploads, the files are stored as Base64-encoded data in the dataset. That's not ideal, as the dataset can become very large. To solve this, the stock runners support the use of a separate file upload service. This service handles the file uploads and stores the files. Only a reference to the file within the file upload service is then stored in the dataset.

📁 Implementation​

To implement a separate store for the file uploads, you need to set up a web service able to handle the following three requests from the runner:

  • put: Upload an attachment to the store;
  • get: Fetch (download) an attachment from the store;
  • delete: Remove an attachment from the store.

The stock runners use the put request when uploading a file. This put operation should return an identification or reference string for the uploaded file. This string is what's get stored in the dataset of the form. When the runner wants to download the uploaded file (for example, for generating a thumbnail preview for uploaded images), the get request is invoked using the identification string. When the user removes an uploaded file (or replaces an uploaded file with another one), the delete request is invoked by the runner.

The attachments property of the runner is used to connect the upload service to the runner. See the example below to see how to do this. It should implement the IRunnerAttachments interface.

tip

The put request also supports a feedback function to inform the runner about the file upload progress.

✅ Confirming file uploads​

Since the files are uploaded before the form itself completes (and received/processed by an endpoint), you should implement a mechanism to handle these "temporary" uploads. The easiest way to do this is to accept any file upload and delete the uploaded files if they are not confirmed within a specific expiration time. Confirmation of uploaded files should occur when the form completes and the form data is submitted to an endpoint. The endpoint receives the form data and then performs a confirmation request for any uploaded files in the dataset. That's the most secure way, as clients cannot confirm files and make them permanent. Only your endpoint can do this.

tip

When confirming files, you could generate delete tokens for each uploaded file and store them with the form data. When the form data is removed from the endpoint, the endpoint can use the delete tokens to remove any file uploads related to the form data being removed.

info

If you have implemented form pausing and resuming, you should also keep uploaded files for paused forms. We recommend specifying an expiration time for paused forms and then applying the same expiration time to uploaded files. This requires an additional step when pausing a form, which is to mark the uploaded files in the file service as paused. This request can be made from the endpoint that processes the paused form data. It would extend the normal expiration time for unconfirmed files to match the expiration time of the paused form. When a paused form resumes, the uploaded files are automatically fetched by the runner (if necessary) using the get request.

🗑️ Removing file uploads​

The file upload service should also be able to remove uploaded files. There are three scenarios for removing file uploads:

  1. A form respondent removes an earlier uploaded file (or replaces an uploaded file with another one) while filling out a form;
  2. An uploaded file is not confirmed within the specified timeframe;
  3. Response data that includes uploaded files is removed.

For scenario 1, the runner performs a delete request whenever the user wants to remove an earlier uploaded file. The runner will include the file identifier in the request so the file upload service knows which file to remove.

For scenario 2, there needs to be a job that runs at regular intervals and checks unconfirmed files that are expired. Those files can be removed from the file store.

Scenario 3 requires the endpoint that contains the form data to make a delete request to the file upload service when form data that contains file uploads is deleted.

👩‍💻 Example​

The Tripetto Studio application has such a separate web service for handling file uploads. This web service is open-source and can be used as a reference implementation using Node.js. You can find the source code here:

▶️ Filestore webservice

Code example​

The code below shows how the filestore web service is used in the runner. It uses SuperAgent to perform the HTTP calls.

import { run } from "@tripetto/runner-autoscroll";
import * as Superagent from "superagent";

run({
  definition: /* Supply your form definition here */,
  attachments: {
    // Handle file uploads
    put: (file: File, onProgress?: (percentage: number) => void) =>
      new Promise<string>(
        (resolve: (id: string) => void, reject: (reason?: string) => void) => {
          const formData = new FormData();

          formData.append("file", file);

          Superagent.post("https://url-to-filestore/upload")
            .send(formData)
            .then((response: Superagent.Response) => {
              if (response.ok && res.body?.id) {
                // Report the file identifier back to the runner.
                // This identifier is what's stored in the form dataset
                return resolve(response.body.id);
              } else {
                return reject(
                  response.status === 413 ? "File is too large." : undefined
                );
              }
            })
            // Report progress back to the runner
            .on("progress", (event: Superagent.ProgressEvent) => {
              if (onProgress && event.direction === "upload") {
                onProgress(event.percent || 0);
              }
            })
            .catch(() => reject());
        }
      ),
    // Handle file downloads
    get: (id: string) =>
      new Promise<Blob>((resolve: (data: Blob) => void, reject: () => void) => {
        Superagent.get(`https://url-to-filestore/get/${id}`)
          .responseType("blob")
          .then((response: Superagent.Response) => {
            if (response.ok) {
              resolve(response.body);
            } else {
              reject();
            }
          })
          .catch(() => reject());
      }),
    // Handle file removals
    delete: (id: string) =>
      new Promise<void>((resolve: () => void, reject: () => void) => {
        Superagent.delete("https://url-to-filestore/delete")
          .send({ id })
          .then((response: Superagent.Response) => {
            if (response.ok) {
              return resolve();
            } else {
              return reject();
            }
          })
          .catch(() => reject());
      }),
  },
});

📦 Supported blocks​

The following blocks support file uploads:

📖 Reference​

Have a look at the complete autoscroll runner API reference for detailed documentation. In the examples above, the following symbols were used: