File upload

[pwalsh]

People who submit service requests also need to optionally be able to submit images, to show the thing they are requesting to be addressed.

There are many ways to provide image support in the application: different ways to use cloud object storage, or upload to a filesystem, etc.

An ideal solution probably looks something like:

• in browser, the user selects to add one or many images to their service request
• the client code reads over the files, gets a hash for each, and then
• the client code hits an API endpoint and gets an upload URL for each file (verified by the hash)
• the client code initiates the upload of each file
• the client code handles a callback for each file that the upload is complete, and gets with that a URL to the file (the URL will not be publicly accessible, it will only be accessible to admin users of the app)
• the client submits the form, with the images being an array of URLs

This pattern, or a variation of it, is pretty straight forward for file system storage, or AWS S3, and also I think minIO. Need to check if this pattern translates to Azure Blob Storage as we do want to support that (but we can also assume a minIO gateway over Azure Blob Storage as an option).

After some discussion with @amirozer it looks like the simplest, first solution will be:

• The client code does what it does to upload images and get back a URL that the image will be available to admin users at
• The client submits an array of URLs for images

This solution works probably works in the short term as the current client runs in a privileged environment with access to non-public image upload APIs.

[pwalsh]

We are going to do:

• The client code does what it does to upload images and get back a URL that the image will be available to admin users
• The client submits an array of URLs for images
• The database stores these URLs, and assumes that they are accessible to StaffUsers and system admins

Will explore other generalizations later.

[pwalsh]

@amirozer @idoivri I think what we have written above from Oct is still totally valid as a high-level approach - it keeps all image handling quite distinct from the core of Gov Flow (it could be implemented as a completely separate micro service, but I suggest not at this stage, explained below), supports the storage of images as public or private (but we should do so 100% private for now), and works well as a general pattern over various cloud provider object storage solutions (or, just a file system).

So, a bit more technical detail:

• We write a client side library (or glue together an existing one), that:
  • takes user-submitted files
  • generates some info about them (like a hash of the file)
  • limits their size (configurable, default to say 2MB)
  • Hits an Api we control (in future a microservice perhaps, but for now a decoupled route on Gov Flow like /integrations/sign-image) to sign an image (described below) which essentially, just give the client back a time-limited URL that the file can be uploaded to
  • After the file is uploaded to the time-limited URL, gets the hosted URL back (also we can use callbacks to check upload progress etc) which is the URL the file is actually hosted at
  • Saves the URL(s) for the file(s) as part of the service request submission

This requires the following on the server:

• An endpoint to sign images for known clients and give the known clients upload URLs for those images
• This endpoint basically is exposing an API to S3's pre-signed POST request functionality, which many S3-compatible object storages support
• Credentials for an S3-compatible bucket at a cloud provider

If we need to use Azure Blob Storage (which is not S3 compatible), we can run a minio service (here is a link to some internal notes I made about that). The great thing about using S3 API is that we are basically using a standard approach and don't need to make the open source codebase support multiple object storages, and, people can always use Minio to get the API, which is a well adopted approach.

We can write the file handling ourselves using the File API or use a full-featured existing lib like Filepond for the management and manipulation of the files themselves.

Here is a simple article I just dug up that gives an overview of this type of approach: https://blog.rocketinsights.com/uploading-images-to-s3-via-the-browser/

A benefit of this approach is that it can quite easily be abstracted to a microservice that you can use at Zen City generally.

[idoivri]

image privacy: for sure things should be private
client side: we need to limit to image MIME types, yes? not sure if we need to protect against bad actors here or not (I guess we should?)
size limiting: we need to make sure the size limit still allows mobile uploads.

General like using full-featured existing libs over reinventing the wheel...

[pwalsh]

@idoivri @amirozer

We have an implementation in #45 which provides all the server support required, and we have two currently closed source code bases that implement the frontend handling of this, for (i) public form submissions and (ii) viewing public form submissions in the admin UI.

We recently had a request to implement image support for comments on service requests - essentially for internal use as assignees within a given government department work on resolving requests.

The following changes need to be done to support the expanded functionality:

1. Add an images field to ServiceRequestComment with the same implementation as the image field on ServiceRequest (We could consider renaming ServiceRequestComment to capture the meaning of comments and/or images)
2. Update the Admin UI to support the new images field on ServicdRequestComment

The following change is highly recommended:

We have two react components - one for putting an image and one for getting an image, abstracting away the handling of the signing of requests with the object storage. Until now, the public form uses the put component and the admin uses the get component, but now the admin also needs to use the put component, so I suggest we pull these components into a library for use in both admin and public form codebases.

[pwalsh]

also need to add a person to the comment entity - the person who commented.

[pwalsh]

#45 was updated to provide all the above

[pwalsh]

This is now implemented.