Direct file uploads

Direct file uploads allow you to stream the incoming multipart streams to a cloud service like s3 or cloudinary without processing them on your server. The flow looks as follows:

Since you pipe the stream directly, your AdonisJS application does not have to allocate any additional memory or CPU computation to parse and persist the data on a disk.

When not to use direct file uploads?

As you will notice later in this guide, direct file uploads are complex as you deal with the streams directly.

We recommend sticking to standard file uploads if your application does not deal with big file uploads. However, sometimes writing simpler code wins over small performance gains.

Usage

The first step is to disable the autoprocessing of files inside the config/bodyparser.ts file. Once autoprocessing is disabled, the bodyparser middleware will forward the multipart stream to your controller so that you can process it manually.

You can disable the autoprocessing for the entire application by setting the autoProcess property to false.

multipart: {
autoProcess: false
}

Or, you can disable it for selected routes by adding their route pattern to the processManually array.

processManually: ['/drive']

Handling the multipart stream

You can handle the multipart stream inside your controller as follows:

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
export default class DriveController {
public async store({ request }: HttpContextContract) {
request.multipart.onFile('input_field_name', {}, (part) => {
someSdk.uploadStream(part)
})
await request.multipart.process()
}
}

Access the processed stream file

Once the stream for a given file has been processed (successfully or with errors), you can access it using the request.file method. For example:

request.multipart.onFile('input_field_name', {}, (part) => {
someSdk.uploadStream(part)
})
await request.multipart.process()
const file = request.input('input_field_name')
if (file.hasErrors) {
return file.errors
}

Validating the stream

You can also validate the stream as it is written to a destination by reporting every chunk to a helper function passed as the second argument to the onFile callback.

request.multipart.onFile(
'input_field_name',
{
extnames: ['pdf', 'jpg', 'png', 'doc', 'xls'],
size: '200mb',
},
(part, reportChunk) => {
part.pause()
part.on('data', reportChunk)
someSdk.uploadStream(part)
})

Have you noticed the part.pause statement?

You have to pause the stream before defining the part.on('data') event listener. Otherwise, the stream will start flowing data before your SDK is ready to consume it.

Error handling

Any errors that occurred within the onFile callback are added to the file instance, and you can access them as follows.

request.multipart.onFile('input_field_name', {}, (part) => {
throw new Error('blow up the stream')
})
await request.multipart.process()
const file = request.input('input_field_name')
console.log(file.errors) // will contain the "blow up the stream"

Attach meta-data to the processed stream

You can attach meta-data to the processed stream file returning an object from the onFile callback. For example, it can be an object holding the URL of the file uploaded to a cloud service.

request.multipart.onFile('input_field_name', {}, (part, reportChunk) => {
part.pause()
part.on('data', reportChunk)
const url = await someSdk.uploadStream(part)
return { url }
})

The url will be available on the file.meta property.

await request.multipart.process()
const file = request.input('input_field_name')
console.log(file.meta) // { url: '...' }

Caveats

When working with the stream directly, you cannot access the form input fields before the entire stream has been processed. This is because the form fields and files are both parts of a single stream, and hence they are available only when the stream is processed.

❌ Incorrect

request.multipart.onFile('input_field_name', {}, (part) => {
// May or may not be available, based upon the position of field
// in the stream
request.input('some_field')
})
await request.multipart.process()

✅ Correct

request.multipart.onFile('input_field_name', {}, (part) => {
})
await request.multipart.process()
// Access after the process method
request.input('some_field')

How is it different from AWS direct uploads?

AWS allows direct file uploads directly from the browser, without even hitting your server.

AdonisJS direct uploads are an alternative to AWS direct uploads, but both approaches have their own up and downsides, as listed below.

AWS direct uploads

AdonisJS direct uploads