There are two types of assets:
- Files that are uploaded
- URLs (URL Assets)
We will cover uploading “File” assets here. For URL assets view this API page.
Uploading files
When uploading files, they go directly to the S3 server (or the service that implements the S3 API, e.g. Minio, DigitalOcean Spaces, Linode Object Storage, etc.)
Firstly, we need to send the metadata about the file to the app. In the response will be signed URLs to allow sending the file in chunks directly to the S3.
Once the file has been uploaded, we will need to notify the app that the file has been completed and pass back the ID for each chunk that was uploaded.
File size limits
There is no file size limit when uploading. The only limit is on your plan’s allocated storage. If the file is larger than your plan’s allocation, you will receive a 402 error.
For more storage space, you’ll have to delete files or upgrade your plan.
Step 1: Sending the metadata
Send a POST request with metadata.
POST https://app.digital-downloads.com/api/v1/assets/signed
{
"name": "MyAmazingFile.zip",
"size": 303287472, // filesize in bytes
"mime": "application/zip"
}
Response
{
"chunk_size": 100000000,
"upload_id": "56f30001-2548-489b-be0b-068be4b8102d",
"urls": [
{
"start": 0,
"end": 100000000,
"part": 1,
"url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=1&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=5145422d9dcfef6374eb4939619735505c3c6e2a171945f917f74294812ec6e3"
},
{
"start": 100000000,
"end": 200000000,
"part": 2,
"url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=2&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=4f56dbbebc3002a8df0037f5a9192d00490179e23af4f0e72894dd61e93c87ea"
},
{
"start": 200000000,
"end": 300000000,
"part": 3,
"url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=3&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=447af2a5eef971882a2028fcfd6c1945b37b8c76dd2edd4d3be2def08abe1447"
},
{
"start": 300000000,
"end": 303287472,
"part": 4,
"url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=4&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=485b0bee23986b5add37dfaf0422a95e897949789f0ec64c0c880237f39ccf2b"
}
],
"id": "e784c8cb-7192-4c6c-9004-17fe31a859fc",
"file_url": "https://app.digital-downloads.com/assets/e784c8cb-7192-4c6c-9004-17fe31a859fc"
}
The URLs that are generated are chunked into 100MB parts, but this could change at any time. The chunk size is provided in the payload.
Using the URLs, send a PUT request to each with the section of the file using the start and end bytes. These chunks do not need to be uploaded in order. Wait until the previous part has finished.
Signed URLs expiry
The S3 signed URL are only valid for 12 hours.
Step 2: Uploading file parts to the S3
Send a PUT request with a chunk of the file. See the JavaScript example.
for (let part of urls) {
// get only the part of the file that is required
let blob = file.slice(part.start, part.end);
// create a new http request and remove the content type
const a = axios.create();
delete a.defaults.headers.put['Content-Type'];
a.put(part.url, blob).then((response) => {
// from the header we need to etag, this is S3's id for each part
// of file so we can re construct it when all pieces are uploaded
return {
ETag: response.headers.etag.split('"').join(''),
PartNumber: part.part,
};
});
}
Step 3: Notifying the app that the file has been uploaded
Once each chunk has been uploaded to the S3, send a POST request with each chunk’s details back to the app.
POST https://app.digital-downloads.com/api/v1/assets/:id/uploaded
{
"parts":[
{
"ETag": "0d5bf6a8d41ed4d3d8813fa27ee67b6f",
"PartNumber": 1
},
{
"ETag": "4f0d43ac4cd8dc4ec545572c44b8642b",
"PartNumber": 2
},
{
"ETag": "00dbc7cf67bef6ae9eb5c47936f1f289",
"PartNumber": 3
},
{
"ETag": "dd2b66242dba17ac0b7e33b7a70e716b",
"PartNumber": 4
}
],
"upload_id": "56f30001-2548-489b-be0b-068be4b8102d"
}
JavaScript example of the flow
This is an example of the code used within the app’s UI to upload files.
async function upload(file) {
// generate the signed urls
let signedResponse = await axios.post('https://app.digital-downloads.com/api/v1/assets/signed', {
name: file.name,
size: file.size,
mime: file.mime,
}).then((r) => r.data);
// create a new http request and remove the content type
const a = axios.create();
delete a.defaults.headers.put['Content-Type'];
const promises = [];
for (let part of signedResponse.urls) {
// get the part of the file to send in this request
let blob = file.file.slice(part.start, part.end);
promises.push(
// send the PUT request to the url with part of the file
a.put(part.url, blob).then((response) => {
// from the header we need to etag, this is S3's id for each part
// of file so we can re construct it when all pieces are uploaded
return {
ETag: response.headers.etag.split('"').join(''),
PartNumber: part.part,
};
})
);
}
// once all parts have uploaded then we need to send
// the parts with the joined S3 etag back to the app
Promise.all(promises).then((parts) => {
axios.put(`https://app.digital-downloads.com/api/v1/assets/${signedResponse.id}/uploaded`, {
parts: parts,
upload_id: signedResponse.upload_id
}).then((r) => {
// completed
});
});
}
Cancelling an upload
If a chunk fails, or if something goes wrong, or you want to cancel the upload, we need to remove the chunks that got uploaded but are no longer needed. Send a POST request to the asset with the upload ID (received from the signed request).
POST https://app.digital-downloads.com/api/v1/assets/:id/cancel
{
"upload_id": "56f30001-2548-489b-be0b-068be4b8102d"
}