Upload to S3 From App With Proxy

In web and mobile applications, it's common to provide users with the ability to upload data. Your awarding may let users to upload PDFs and documents, or media such as photos or videos. Every modern web server technology has mechanisms to permit this functionality. Typically, in the server-based environment, the procedure follows this menses:

1. The user uploads the file to the awarding server.
2. The awarding server saves the upload to a temporary space for processing.
3. The application transfers the file to a database, file server, or object store for persistent storage.

While the process is unproblematic, it tin can accept significant side-effects on the performance of the spider web-server in busier applications. Media uploads are typically large, so transferring these can represent a big share of network I/O and server CPU fourth dimension. You must likewise manage the state of the transfer to ensure that the entire object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For example, in a spider web application that specializes in sending vacation greetings, it may feel most traffic only effectually holidays. If thousands of users try to upload media around the same fourth dimension, this requires you to scale out the application server and ensure that at that place is sufficient network bandwidth available.

By direct uploading these files to Amazon S3, y'all can avoid proxying these requests through your application server. This tin significantly reduce network traffic and server CPU usage, and enable your application server to handle other requests during busy periods. S3 too is highly available and durable, making it an platonic persistent store for user uploads.

In this blog mail service, I walk through how to implement serverless uploads and show the benefits of this approach. This pattern is used in the Happy Path web application. You can download the code from this weblog post in this GitHub repo.

Overview of serverless uploading to S3

When yous upload direct to an S3 bucket, you lot must showtime asking a signed URL from the Amazon S3 service. You can and so upload directly using the signed URL. This is two-step process for your application forepart end:

1. Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda office. This gets a signed URL from the S3 bucket.
2. Directly upload the file from the application to the S3 saucepan.

To deploy the S3 uploader example in your AWS account:

1. Navigate to the S3 uploader repo and install the prerequisites listed in the README.doctor.
2. In a final window, run:
   git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided

3. At the prompts, enter s3uploader for Stack Name and select your preferred Region. In one case the deployment is consummate, notation the APIendpoint output.The API endpoint value is the base URL. The upload URL is the API endpoint with /uploads appended. For instance: https://ab123345677.execute-api.us-west-2.amazonaws.com/uploads.

Testing the awarding

I show two ways to test this application. The first is with Postman, which allows you to directly call the API and upload a binary file with the signed URL. The second is with a basic frontend awarding that demonstrates how to integrate the API.

To test using Postman:

1. First, copy the API endpoint from the output of the deployment.
2. In the Postman interface, paste the API endpoint into the box labeled Enter request URL.
3. Choose Send.
4. Afterwards the request is complete, the Body section shows a JSON response. The uploadURL attribute contains the signed URL. Copy this aspect to the clipboard.
5. Select the + icon next to the tabs to create a new asking.
6. Using the dropdown, change the method from Go to PUT. Paste the URL into the Enter request URL box.
7. Choose the Body tab, then the binary radio button.
8. Choose Select file and choose a JPG file to upload.
   Cull Send. Yous see a 200 OK response later the file is uploaded.
9. Navigate to the S3 console, and open the S3 bucket created past the deployment. In the bucket, you come across the JPG file uploaded via Postman.

To examination with the sample frontend awarding:

1. Copy index.html from the example's repo to an S3 saucepan.
2. Update the object'south permissions to make information technology publicly readable.
3. In a browser, navigate to the public URL of index.html file.
4. Select Choose file and then select a JPG file to upload in the file picker. Choose Upload image. When the upload completes, a confirmation message is displayed.
5. Navigate to the S3 console, and open the S3 bucket created by the deployment. In the saucepan, y'all see the second JPG file you uploaded from the browser.

Understanding the S3 uploading process

When uploading objects to S3 from a web application, you must configure S3 for Cross-Origin Resources Sharing (CORS). CORS rules are defined equally an XML document on the saucepan. Using AWS SAM, you tin configure CORS equally part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Blazon: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - Head           AllowedOrigins:             - "*"                  

The preceding policy allows all headers and origins – it's recommended that you use a more restrictive policy for production workloads.

In the offset step of the process, the API endpoint invokes the Lambda role to make the signed URL request. The Lambda role contains the post-obit code:

          const AWS = require('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Chief Lambda entry bespeak exports.handler = async (event) => {   return expect getUploadURL(outcome) }  const getUploadURL = async function(effect) {   const randomID = parseInt(Math.random() * 10000000)   const Key = `${randomID}.jpg`    // Get signed URL from S3   const s3Params = {     Saucepan: process.env.UploadBucket,     Key,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'paradigm/jpeg'   }   const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)   render JSON.stringify({     uploadURL: uploadURL,     Cardinal   }) }                  

This function determines the proper noun, or central, of the uploaded object, using a random number. The s3Params object defines the accepted content type and also specifies the expiration of the fundamental. In this instance, the cardinal is valid for 300 seconds. The signed URL is returned as office of a JSON object including the key for the calling application.

The signed URL contains a security token with permissions to upload this single object to this saucepan. To successfully generate this token, the code calling getSignedUrlPromise must have s3:putObject permissions for the bucket. This Lambda office is granted the S3WritePolicy policy to the bucket by the AWS SAM template.

The uploaded object must friction match the same file proper noun and content type equally defined in the parameters. An object matching the parameters may exist uploaded multiple times, providing that the upload procedure starts before the token expires. The default expiration is fifteen minutes only yous may want to specify shorter expirations depending upon your use case.

Once the frontend application receives the API endpoint response, it has the signed URL. The frontend awarding then uses the PUT method to upload binary data direct to the signed URL:

          permit blobData = new Blob([new Uint8Array(array)], {type: 'image/jpeg'}) const result = await fetch(signedURL, {   method: 'PUT',   body: blobData })                  

At this bespeak, the caller application is interacting directly with the S3 service and non with your API endpoint or Lambda function. S3 returns a 200 HTML condition lawmaking once the upload is complete.

For applications expecting a large number of user uploads, this provides a simple way to offload a big corporeality of network traffic to S3, abroad from your backend infrastructure.

Adding authentication to the upload process

The current API endpoint is open, bachelor to any service on the internet. This means that anyone can upload a JPG file in one case they receive the signed URL. In most production systems, developers want to apply authentication to command who has admission to the API, and who can upload files to your S3 buckets.

You lot tin restrict admission to this API by using an authorizer. This sample uses HTTP APIs, which back up JWT authorizers. This allows you to command admission to the API via an identity provider, which could be a service such every bit Amazon Cognito or Auth0.

The Happy Path application only allows signed-in users to upload files, using Auth0 as the identity provider. The sample repo contains a second AWS SAM template, templateWithAuth.yaml, which shows how yous can add an authorizer to the API:

                      MyApi:     Type: AWS::Serverless::HttpApi     Properties:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audience:                 - https://auth0-jwt-authorizer             IdentitySource: "$request.header.Potency"         DefaultAuthorizer: MyAuthorizer                  

Both the issuer and audience attributes are provided by the Auth0 configuration. Past specifying this authorizer as the default authorizer, information technology is used automatically for all routes using this API. Read function 1 of the Ask Around Me series to learn more than about configuring Auth0 and authorizers with HTTP APIs.

After authentication is added, the calling web application provides a JWT token in the headers of the asking:

          const response = wait axios.go(API_ENDPOINT_URL, {   headers: {     Authorization: `Bearer ${token}`         } })                  

API Gateway evaluates this token before invoking the getUploadURL Lambda function. This ensures that only authenticated users can upload objects to the S3 bucket.

Modifying ACLs and creating publicly readable objects

In the current implementation, the uploaded object is non publicly accessible. To make an uploaded object publicly readable, you must fix its access control list (ACL). There are preconfigured ACLs available in S3, including a public-read option, which makes an object readable by anyone on the cyberspace. Set up the appropriate ACL in the params object earlier calling s3.getSignedUrl:

          const s3Params = {   Bucket: procedure.env.UploadBucket,   Key,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'image/jpeg',   ACL: 'public-read' }                  

Since the Lambda function must have the advisable bucket permissions to sign the request, you must too ensure that the function has PutObjectAcl permission. In AWS SAM, y'all tin can add the permission to the Lambda role with this policy:

                      - Statement:           - Effect: Let             Resource: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Action:               - s3:putObjectAcl                  

Conclusion

Many web and mobile applications permit users to upload data, including large media files like images and videos. In a traditional server-based application, this can create heavy load on the application server, and besides use a considerable amount of network bandwidth.

By enabling users to upload files to Amazon S3, this serverless pattern moves the network load abroad from your service. This can make your application much more than scalable, and capable of handling spiky traffic.

This weblog post walks through a sample awarding repo and explains the process for retrieving a signed URL from S3. It explains how to the test the URLs in both Postman and in a web application. Finally, I explain how to add hallmark and brand uploaded objects publicly accessible.

To learn more, see this video walkthrough that shows how to upload directly to S3 from a frontend spider web application. For more serverless learning resources, visit https://serverlessland.com.

tookesackled.blogspot.com

Source: https://aws.amazon.com/blogs/compute/uploading-to-amazon-s3-directly-from-a-web-or-mobile-application/

Share this post