This article describes the method of automating the conversion of .wav or any type of media (Audio or Video) files to any other media file formats. Normally if we want to convert any media file to some other format, we can use Elastic Transcoder to do the process manually. In that case, only 1 file can be converted at a time that too by manual selection of the file. Here we are automatic that procedure so that whenever a new .wav file is uploaded to a specific bucket in S3, that files will get converted into the required format within seconds, automatically. The AWS services used in this process are S3, Lambda, Elastic Transcoder & SNS notification. In this scenario, an AWS S3 bucket is hosting a WordPress website where the website admin uploads .wav files via WordPress. The user needs the .wav files to be automatically converted .mp3 and get stored in the same S3 bucket, in a different folder.

A short introduction to the AWS services used.

What is S3?

Amazon Simple Storage Service (S3)is the storage for the Internet. Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Think of it as an online storage with infinite storage capability.

What is AWS Lambda?

AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second. With AWS Lambda, you can run code for virtually any type of application or backend service – all with zero administration. All you need to do is supply your code in one of the languages that AWS Lambda supports (currently Node.js, Java, C#, Go and Python). Also you can use AWS Lambda to run your code in response to events, such as changes to data in an Amazon S3 bucket, which is the case here in this example or an Amazon DynamoDB table; to run your code in response to HTTP requests using Amazon API Gateway; or invoke your code using API calls made using AWS SDKs.

What is AWS Elastic Transcoder?

Amazon Elastic Transcoder lets you convert media files that you have stored in Amazon S3 into media files in the formats required by consumer playback devices. It takes the files from the S3, converts it to the desired format and saves the new format file back to S3 while leaving the original file intact. The components of Elastic Transcoder are
1. Pipelines – Queues that manage jobs
2. Jobs – The conversion task which has to be performed
3. Presets – Templates that specify the settings for the transcoded media file

What is SNS?

Amazon Simple Notification Service (Amazon SNS) is a fast, flexible, fully managed push messaging service. Amazon SNS makes it simple and cost-effective to push to mobile devices such as iPhone, iPad, Android, Kindle Fire, and Internet-connected smart devices, as well as pushing to other distributed services. Besides pushing cloud notifications directly to mobile devices, Amazon SNS can also deliver notifications by SMS text message or email, to Amazon Simple Queue Service (SQS) queues, or to any HTTP endpoint.

The high-level architecture is as below.

LET THE CODING BEGIN

The S3 bucket

Log in to the AWS console. If the S3 is where the WordPress website is hosted, in the S3 bucket there will be a folder named content\uploads where all the uploaded .wav files go. (There could be many more folders inside uploads folder as well). We shall create another folder named converted inside the bucket where “content” folder is located and this is where the converted .mp3 files will be saved.

WordPress upload folder for .wav files

New folder for converted .mp3 files

Setup the notifications

SNS notifications are not important for this process to work, however, it gives confirmation mail whenever a file is processed else not it will send out a failure mail. To set it up, we need to create 2 topics named Success and Failed from the Services>Simple Notification Service.

Next is to create a subscription to your email id so that you will receive a mail upon each successful and unsuccessful transcode process.

CREATE SUBSCRIPTION

The ARN has to be copied from the topic and the endpoint can be your email id to which the notification of the task has to be received.

Once the two topics for Success & Failed are set up, please go to your mail ID and click on the confirmation links to remove the PendingConfirmation status.

Setup the Elastic Transcoder

It’s time to set up the service that does the actual media format conversion job.

Start with creating a new pipeline

Create the pipeline and set the notifications as shown in the below screenshot.

After creating the pipeline, take a note of the pipeline ID.

Set access via IAM

We need to create an IAM role for LAMBDA so that it can access S3, Cloudwatch, Transcoder & SNS. Open IAM and create a new role for Lambda as shown below.

Select the policies below:
AmazonS3FullAccess
CloudWatchFullAccess
AmazonSNSFullAccess
AmazonElasticTranscoder_FullAccess

Setup LAMBDA

Access the Lambda dashboard and create a function as below.

Click on the Function

Scroll down to the code window and paste the below code. (If you do not see the inline code editor in Google Chrome, please use Internet Explorer or Edge)

Code

'use strict';
var AWS = require('aws-sdk'),
    transcoder = new AWS.ElasticTranscoder({
        apiVersion: '2012-09-25',
        region: 'ap-south-1'
    });
exports.handler = (event, context, callback) => {
    let fileName = event.Records[0].s3.object.key;
    var srcKey =  decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " 
"));
    var newKey = fileName.split('.')[0];
    console.log('New video has been uploaded:', fileName);
transcoder.createJob({
     PipelineId: process.env.PIPELINE_ID,
     Input: {
      Key: srcKey,
      FrameRate: 'auto',
      Resolution: 'auto',
      AspectRatio: 'auto',
      Interlaced: 'auto',
      Container: 'auto'
     },
     Output: {
      Key: getOutputName(fileName),
      ThumbnailPattern: '',
      PresetId: '1351620000001-300040',
      Rotate: 'auto'
     }
    }, function(err, data){
        if(err){
            console.log('Something went wrong:',err)
        }else{
            console.log('Converting is done');
        }
     callback(err, data);
    });
};
function getOutputName(srcKey){
 let baseName = srcKey.replace('converted/','');
 let withOutExtension = removeExtension(baseName);
 return 'converted/' + withOutExtension + '.mp3';
}
function removeExtension(srcKey){
    let lastDotPosition = srcKey.lastIndexOf(".");
    if (lastDotPosition === -1) return srcKey;
    else return srcKey.substr(0, lastDotPosition);
}

Add environment variable pipeline_id and the value should be the pipeline id which you have noted down earlier from the Elastic Transcoder pipeline creation.

Next step is to add a trigger for the lambda function so that the lambda code will be executed when a file is uploaded to the bucket.

Click on S3 in the trigger section on the left and scroll down to configure the trigger.

You can now test the function by uploading a .wav file to the bucket and check the output .mp3 in the converted folder. Also, you may check the cloud watch logs as well as the mail you will be receiving after completion of the conversion.

The converted files get saved at the output location mentioned in the code.

Cloud watch logs

Confirmation mail received

That’s it! This method can be used to convert from any format to any other format. In the code, you need to edit the extension .mp3 to the desired format. Also in the trigger, change the .wav to the desired source format. One more change which would be required is the 5th line of the code – region: ‘ap-south-1’. Change it to whichever region is your default region. Let me know in the comments below for any queries or suggestions.


 

3 COMMENTS

  1. My lambda function runs multiple times, so each video is converted multiple times (from 11 to 27 times). How I can evade this?

  2. The behavior you’re experiencing is due to the fact that Lambda function can be executed multiple times in response to an event. This applies to any event type; for example cron job and S3 event notification are both push event sources hence it would result in a similar behavior.

    I would recommend implementing a mechanism in your code to prevent idempotency (repeated execution of the function)
    https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html

LEAVE A REPLY

Please enter your comment!
Please enter your name here