Create a Managed FFmpeg Workflow for Your Media Jobs Using AWS Batch

FFmpeg is an industry standard, open source, widely used utility for handling video. FFmpeg has many capabilities, including encoding and decoding all video compression formats, encoding and decoding audio, encapsulating and extracting audio and video from transport streams, and much more.

If AWS customers want to use FFmpeg on AWS, they have to maintain FFmpeg by themselves through an Amazon Elastic Compute Cloud (Amazon EC2) instance and develop a workflow manager to ingest and manipulate media assets. It’s painful.

In this post, I will show how to integrate FFmpeg with AWS Services to build a more easily managed FFmpeg. We’ve  created an open source solution to deploy FFmpeg packaged in a container and managed by AWS Batch. When finished, you will execute an FFmpeg command as a job through a REST API. This solution improves usability and offers relief from the management learning curve and maintenance costs of open source FFmpeg on AWS.

Solution overview

AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. There is no additional charge for AWS Batch, you pay only for AWS compute resources. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as Amazon EC2 and Spot Instances.

As of February 2023, AWS offers 15 general purpose EC2 instance families, 11 compute optimized instance families and 14 accelerated computing instances. By correlating each instance family specification with the FFmpeg hardware acceleration API, we’ve highlighted these instances that can optimize the performance of FFmpeg:

NVIDIA GPU-powered Amazon EC2 instances : P3 instance family comes equipped with the NVIDIA Tesla V100 GPU. G4dn instance family is powered by NVIDIA T4 GPUs and Intel Cascade Lake CPUs. These GPUs are well suited for video coding workloads and offers enhanced hardware-based encoding/decoding (NVENC/NVDEC).

Xilinx media accelerator cards : VT1 instances are powered by up to 8 Xilinx® Alveo U30 media accelerator cards and support up to 96 vCPUs, 192GB of memory, 25 Gbps of enhanced networking, and 19 Gbps of EBS bandwidth. The Xilinx Video SDK includes an enhanced version of FFmpeg that can communicate with the hardware accelerated transcode pipeline in Xilinx devices. VT1 instances deliver up to 30% lower cost per stream than Amazon EC2 GPU-based instances and up to 60% lower cost per stream than Amazon EC2 CPU-based instances.
EC2 instances powered by Intel : M6i/C6i instances are powered by 3rd generation Intel Xeon Scalable processors (code named Ice Lake) with an all-core turbo frequency of 3.5 GHz.
AWS Graviton-bases instances : Encoding video on C7g instances, the last AWS Graviton processor family, costs measured 29% less for H.264 and 18% less for H.265 compared to C6i, as described in this blog post ‘Optimized Video Encoding with FFmpeg on AWS Graviton’

AMD-powered EC2 instances: M6a instances are powered by 3rd generation AMD EPYC processors (code named Milan).
Serverless compute with Fargate: Fargate allows to have a completely serverless architecture for your batch jobs. With Fargate, every job receives the exact amount of CPU and memory that it requests.

We are going to create a managed video encoding pipeline using AWS Batch with FFmpeg in container images. For example, you will be able to make a simple transmuxing operation, add an audio silent track, extract audio/video track, change video container file, concatenate video files, generate thumbnails, or create a timelapse.  As a starting point, this pipeline uses Intel (C5), Graviton (C6g), Nvidia (G4dn), AMD (C5a, M5a), and Fargate instance families.

The architecture includes 5 key components:

Container images are stored in a Amazon Elastic Container Registry (Amazon ECR). Each container includes an FFmpeg library with a Python wrapper. Container images are specialized per CPU/GPU architecture : ARM64, x86-64, NVIDIA.

AWS Batch is configured with a queue and compute environment per CPU architecture. AWS Batch schedules job queues using Spot Instance compute environments only, to optimize cost.
Customers submit jobs through AWS SDKs with the ‘SubmitJob’ operation or use the Amazon API Gateway REST API to easily submit a job with any HTTP library.
All media assets ingested and produced are stored in an Amazon Simple Storage Service (Amazon S3)

Observability is managed by Amazon CloudWatch and AWS X-Ray. All X-Ray traces are exported on Amazon S3 to benchmark which compute architecture is better for a specific FFmpeg command.


You need the following prerequisites to set up the solution:

An AWS account with privileges to create AWS Identity and Access Management (IAM) roles and policies. For more information, see Overview of access management: Permissions and policies.
Latest version of AWS Cloud Development Kit (AWS CDK) with bootstrapping already done.
Latest version of Task.
Latest version of Docker.
Latest version of Python 3.

Deploy the solution with AWS CDK

To deploy the solution “AWS Batch with FFmpeg” on your account, complete the following steps:

Clone the GitHub repository

Execute this list of commands:

# Create a local Python virtual environment and install requirements

task venv

# Activate the Python virtual environment

source .venv/bin/activate

# Deploy the CDK stack

task cdk:deploy

# Collect AWS CloudFormation outputs from the stack

task env

# Build and push docker images for AMD64 processor architecture

task app:docker-amd64

# Build and push docker images for ARM64 processor architecture

task app:docker-arm64

# Build and push docker images for NVIDIA processor architecture

task app:docker-nvidia

AWS CDK outputs the new Amazon S3 bucket where you can upload and download video assets, and the Amazon API Gateway REST endpoint with which you can submit video jobs.

Use the solution

Once the “AWS Batch with FFmpeg” solution is installed, you can execute FFmpeg commands with the AWS SDKs, the AWS Command Line Interface (AWS CLI) or the API. The solution respects the typical syntax of the FFmpeg command described in the official documentation:

ffmpeg [global_options] {[input_file_options] -i input_url} … {[output_file_options] output_url} …

Parameters of the solution are:

global_options: FFmpeg global options described in the official documentation.

input_file_options: FFmpeg input file options described in the official documentation.

input_url: AWS S3 url synced to the local storage and tranformed to local path by the solution.

output_file_options: FFmpeg output file options described in the official documentation.

output_url: AWS S3 url synced from the local storage to AWS S3 storage.

compute: Instances family used to compute the media asset : intel, arm, amd, nvidia,

name: metadata of this job for observability.

In this example, we use the AWS SDK for Python (Boto3) and we want to cut a specific part of a video. As a prerequisite, we uploaded a video in the Amazon S3 bucket created by the solution. Now, we complete the parameters below:

import boto3

import requests

from urllib.parse import urlparse

from aws_requests_auth.boto_utils import BotoAWSRequestsAuth

# AWS CloudFormation output of the Amazon S3 bucket created by the solution : s3://batch-ffmpeg-stack-bucketxxxx/

s3_bucket_url = “<S3_BUCKET>”

# Amazon S3 key of the media Asset uploaded on S3 bucket, to compute by FFmpeg command : test/myvideo.mp4

s3_key_input = “<MEDIA_ASSET>”

# Amazon S3 key of the result of FFmpeg Command : test/output.mp4

s3_key_output = “<MEDIA_ASSET>”

# EC2 instance family : `intel`, `arm`, `amd`, `nvidia`, `fargate`

compute = “intel”

job_name = “clip-video”


    “name”: job_name,

    #”global_options”:  “”,

    “input_url” : s3_bucket_url + s3_key_input,

    #”input_file_options” : “”,

    “output_url” : s3_bucket_url + s3_key_output,

    “output_file_options”: “-ss 00:00:10 -t 00:00:15 -c:v copy -c:a copy”


And, I submit the FFmpeg command with the AWS SDK for Python (Boto3) :

batch = boto3.client(“batch”)

result = batch.submit_job(


    jobQueue=”batch-ffmpeg-job-queue-” + compute,

    jobDefinition=”batch-ffmpeg-job-definition-” + compute,



We can also submit the same FFmpeg command with the REST API through an HTTP POST method. I control access to this Amazon API Gateway API with IAM permissions:

# AWS Signature Version 4 Signing process with Python Requests

def apig_iam_auth(rest_api_url):

    domain = urlparse(rest_api_url).netloc

    auth = BotoAWSRequestsAuth(

        aws_host=domain, aws_region=”<AWS_REGION>”, aws_service=”execute-api”


    return auth

# AWS CloudFormation output of the Amazon API Gateway REST API created by the solution :

api_endpoint = “<API_ENDPOINT>”

auth = apig_iam_auth(api_endpoint)

url= api_endpoint + compute + ‘/ffmpeg’

response =, json=command, auth=auth, timeout=2)

Per default, AWS Batch chooses an available EC2 instance type. If you want to override it, you can add the `nodeOverride` property when you submit a job with the SDK:

instance_type = ‘c5.large’

result = batch.submit_job(


    jobQueue=”batch-ffmpeg-job-queue-” + compute,

    jobDefinition=”batch-ffmpeg-job-definition-” + compute,



            “nodePropertyOverrides”: [


                    “targetNodes”: “0,n”,

                    “containerOverrides”: {

                        “instanceType”: instance_type,






And with the REST API :

command[‘instance_type’] = instance_type

url= api_endpoint + compute + ‘/ffmpeg’

response =, json=command, auth=auth, timeout=2)


AWS Customers also want to use this solution to benchmark the video encoding performance of Amazon EC2 instance families. This solution analyzes performance and video quality metrics with AWS X-Ray.

AWS X-Ray helps developers analyze and debug applications. With X-Ray, we can understand how our application and its underlying services are performing to identify and troubleshoot the cause of performance issues.

We defined 3 X-Ray segments: Amazon S3 download, FFmpeg Execution, and Amazon S3 upload.

In the AWS Console (AWS Systems Manager > Parameter Store),  switch the AWS Systems Manager Parameter /batch-ffmpeg/ffqm to TRUE :  The video quality metrics PSNR, SSIM, VMAF are then calculated by FFmpeg and exported as AWS X-RAY metadata and as a JSON file uploaded in the Amazon S3 bucket with the key prefix /metrics/ffqm.

All JSON files are crawled by an Amazon Glue Crawler. This crawler provides an Amazon Athena table with which you can execute SQL requests to analyse the performance of our workload.

For example, we created a visual bar chart with Amazon QuickSight where our Athena table is our dataset. As shown in the chart here, for the job name compress-video launched with several instance types, the most efficient instance type is c5a.2xlarge.

Extend the solution

You can customize and extend the solution as you want. For example, you can customize the FFmpeg Docker image by adding libraries or upgrading the FFmpeg version. All docker files are located in application/docker-images/. You can customize the list of Amazon EC2 instances used by the solution with new instance types to optimize performance, updating the CDK stack located in this CDK file cdk/


There is no additional charge for AWS Batch. You pay only for AWS resources created to store assets and run the solution. We use Spot instances to optimize the cost. With metrics provided by AWS X-Ray, you can benchmark all instances to find the best one for your use case.


To avoid incurring unnecessary charges, clean up the resources you created for testing this solution.

Delete all objects in the Amazon S3 bucket.
Inside the Git repository, execute this command in a terminal : task cdk:destroy


In this post, we covered the process of setting up an FFmpeg workflow managed by AWS Batch. The solution includes an option to benchmark the video encoding performance of Amazon EC2 instance families. The solution is managed by AWS Batch, a scalable, and cost effective service using EC2 Spot instances.

This solution is open source and available at You can give us feedback through GitHub issues.

Flatlogic Admin Templates banner

Creating full-stack web applications based on database schema for free!

Today, we’re excited to announce the free pricing plan for our CRUD tool. That means that more customers than ever before can use Flatlogic for creating a proper CRUD React/Vue/Angular web application based on the database schema.

What can you do with the free plan for the CRUD tool

You can expect the same features and functionality whether you choose our Paid access or Free option. But obviously with some limitations:

Create up to 10 applications on different technology stacks: React, Vue, Angular, Node.js, Laravel, PostgreSQL, MySQL
Create a full-fledged CRUD application (up to 3 database entities) with code written using best practices
Download or push source code to Github an unlimited number of times
Preview the code before downloading/pushing it on Github
Get updates on libraries used in your app stack
Host instantly your application demo and see the latest deployment logs

Free vs Paid access to the CRUD tool: Choose what you need

Since launching Flatlogic, we see constant growth in the creation of CRUD applications. But we understand that not all of our users need the full functionality of the CRUD tool. We also understand that in the modern world of developer tools, before investing huge resources in mastering new tools, you want to get some value here and now for free – with this motivation we introducing a free plan for creating CRUD applications.

Thus the free plan fits your needs. If you:

Want quickly prototype the CRUD app

Want to test or get familiar with new technologies before investing much time in them
See the best practice on how to organize modern CRUD apps using best practices
Prefer to use starter kits/boilerplates while starting CRUD web applications

And while our Professional plan has always offered a more enhanced way to create CRUD applications, it also required additional steps to understand your needs before getting access.

Now, you can sign up for a free account for a CRUD tool, create applications in minutes, and understand without any hassle whether Flatlogic fits your needs or not.

Create a CRUD app for free!

The post Creating full-stack web applications based on database schema for free! appeared first on Flatlogic Blog.Flatlogic Admin Templates banner

Build Your Own Little Menu Bar App for All Those Things you Forget

I have a little Pinned Pen of Things I Forget that I revisit quite a bit… when… I forget… things. It occurred to me the other day that it would be even more convenient as a menu bar app (macOS) than a Pen. Maybe an excuse to learn Swift? Nahhh. There’s always an app for that. I know Fluid can do that, but I wasn’t 100% sure if that’s up to date anymore or what. As I happen to be a Setapp subscriber, I know they have Unite as well, which is very similar.

So let’s do this! (This is a macOS thing, for the record.)

1) Make a little website of Things You Forget that allows you to one-click copy to clipboard.

You don’t need a special URL or anything for it. And good news, CodePen Projects can deploy your Project to a URL with a click.

Here’s my example Project.

2) Deploy it

Now I click that Deploy button in the footer and can kick it out to a live-on-the-internet URL easily.

3) Turn it into a Menu Bar App with Unite

Then I launch Unite and give it the barebones information it needs:

That’ll instantly make the app. It takes like 2 seconds. From here, two things:

Paste a better image into the Get Info window so you can have a cool icon.
If you’re on an M1 Mac, you might have to click the “Open using Rosetta” icon in order for it to work.

That second one is a little unfortunate. Makes me wish I tried Fluid, but hey, this ultimately worked. You’ll know you need to do it if you see this:

4) Enjoy

A totally custom-designed app to help your own brain!

The post Build Your Own Little Menu Bar App for All Those Things you Forget appeared first on CodePen Blog.Flatlogic Admin Templates banner