Month: March 2023

In the last few years as companies have modernized their business applications, many have moved to microservices based architectures using containers on Kubernetes. A lot of the initial focus was on designing and building new cloud native architectures to support the applications. As environments have grown, we’ve seen a shift in focus to optimize resource allocation and right-size workloads to reduce costs.

In this blog post we will share guidance on how to optimize resource allocation and right-size applications in Kubernetes environments using Goldilocks. We’ll walk through how to install Goldilocks as well as a sample application to view the suggested resource recommendations. This applies to all Kubernetes applications, including those running on Amazon Elastic Kubernetes Service (Amazon EKS), that are deployed with managed node groups, self-managed node groups, and AWS Fargate.

Right-sizing applications on Kubernetes

In Kubernetes, resource right-sizing is done through setting resource specifications in the application manifest. These settings directly impact:

Performance — Kubernetes applications running on the same node will arbitrarily compete for resources without proper resource specifications. This can adversely impact application performance.
Cost Optimization — Applications deployed with oversized resource specifications will result in increased costs and underutilized infrastructure.
Autoscaling — The Kubernetes Cluster Autoscaler and Horizontal Pod Autoscaling require resource specifications to function.

The most common resource specifications in Kubernetes are for CPU and memory requests and limits.

Requests and Limits

Containerized applications are deployed on Kubernetes as Pods. CPU and memory requests and limits are an optional part of the Pod definition. CPU is specified in units of Kubernetes CPUs while memory is specified in bytes, usually as mebibytes (Mi).

Requests and limits each serve different functions in Kubernetes and affect scheduling and resource enforcement differently.

Scheduling

The Kubernetes scheduler only considers requests when determining where to place Pods in your cluster. Acceptable nodes are those that have enough available resources to satisfy the Pod’s resource requests.  Limits are not considered by the scheduler.

Resource Enforcement

The container runtime on the node where your Pods are running is responsible for resource enforcement.  Both requests and limits are factors in ensuring applications have access to their required compute resources. Their effect on CPU and memory is different:

CPU — If no limits are specified, then each Pod on a node can use all the available CPU on the host. As soon as available CPU is exhausted, Pods are throttled using a Linux primitive called cgroups. This is a resource sharing primitive that ensures each Pod gets its fair share of CPU time. CPU requests determine that fair share and are weighted to give more CPU time to Pods with larger CPU requests. If a limit is specified then CPU time will not exceed the specific limit.
Memory — Just like CPU, if no memory limits are specified, then each Pod can use all the available memory on the host. Unlike CPU, when memory is exhausted, there is no sharing mechanism. The Pod will either be terminated by the Linux Out-of-memory (OOM) killer or the kubelet will evict the Pod. The same process will happen if a Pod’s memory usage exceeds its limit.

Vertical Pod Autoscaler

So how do application owners choose the “right” values for their CPU and memory resource requests? An ideal solution is to load test the application in a development environment and measure resource usage using observability tooling. While that might make sense for your organization’s most critical applications, it’s likely not feasible for every containerized application deployed in your cluster.

Fortunately, there is a Kubernetes project that has a feature specifically designed to help provide resource recommendations — the Vertical Pod Autoscaler (VPA). VPA is a Kubernetes sub-project owned by the Autoscaling special interest group (SIG). It’s designed to automatically set Pod requests based on observed application performance. VPA collects resource usage using the Kubernetes Metric Server by default but can be optionally configured to use Prometheus as a data source.

VPA has a recommendation engine that measures application performance and makes sizing recommendations. The VPA recommendation engine can be deployed stand-alone so VPA will not perform any autoscaling actions. It’s configured by creating a VerticalPodAutoscaler custom resource for each application and VPA updates the object’s status field with resource sizing recommendations.

Creating VerticalPodAutoscaler objects for every application in your cluster and trying to read and interpret the JSON results is challenging at scale. Goldilocks is an open source project that makes this easy.

Goldilocks

Goldilocks is an open source project from Fairwinds that is designed to help organizations get their Kubernetes application resource requests “just right”. It takes its name, very appropriately, from the well known fairly tale Goldilocks and the Three Bears. Goldilocks builds on top of the Kubernetes Vertical Pod Autoscaler and provides:

A controller that automates the creation of VerticalPodAutoscaler objects for workloads in your cluster.
A dashboard that displays resource recommendations for all the monitored workloads.

The default configuration of Goldilocks is an opt-in model. You choose which workloads are monitored by adding the goldilocks.fairwinds.com/enabled: true label to a namespace.

Solution Overview

Let’s walk through how to install Goldilocks, including its dependencies Metrics Server and Vertical Pod Autoscaler. Then we’ll install a sample application to view the suggested resource recommendations. The diagram shown here illustrates all of the components on an Amazon EKS cluster and their interactions.

The Metrics Server collects resource metrics from the Kubelet running on worker nodes and exposes them through Metrics API for use by the Vertical Pod Autoscaler. The Goldilocks controller watches for namespaces with the goldilocks.fairwinds.com/enabled: true label and creates VerticalPodAutoscaler objects for each workload in those namespaces.

In this blog post, we will be creating a namespace called javajmx-sample and will be creating a tomcat deployment. We will label this namespace in order to get a recommendation from Goldilocks. As soon as we label the namespace, we will be able to see a VPA object called goldilocks-tomcat-example created.

Prerequisites

You will need the following to complete the steps in this post:

AWS Command Line Interface (AWS CLI) version 2
kubectl
helm
If you don’t have an Amazon EKS cluster, you can create one using the eksctl

Step 1: Deploying the Metrics Server

In this step, we will be deploying the Metrics server which provides the resource metrics to be used by Vertical Pod Autoscaler.

Let’s verify the status of the metrics-server. Once successfully deployed, you should be able to see the resource utilization of the deployments within seconds:

Step 2 : Enable namespaces which needs resource recommendation from Goldilocks

We will be deploying sample workloads in the javajmx-sample namespace and we will get the resource recommendation for the applications running on it. Let’s go ahead and create the namespace and label it.

To ensure the label was applied successfully, run describe on the javajmx-sample namespace

Step 3 : Deploy Goldilocks

We will be using a helm chart to deploy Goldilocks. The deployment creates three objects :

Goldilocks-controller: responsible for creating the VPA objects for the workloads whose namespace is enabled for a Goldilocks recommendation

Goldilocks-vpa-recommender:  responsible for providing the resource recommendations for the workloads

Goldilocks-dashboard: summarizes the resource recommendation of the workloads and will also provide the yaml manifest for implementing the recommendation.

To deploy Goldilocks, run the following helm commands:

Now, we will use kubectl to verify if the deployment was successful:

Step 4 : Deploy the sample application

In this step, we will be deploying a sample application in the javajmx-sample namespace to get recommendations from Goldilocks. The application tomcat-example  is initially provisioned with a CPU and Memory request of 100m and 180Mi respectively and limits of 300m CPU and 300 Mi Memory.

As mentioned earlier, Goldilocks will be creating VPAs for each deployment in a Goldilocks enabled namespace. Using the kubectl command, we can verify that a VPA was created in thejavajmx-sample namespace for the goldilocks-tomcat-example:

Step 5 : Review the Goldilocks recommendation dashboard

Goldilocks-dashboard will expose the dashboard in the port 8080 and we can access it to get the resource recommendation.  We now run this kubectl command to access the dashboard:

We can now open a browser to http://localhost:8080 to display the Goldilocks dashboard.

Let’s analyze the javajmx-sample namespace to see the recommendations provided by Goldilocks. We should be able to see the recommendations for the goldilocks-tomcat-example deployment.

Here the screen shows the request and limit recommendations for the javajmx-sample workload. The Current column under each Quality of Service (QoS) indicates the currently configured CPU and Memory request and limits. The Guaranteed and Burstable column under each QoS indicates the recommended CPU and Memory request limits for the respective QoS.

We can clearly notice  that we have over provisioned the resources and Goldilocks has made the recommendations to optimize the CPU and Memory request. The recommended level for CPU request and CPU limit is 15m and 15m compared to the current setting of 100m and 300m for Guaranteed QoS.  Memory request and limits are recommended to be 105M and 105M, compared to the current setting of 180Mi and 300 Mi.

Notice that the recommendations are available for two different Quality of Service (QoS) types: Guaranteed and Burstable. Kubernetes provides different levels of Quality of Service to pods depending on what they request and what limits are set for them. Pods that need to stay up and consistently good can request guaranteed resources, while pods with less exacting requirements can use resources with less or no guarantee.

Guaranteed (QoS) pods are considered top priority and are guaranteed to not be killed until they exceed their limits. If limits, and optionally requests, (not equal to 0) are set for all resources across all containers and limits and requests  are equal, then the pod is classified as Guaranteed.

Burstable (QoS) pods have some form of minimal resource guarantee, but can use more resources when available. Under system memory pressure, these containers are more likely to be killed once they exceed their requests and no Best-Effort pods exist. If requests, and optionally limits, are set (not equal to 0) for one or more resources across one or more containers, and they are not equal, then the pod is classified as Burstable.

To follow the recommended resource specification, customers can simply copy  the respective manifest file for the QoS class they are interested in and deploy the workloads which will then be right-sized and optimized.

For example, if we decide to apply the recommendations for the Guaranteed QoS, we could copy the YAML from the dashboard as shown here and apply them to the deployment object:

Let’s run the kubectl edit command to the deployment to apply the recommendations:

The resources section in the containers spec  shows that we have successfully applied the recommendation of request and limits for CPU, and memory:

Once we apply the recommendations, we should be able to verify that the pod is trying to restart and come online with the updated resource configuration. Let’s verify the same by running the kubectl describe  command on the tomcat-example deployment:

The output should look like the following:

Cleanup

To delete the deployments and sample workloads we created in the blog, execute the following commands:

Conclusion

This post demonstrated how Goldilocks can be used to efficiently rightsize the resource requests for Kubernetes applications. Customers in modernization efforts often have minimal time to decide the resource requirements for their applications, which usually involves a complex process of reviewing monitoring dashboards. By adopting the recommendations from Goldilocks, customers can shorten the time to market for their applications and optimize their Amazon EKS costs.

Further reading

EKS Best practices
Blog: Using Prometheus to Avoid Disasters with Kubernetes CPU Limits

Goldilocks project

Do you want to investigate the potential of having a custom CRM system for your business? What are the characteristics of custom CRM systems? Or curious about the cost and what it would take to build a custom CRM system? 

Customer relationship management (CRM) systems are widely available to businesses today. Custom software development is the greatest choice for creating and executing software from the beginning because one-size-fits-all software is not available.  Custom software enables businesses to focus a greater emphasis on the people who are a part of the organization, including employees, vendors, clients, and service users. The only way to guarantee that the software precisely meets the company’s demands is to create custom CRM software.
A custom CRM can help you ensure that your business takes advantage of all opportunities to engage, convert, and retain clients. Regardless of the size of your business, a custom CRM system might streamline your operations, improve your interactions with current clients, and elicit new leads and business opportunities.

This article will help you understand the advantages of a custom CRM system, the implementation requirements, and the development costs. You’ll also be able to understand why a customized CRM system is perfect for your business.

What is Custom CRM System?

CRM stands for Customer Relationship Management and refers to all strategies, techniques, tools, and technologies used by enterprises for developing, retaining, and acquiring customers.

Off-the-shelf (OTS) CRM is ready-to-use software that businesses may buy unmodified and use right away. It often includes modules for keeping track of sales possibilities, managing marketing campaigns, and setting up procedures for customer service.

Custom CRM systems can foster customer loyalty and automate several processes, saving businesses time and money. The main purpose of CRM software is to allow salespeople and marketers to better manage and analyze relationships with the business’s customers and potential customers.

Custom CRM vs. Off-the-Shelf CRM: What are the differences?

The biggest difference between a custom CRM and an off-the-shelf CRM is flexibility. An off-the-shelf CRM is frequently already created and prepared for usage, in contrast to a custom CRM, which is created especially for the requirements of the firm. Off-the-shelf CRMs can provide less adequate features and possibilities, but custom CRMs can be designed to satisfy the particular wants of the firm. Custom CRMs typically require more technical expertise and effort to set up and run than off-the-shelf CRMs do.

With a custom CRM, businesses may modify the program to meet their unique needs and requirements, ensuring the application is appropriate for their particular industry. Moreover, custom CRMs provide businesses the opportunity to integrate the application with their existing internal hardware and software, creating a seamless user experience. On the other hand, off-the-shelf CRMs were unable to offer as many customization and integration choices.

Usage of Custom CRM

With a customized CRM, any department in your company – from sales and customer service to business development, recruiting, and marketing – can benefit from improved methods of managing external connections and activities that are integral to success. As no two businesses are alike, a custom CRM is the only way to get precisely what you need without any unnecessary extras or having to make do with features that may not have been included in an off-the-shelf solution.

When it comes to investing in a CRM, you have three choices: 

buying an existing system, 
creating one with an in-house team, 
or outsourcing the development of a custom CRM. 

When you’re considering a CRM investment, it’s important to look into all of your options. Although buying an off-the-shelf CRM system could be the most cost-effective option, it might not be the greatest fit for the unique requirements of your company. While developing a custom CRM in-house can be a terrific option, it often takes a significant amount of time and resources and costs over $50,000. Instead, you might outsource the building of a custom CRM, which would be more cost-effective and efficient. These factors make it crucial to thoroughly consider all of your alternatives before making a choice.

Why is custom CRM important for your business? 

Custom CRM is important for your business since it offers a potent tool for managing client interactions and simplifying sales processes. It gives the ability to recognize, monitor, and control customer interactions, automate lead management, and examine customer data to learn more about how customers behave. This helps you better understand customer needs and create tailored customer experiences that drive customer loyalty and boost sales. As a consequence, improves client retention by raising customer satisfaction and a better understanding of their needs.

Types of Custom CRM

There are three major types of custom CRMs based on their function and the features they provide.

Collaborative

By establishing a clear framework for data exchange, collaborative CRM is intended to improve cooperation and communication. To create a custom CRM adapted to particular needs, it may be utilized internally inside a company or between external teams, such as partners.  Common features of this kind of system include group discussions, content sharing, and real-time activity updates.

Analytical

Analytical CRM systems are intended to help in planning. Such a system offers helpful data, analytics, and insights. It must be able to compile data from several sources, process it, and deliver real-time changes to be useful.

Operational

Operational CRMs focus heavily on simplifying and automating company processes to increase productivity. Lead processing, automated messaging to clients via various channels, and follow-ups are frequently helpful. You have two options when creating your solution: either choose certain features or mix various CRM kinds.

Benefits of Custom CRM

Because of their numerous benefits, custom CRM systems are becoming increasingly popular among businesses. These technologies not only enable businesses to give great and individualized customer care, but they also have a wide range of positive effects on customers. As a result, businesses are using CRM systems more often to increase customer service. Some of the benefits of custom CRM systems are:

Benefit #1 Time Saving

Organizations can easily obtain the information they need to complete some essential activities thanks to custom CRM systems. They can also automate certain monotonous chores. Therefore they may spend less time providing services to their clients. This allows them to focus on other important duties while letting the custom CRM handle additional tasks like data processing, analysis, customer care, sales, and marketing.

Benefit #2 Improved Efficiency

Using a custom CRM for task management gives employees a straightforward way to access the information they need to do their jobs and each employee can use their individualized dashboard. This makes their work easier and more productive; in fact, 60% of businesses report an increase in productivity from implementing a custom CRM.

Benefit #3 Enhanced Customer Relationship

Data about customers, such as their preferences, needs, and pain points, is readily available from a custom CRM. 84% of consumers believe that a business’s experience is just as important as the goods and services they provide. Custom CRM enables you to provide customized services to your consumers while addressing their pain points thanks to all the data available from the system. This can help you improve the way you communicate with your consumers, which will increase their likelihood of becoming repeat customers and help you grow your business.

Benefit #4 Access to in-depth Report

Making data-driven choices, such as adjusting prices or marketing tactics, requires the usage of reports that are produced by custom CRM using customer data. A custom CRM can give you reports that will help your business decision-making process because employing itself might boost report accuracy by 42%, according to studies.

Benefit #5 Increased Income

Сustom CRM system can handle practically all of your company’s requirements for attracting new clients, turning them into customers, and offering top-notch customer service. Sales are boosted as a result since the data it gives may be utilized to deliver better customer service and foster customer loyalty.

What to Consider Before Building Custom CRM Software

A good CRM tool will let you store contact information for clients and prospects, identify sales opportunities, keep a record of service issues, and manage marketing campaigns and tactics – all in one central location. Easy access to this data about customer interaction will allow anyone in your company to make informed decisions based on analytics. Before beginning to build a custom CRM system, there are several crucial considerations to be made. By doing this, you may decide as soon as possible and avoid making costly mistakes.

Setting the Goals

Prioritize the most crucial custom CRM platform goals after determining their importance. This will enable you to select the custom CRM system’s features and design that are most appropriate for your business.

Types of Custom CRM Systems

Decide on the type of CRM solutions you need after you’ve defined your objectives. CRM is divided into three categories: organizational, analytical, and collaborative. Each is made for a specific purpose.

Access Roles & Levels

Since several departments could utilize the custom CRM for various purposes, we recommend adding user functions and permissions. For instance, this may apply to top management, marketing directors, sales, and customer care personnel.

Custom CRM Features

Base your decision on which functionalities to include in your custom CRM on your objectives. Pay attention to the features that will be most beneficial for your business needs. Some of the most important features of custom CRM are dashboards, reports, tasks, contact management, lead management, and mobile access.

SaaS or Internal Software

Think about whether you’re going to create a custom CRM system for internal use or whether you’ll eventually transform it into a SaaS platform. It can be challenging and expensive to change the software architecture. Make sure your architecture is adaptable and scalable from the start if you choose the latter.

Cost of CRM Software Development 

Regardless of the size of your business, custom CRM software can benefit you. Estimating the associated costs can be difficult, so we’re here to explain the anticipated expenses. The cost of a custom CRM system depends on the features you want to include, the technical complexity, the development team’s experience, and other functionalities such as security measures.

To put things into perspective, if you manage a mid-sized business with more than 25 employees and your enterprise-sized subscription costs $125 per user each month, it works out to $3125 per month, $37,500 per year, and $187,500 over five years. The same amount of money, however, might be used to purchase custom CRM software that would be made to meet your unique needs. Determining the precise cost of your project is something we always advise doing in consultation with our specialists. We are aware that every customer is unique and needs a customized strategy to enable the smooth growth of their product. 

Summing Up

Are you looking for a way to improve your business performance? A custom CRM system can be the perfect solution. It provides businesses with a powerful platform for better managing customer relationships and increasing sales. With the right planning and guidance, you can create your own high-quality CRM from scratch that is tailored to your business needs. By centralizing customer data and tracking customer interactions, you can gain insights into customer behavior, create targeted marketing campaigns, and automate specific processes. This can help you improve efficiency, increase customer satisfaction, and boost sales. 

At Flatlogic, we provide full-cycle custom CRM development services to help you turn your concept into a working product. By leveraging our expertise and technical capabilities, you can create a high-quality custom CRM tailored to your business needs that will become the heart of your operations just in a few minutes. 

Custom CRM with Flatlogic

Flatlogic Platform offers an easy way to generate a custom CRM system with full control over the source code. You can make sure that you have the right features, scalability, and performance to match your business needs. Plus, with no-code development, you don’t need to be an expert programmer to make the necessary changes – making it easier to scale and customize as your business grows. With Flatlogic Platform, you have the flexibility to create a custom CRM solution that is tailored to your needs, while still having the scalability of more traditional development.

How to Create Custom CRM with Flatlogic Platform?

Using the Flatlogic Full-Stack Generator you can create CRUD and static applications in a few minutes. To start using the Platform, you need to register on the Flatlogic website. Clicking the “Sign in” button in the header will allow you to register for a Flatlogic account.

Step 1. Choosing the Tech Stack

In this step, you’re setting the name of your application and choosing the stack: Frontend, Backend, and Database.

Step 2. Choosing the Starter Template

In this step, you’re choosing the design of the web app.

Step 3. Schema Editor

In this step, you can create your database schema from scratch, import an existing schema or select one of the suggested schemas. 

To import your existing database, click the Import SQL button and select your .sql file. After that, your database will be opened in the Schema Editor where you can further edit your data (add/edit/delete entities).

If you are not familiar with database design and find it difficult to understand what tables are, we have prepared some ready-made sample schemas of real applications that you can modify for your application:

E-commerce app;
Time tracking app;
Book store;
Chat (messaging) app;
Blog.

Or, you can define a database schema and add a description by clicking on the “Generate with AI” button. You need to type the application’s description in the text area and hit “Send”. The application’s schema will be ready in around 15 seconds. You may either hit deploy immediately or review the structure to make manual adjustments.

Next, you can connect your GitHub and push your application code there. Or skip this step by clicking the Finish and Deploy button and in a few minutes, your application will be generated.

The post Custom CRM System: Benefits, Requirements & Cost of Development appeared first on Flatlogic Blog.

FFmpeg is an industry standard, open source, widely used utility for handling video. FFmpeg has many capabilities, including encoding and decoding all video compression formats, encoding and decoding audio, encapsulating and extracting audio and video from transport streams, and much more.

If AWS customers want to use FFmpeg on AWS, they have to maintain FFmpeg by themselves through an Amazon Elastic Compute Cloud (Amazon EC2) instance and develop a workflow manager to ingest and manipulate media assets. It’s painful.

In this post, I will show how to integrate FFmpeg with AWS Services to build a more easily managed FFmpeg. We’ve  created an open source solution to deploy FFmpeg packaged in a container and managed by AWS Batch. When finished, you will execute an FFmpeg command as a job through a REST API. This solution improves usability and offers relief from the management learning curve and maintenance costs of open source FFmpeg on AWS.

Solution overview

AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. There is no additional charge for AWS Batch, you pay only for AWS compute resources. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as Amazon EC2 and Spot Instances.

As of February 2023, AWS offers 15 general purpose EC2 instance families, 11 compute optimized instance families and 14 accelerated computing instances. By correlating each instance family specification with the FFmpeg hardware acceleration API, we’ve highlighted these instances that can optimize the performance of FFmpeg:

NVIDIA GPU-powered Amazon EC2 instances : P3 instance family comes equipped with the NVIDIA Tesla V100 GPU. G4dn instance family is powered by NVIDIA T4 GPUs and Intel Cascade Lake CPUs. These GPUs are well suited for video coding workloads and offers enhanced hardware-based encoding/decoding (NVENC/NVDEC).

Xilinx media accelerator cards : VT1 instances are powered by up to 8 Xilinx® Alveo U30 media accelerator cards and support up to 96 vCPUs, 192GB of memory, 25 Gbps of enhanced networking, and 19 Gbps of EBS bandwidth. The Xilinx Video SDK includes an enhanced version of FFmpeg that can communicate with the hardware accelerated transcode pipeline in Xilinx devices. VT1 instances deliver up to 30% lower cost per stream than Amazon EC2 GPU-based instances and up to 60% lower cost per stream than Amazon EC2 CPU-based instances.
EC2 instances powered by Intel : M6i/C6i instances are powered by 3rd generation Intel Xeon Scalable processors (code named Ice Lake) with an all-core turbo frequency of 3.5 GHz.
AWS Graviton-bases instances : Encoding video on C7g instances, the last AWS Graviton processor family, costs measured 29% less for H.264 and 18% less for H.265 compared to C6i, as described in this blog post ‘Optimized Video Encoding with FFmpeg on AWS Graviton’

AMD-powered EC2 instances: M6a instances are powered by 3rd generation AMD EPYC processors (code named Milan).
Serverless compute with Fargate: Fargate allows to have a completely serverless architecture for your batch jobs. With Fargate, every job receives the exact amount of CPU and memory that it requests.

We are going to create a managed video encoding pipeline using AWS Batch with FFmpeg in container images. For example, you will be able to make a simple transmuxing operation, add an audio silent track, extract audio/video track, change video container file, concatenate video files, generate thumbnails, or create a timelapse.  As a starting point, this pipeline uses Intel (C5), Graviton (C6g), Nvidia (G4dn), AMD (C5a, M5a), and Fargate instance families.

The architecture includes 5 key components:

Container images are stored in a Amazon Elastic Container Registry (Amazon ECR). Each container includes an FFmpeg library with a Python wrapper. Container images are specialized per CPU/GPU architecture : ARM64, x86-64, NVIDIA.

AWS Batch is configured with a queue and compute environment per CPU architecture. AWS Batch schedules job queues using Spot Instance compute environments only, to optimize cost.
Customers submit jobs through AWS SDKs with the ‘SubmitJob’ operation or use the Amazon API Gateway REST API to easily submit a job with any HTTP library.
All media assets ingested and produced are stored in an Amazon Simple Storage Service (Amazon S3)

Observability is managed by Amazon CloudWatch and AWS X-Ray. All X-Ray traces are exported on Amazon S3 to benchmark which compute architecture is better for a specific FFmpeg command.

Prerequisites

You need the following prerequisites to set up the solution:

An AWS account with privileges to create AWS Identity and Access Management (IAM) roles and policies. For more information, see Overview of access management: Permissions and policies.
Latest version of AWS Cloud Development Kit (AWS CDK) with bootstrapping already done.
Latest version of Task.
Latest version of Docker.
Latest version of Python 3.

Deploy the solution with AWS CDK

To deploy the solution “AWS Batch with FFmpeg” on your account, complete the following steps:

Clone the GitHub repository https://github.com/aws-samples/aws-batch-with-ffmpeg

Execute this list of commands:

# Create a local Python virtual environment and install requirements

task venv

# Activate the Python virtual environment

source .venv/bin/activate

# Deploy the CDK stack

task cdk:deploy

# Collect AWS CloudFormation outputs from the stack

task env

# Build and push docker images for AMD64 processor architecture

task app:docker-amd64

# Build and push docker images for ARM64 processor architecture

task app:docker-arm64

# Build and push docker images for NVIDIA processor architecture

task app:docker-nvidia

AWS CDK outputs the new Amazon S3 bucket where you can upload and download video assets, and the Amazon API Gateway REST endpoint with which you can submit video jobs.

Use the solution

Once the “AWS Batch with FFmpeg” solution is installed, you can execute FFmpeg commands with the AWS SDKs, the AWS Command Line Interface (AWS CLI) or the API. The solution respects the typical syntax of the FFmpeg command described in the official documentation:

ffmpeg [global_options] {[input_file_options] -i input_url} … {[output_file_options] output_url} …

Parameters of the solution are:

global_options: FFmpeg global options described in the official documentation.

input_file_options: FFmpeg input file options described in the official documentation.

input_url: AWS S3 url synced to the local storage and tranformed to local path by the solution.

output_file_options: FFmpeg output file options described in the official documentation.

output_url: AWS S3 url synced from the local storage to AWS S3 storage.

compute: Instances family used to compute the media asset : intel, arm, amd, nvidia,

name: metadata of this job for observability.

In this example, we use the AWS SDK for Python (Boto3) and we want to cut a specific part of a video. As a prerequisite, we uploaded a video in the Amazon S3 bucket created by the solution. Now, we complete the parameters below:

import boto3

import requests

from urllib.parse import urlparse

from aws_requests_auth.boto_utils import BotoAWSRequestsAuth

# AWS CloudFormation output of the Amazon S3 bucket created by the solution : s3://batch-ffmpeg-stack-bucketxxxx/

s3_bucket_url = “<S3_BUCKET>”

# Amazon S3 key of the media Asset uploaded on S3 bucket, to compute by FFmpeg command : test/myvideo.mp4

s3_key_input = “<MEDIA_ASSET>”

# Amazon S3 key of the result of FFmpeg Command : test/output.mp4

s3_key_output = “<MEDIA_ASSET>”

# EC2 instance family : `intel`, `arm`, `amd`, `nvidia`, `fargate`

compute = “intel”

job_name = “clip-video”

command={

    “name”: job_name,

    #”global_options”:  “”,

    “input_url” : s3_bucket_url + s3_key_input,

    #”input_file_options” : “”,

    “output_url” : s3_bucket_url + s3_key_output,

    “output_file_options”: “-ss 00:00:10 -t 00:00:15 -c:v copy -c:a copy”

}

And, I submit the FFmpeg command with the AWS SDK for Python (Boto3) :

batch = boto3.client(“batch”)

result = batch.submit_job(

    jobName=job_name,

    jobQueue=”batch-ffmpeg-job-queue-” + compute,

    jobDefinition=”batch-ffmpeg-job-definition-” + compute,

    parameters=command,

)

We can also submit the same FFmpeg command with the REST API through an HTTP POST method. I control access to this Amazon API Gateway API with IAM permissions:

# AWS Signature Version 4 Signing process with Python Requests

def apig_iam_auth(rest_api_url):

    domain = urlparse(rest_api_url).netloc

    auth = BotoAWSRequestsAuth(

        aws_host=domain, aws_region=”<AWS_REGION>”, aws_service=”execute-api”

    )

    return auth

# AWS CloudFormation output of the Amazon API Gateway REST API created by the solution : https://xxxx.execute-api.xx-west-1.amazonaws.com/prod/

api_endpoint = “<API_ENDPOINT>”

auth = apig_iam_auth(api_endpoint)

url= api_endpoint + compute + ‘/ffmpeg’

response = requests.post(url=url, json=command, auth=auth, timeout=2)

Per default, AWS Batch chooses an available EC2 instance type. If you want to override it, you can add the `nodeOverride` property when you submit a job with the SDK:

instance_type = ‘c5.large’

result = batch.submit_job(

    jobName=job_name,

    jobQueue=”batch-ffmpeg-job-queue-” + compute,

    jobDefinition=”batch-ffmpeg-job-definition-” + compute,

    parameters=command,

    nodeOverrides={

            “nodePropertyOverrides”: [

                {

                    “targetNodes”: “0,n”,

                    “containerOverrides”: {

                        “instanceType”: instance_type,

                    },

                },

            ]

        },

    )

And with the REST API :

command[‘instance_type’] = instance_type

url= api_endpoint + compute + ‘/ffmpeg’

response = requests.post(url=url, json=command, auth=auth, timeout=2)

Metrics

AWS Customers also want to use this solution to benchmark the video encoding performance of Amazon EC2 instance families. This solution analyzes performance and video quality metrics with AWS X-Ray.

AWS X-Ray helps developers analyze and debug applications. With X-Ray, we can understand how our application and its underlying services are performing to identify and troubleshoot the cause of performance issues.

We defined 3 X-Ray segments: Amazon S3 download, FFmpeg Execution, and Amazon S3 upload.

In the AWS Console (AWS Systems Manager > Parameter Store),  switch the AWS Systems Manager Parameter /batch-ffmpeg/ffqm to TRUE :  The video quality metrics PSNR, SSIM, VMAF are then calculated by FFmpeg and exported as AWS X-RAY metadata and as a JSON file uploaded in the Amazon S3 bucket with the key prefix /metrics/ffqm.

All JSON files are crawled by an Amazon Glue Crawler. This crawler provides an Amazon Athena table with which you can execute SQL requests to analyse the performance of our workload.

For example, we created a visual bar chart with Amazon QuickSight where our Athena table is our dataset. As shown in the chart here, for the job name compress-video launched with several instance types, the most efficient instance type is c5a.2xlarge.

Extend the solution

You can customize and extend the solution as you want. For example, you can customize the FFmpeg Docker image by adding libraries or upgrading the FFmpeg version. All docker files are located in application/docker-images/. You can customize the list of Amazon EC2 instances used by the solution with new instance types to optimize performance, updating the CDK stack located in this CDK file cdk/batch_job_ffmpeg_stack.py.

Cost

There is no additional charge for AWS Batch. You pay only for AWS resources created to store assets and run the solution. We use Spot instances to optimize the cost. With metrics provided by AWS X-Ray, you can benchmark all instances to find the best one for your use case.

Cleanup

To avoid incurring unnecessary charges, clean up the resources you created for testing this solution.

Delete all objects in the Amazon S3 bucket.
Inside the Git repository, execute this command in a terminal : task cdk:destroy

Summary

In this post, we covered the process of setting up an FFmpeg workflow managed by AWS Batch. The solution includes an option to benchmark the video encoding performance of Amazon EC2 instance families. The solution is managed by AWS Batch, a scalable, and cost effective service using EC2 Spot instances.

This solution is open source and available at http://github.com/aws-sample/aws-batch-with-ffmpeg/. You can give us feedback through GitHub issues.

If you are looking for an effective and efficient way to manage your business, then a custom enterprise resource planning (ERP) system is the perfect solution. Custom ERP systems allow businesses to collect, store, and analyze information from several departments in one database, making it easier for executives to manage all fundamental business operations. Furthermore, the implementation of such software optimizes various workflows and processes. While there are various commercially available systems, custom ERP development ensures that you get a solution tailored specifically to your needs.

There is always room for development, even if your product is excellent. This is especially true for businesses looking for a custom ERP solution. The increasing need for specific software solutions serves as evidence of the need for a unique ERP system. 

If you are considering an ERP system for your business when making this important decision, ask yourself: What are the benefits of a custom ERP system? How does a custom ERP system compare to an off-the-shelf version? Is it the right fit for my business needs? Whatever you decide, it’s important to choose the right system for your business. In this article, we’ll talk about all of the above in developing custom ERP systems and provide some helpful tips on how to choose the right one.  Let’s dive into the details.

What is custom ERP?

ERP stands for Enterprise Resource Planning is a sophisticated software package that automates all business operations procedures. It aids businesses in optimizing efficiency and accuracy by automating data input and offering real-time visibility into their operations, consequently reducing the time and costs associated with managing operations.

An Off-the-shelf (OTS) ERP system is a pre-packaged, ready-to-deploy solution that comes with verified capabilities. It is designed to meet the needs of a wide range of businesses, however, its “one-size-fits-all” approach may not be the best solution for addressing specific challenges.

Custom ERP (as opposed to OTS solutions) system is designed around the unique requirements of a particular business. It offers a complete solution that unifies all of the business operations into a single platform and is created to match the particular needs of a particular business.

How Custom ERPs Different From Off-the-shelf ERPs

Custom ERPs are created especially for the individual or business that requires them, in contrast to off-the-shelf ERPs. This makes it possible to develop a solution that is more tailored to the unique requirements and needs of the customer and is also more effective, efficient, and affordable.  The capacity for custom ERPs to often be upgraded and modified over time typically allows for more flexibility and scalability.  Moreover, custom ERPs may be integrated into other systems to increase their versatility. Off-the-shelf ERPs, on the other hand, are pre-packaged, unchangeable systems with frequently limited scalability.

Usage of custom ERP

Custom ERP systems are produced to assist businesses in managing their operations more effectively. As they are customized to the particular requirements of the organization, they are more adaptable and able to offer a greater level of control. Sales, inventories, financial and operational procedures, customer support, and more could all be managed by custom ERP systems. Moreover, they can offer data to assist organizations in making wiser decisions. Customer Relationship Management (CRM), Human Resources (HR), supply chain management, and other business applications are frequently incorporated into custom ERP systems. This facilitates data access and the automation of numerous business operations. Custom ERP systems can help gain businesses a complete picture of their operations and can be extremely motivated, customer experience, and revenue.

Benefits of custom ERP

​​The main benefit of having custom software is that it can help focus on the specific needs and resources of businesses. It is essential to understand how to effectively incorporate and monitor all data management and workflows of the business from a single platform. These are some of the benefits of possessing a custom ERP system:

Benefit #1: Improved System Performance and Accessibility 

The goal of custom ERP is to be as effective and efficient as possible while offering more application availability than other systems. Businesses may easily adapt the software to their changing demands thanks to its modular architecture, saving money on potentially less-effective items.

Benefit #2: Balance Between Integration & Specification

Businesses may gain from an integrated workflow across departments with a custom ERP system, which will lead to better connections between buyers and suppliers. Different departments and teams can have their customized system thanks to the fact that custom ERP systems are created to match the specific needs of each business. Software developers may ensure the system is suited to the precise specifications and circumstances of the company’s numerous internal divisions and departments by developing and building a custom ERP that is adapted to the company’s particular system needs, workflow, and applications.

Benefit #3: No Need for Modifications

Businesses may save time and effort by using custom ERP systems instead of converting their current database architectures, applications, and tech stacks to the new system. Software development businesses may offer a ‘plug and play’ system by designing a custom ERP that suits the company’s current infrastructure, networks, capabilities, and resources, allowing enterprises to swiftly adopt the new system and begin profiting from its features.

Benefit #4: Custom Solutions for Your Business Needs

Custom ERP systems are created to deal with issues frequently encountered by businesses. The developers of such systems build the framework, features, and functionalities based on what they’ve picked up from past customers. This is a major plus point since it draws from the knowledge of many clients and businesses. 

Benefit #5: Automation Without Workflow Changes

Custom ERP can automate multiple operations within the company’s existing systems, databases, infrastructure, networks, and applications, without requiring extensive changes and modifications. This guarantees seamless integration while reducing any gaps between the company’s existing system and custom ERP. One important duty that may be automated with a custom ERP is the periodic report generation process, which makes it simpler to keep track of business activities.

Benefit #6: Improved User Experience and Customer Interface

Without a custom ERP system, many companies suffer from a lack of cohesion between their frontend website and backend database, middleware, and application systems. Even with a custom ERP system, plug-and-play capabilities for productivity applications and workflow systems are not always available. Software development companies can provide a solution to this issue by creating custom ERP frameworks and systems that allow for easy integration of existing infrastructure, workflow processes, and productivity applications.

Benefit #7: Monitor Performance with One Reporting System

Gain total visibility into all business processes, from Finance and Accounts Management, Human Resources to Manufacturing, Marketing, and Sales, and Supply Chain and Warehouse Management. Automate departmental workflow and track each department’s activities with a single reporting system, allowing for easy analysis of performance statistics and assurance that nothing is overlooked.

Benefit #8: Customized Reports & BI

Business is all about understanding the past and using that knowledge to make better decisions in the present. Statistics and analytics are essential for this, as they provide us with the data and insights needed to make informed choices. We can tailor our reports to our exact needs and integrate our business intelligence tools to gain even more statistics and insights.

Benefit #9: Boost Sales Through Enhanced CRM

Businesses may build a CRM that fits their unique demands and operations by deploying a custom ERP system. Their sales teams will be more efficient and productive thanks to our unique CRM system and platform. Several businesses are currently looking toward the use of consumer-level ERP solutions. The business may expand its marketing efforts and enhance the conversion of leads into sales by creating an ERP system that incorporates customer and CRM data. A smooth cooperation between the many departments involved in sales will be made possible by the custom ERP’s complete integration with all of the business’s sales platforms and assets. With this technology in place, sales teams may use customer behavior and purchase history information more effectively.

How to Formulate Clear Requirements for Custom ERP Development

Below are some identified necessary factors to create a custom ERP system that meets your individual needs. They include outlining your core requirements, considering how the system will fit with existing processes, and determining its scalability. With the help of this information, you can build the ideal custom ERP system for your business with confidence.

Functionality 

If you are looking to implement an ERP system, it is necessary to consider the functionality that it needs to include. To make this process easier, you can examine the features of your current system and identify what can be improved. Additionally, it is also beneficial to discuss with your team and ask questions such as: 

What processes can be automated for resource optimization?
What visibility or excessive reporting does each department require? Which information do different departments need access to? 
Why is a custom ERP system being considered in the first place? 

Answering these questions will help you to determine the functionality that needs to be included and assign user experience in the future.

Industry Research

Success depends on being current with developments in your industry. Research for most successful competitors and the top-performing apps for your sector. Learn about the benefits and potential pitfalls of these technologies by watching demos and reading reviews.  To ensure that your expectations are realistic and your custom ERP system meets the specific needs of your business. It is best to engage an experienced team of custom ERP development professionals to conduct a comprehensive market analysis.

Integration Requirements

The key to successful development is integrating your custom ERP system with external applications. The development team will have a clear knowledge of the project scope if you gather all of the software applications that each department in your company uses. Systems for managing client relationships, inventories, online storefronts, project management tools, accounting software, and more are examples of apps that should be on the list. Your development team will be better equipped to provide you with accurate time and cost estimates if you have this information available.

Deployment Type

Businesses must choose between an on-premises and cloud option considering custom ERP development. On-premise software is installed on the company’s hardware, while cloud-based systems are hosted on the vendor’s server and accessed online. In the past, most companies opted for on-premise ERP systems cloud solutions have become increasingly popular due to the numerous advantages they offer:

faster implementation, 
lower initial costs, 
less responsibility for security, 
easier collaboration between teams, 
automated updates provided by the vendor.

Cost of ERP Software Development 

Custom ERP solutions can range significantly in cost depending on the complexity and scope of the project. Factors that influence the total cost of a project include the number of people on the development team, the features and integrations required, and the duration of the project. Generally, custom ERP development projects can cost anywhere between $25,000 to $5,000,000, making it a more expensive option when compared to the average cost of a third-party system for resource planning which typically runs from $9,000 per user for mid-sized and large-scale enterprises respectively. The most crucial factors:

Size of Company

The size of your business will affect the amount of time and effort needed to create an ERP system. As the size of your business grows, the demand for a robust and comprehensive custom ERP system increases. This entails a more complex organizational structure, various business processes, and the need to integrate multiple software products across departments. To meet these demands, an experienced ERP development provider must be engaged to create a custom ERP system with all the required modules, integrations, and features. This will ensure that the custom ERP system is tailored to meet the specific needs of your business, no matter how large it may be.

Tech Stack 

When considering custom ERP development, there is a distinction between open-source software and proprietary solutions. Open-source software is free to the public, while proprietary solutions require the provider to pay a subscription fee for the platform. Additionally, the more sophisticated the technology, the more costly the developer’s time will be. When selecting the right custom ERP solution for your business, it is important to consider the cost implications of open-source versus proprietary solutions, as well as the complexity of the technology. This will ensure that you get the most suitable custom ERP system for your business needs.

Development Team 

For a comprehensive and versatile custom ERP system, it is necessary to hire a larger team of specialists, which will help to reduce the overall implementation timeline. Generally, the cost of a medium-complexity custom ERP solution is around $40,000 per module. This cost is comparable for testing, deployment, and data migration, so for a custom ERP system with 5 modules, the total cost will be approximately $250,000. To ensure that your custom ERP system is cost-efficient, it is important to accurately estimate the amount of time and resources that will be needed.

Summary

Custom ERP development allows companies to have a tailored system that meets their specific needs, allowing them to manage resources more efficiently, streamline their workflows, and modernize the enterprise. By investing in a customized solution, companies can ensure that their unique requirements are taken into account, as well as facilitate the adaptation of staff to the new system and avoid overspending on unnecessary features. 

Custom ERP with Flatlogic

Flatlogic Platform offers an easy way to generate a custom ERP solution with full control over the source code and scalability. With full control over the source code, you can make sure that you have the right features, scalability, and performance to match your business needs. Plus, with no-code development, you don’t need to be an expert programmer to make the necessary changes – making it easier to scale and customize as your business grows. With Flatlogic Platform, you have the flexibility to create a custom ERP solution that is tailored to your needs, while still having the scalability of more traditional development.

How to Create Custom ERP with Flatlogic Platform?

Using the Flatlogic Full-Stack Generator you can create CRUD and static applications in a few minutes. To start using the Platform, you need to register on the Flatlogic website. Clicking the “Sign in” button in the header will allow you to register for a Flatlogic account.

Step 1. Choosing the Tech Stack

In this step, you’re setting the name of your application and choosing the stack: Frontend, Backend, and Database.

Step 2. Choosing the Starter Template

In this step, you’re choosing the design of the web app.

Step 3. Schema Editor

In this step, you can create your database schema from scratch, import an existing schema or select one of the suggested schemas. 

To import your existing database, click the Import SQL button and select your .sql file. After that, your database will be opened in the Schema Editor where you can further edit your data (add/edit/delete entities).

If you are not familiar with database design and find it difficult to understand what tables are, we have prepared some ready-made sample schemas of real applications that you can modify for your application:

E-commerce app;
Time tracking app;
Book store;
Chat (messaging) app;
Blog.

Or, you can define a database schema and add a description by clicking on the “Generate with AI” button. You need to type the application’s description in the text area and hit “Send”. The application’s schema will be ready in around 15 seconds. You may either hit deploy immediately or review the structure to make manual adjustments.

Next, you can connect your GitHub and push your application code there. Or skip this step by clicking the Finish and Deploy button and in a few minutes, your application will be generated.

The post Custom ERP System: Benefits, Requirements & Cost of Development appeared first on Flatlogic Blog.

Organizations that interface with online payments are continuously monitoring and guarding against fraudulent activity. Transactional fraud usually presents itself as discrete data points, making it challenging to identify multiple actors involved in the same group of transactions. Even a single actor operating over a period of time can be hard to detect. Visibility is key to prevent fraud incidents from occurring and to give meaningful knowledge of the activities within your environment to data, security, and operations engineers.

Understanding the connections between individual data points can reduce the time for customers to detect and prevent fraud. You can use a graph database to store transaction information along with the relationships between individual data points. Analyzing those relationships through a graph database can uncover patterns difficult to identify with relational tables. Fraud graphs enable customers to find common patterns between transactions, such as phone numbers, locations, and origin and destination accounts. Additionally, combining fraud graphs with full text search provides additional benefits as it can simplify analysis and integration with existing applications.

In our solution, financial analysts can upload graph data, which gets automatically ingested into the Amazon Neptune graph database service and replicated into Amazon OpenSearch Service for analysis. Data ingestion is automated with Amazon Simple Storage Service (Amazon S3) and Amazon Simple Queue Service (Amazon SQS) integration. We do data replication through AWS Lambda functions and AWS Step Functions for orchestration. The design is using open source tools and AWS Managed Services to build resources and is available in this https://github.com/aws-samples/neptune-fraud-detection-with-opensearch GitHub repository under an MIT-0 license. You will use Terraform and Docker to deploy the architecture, and will be able to send search requests to the system to explore the dataset.

Solution overview

This solution takes advantage of native integration between AWS services for scalability and performance, as well as the Neptune-to-OpenSearch Service replication pattern described in Neptune’s official documentation.

The process for this solution consists of the following steps, also shown in the architecture diagram here:

Financial analyst uploads graph data files to an Amazon S3 bucket.

Note: The data files are in a Gremlin load data format (CSV) and can include vertex files and edge files.

The action of the upload invokes a PUT object event notification with a destination set to an Amazon SQS queue.
The SQS queue is configured as an AWS Lambda event source, which invokes a Lambda function.
This Lambda function sends an HTTP request to an Amazon Neptune database to load data stored in an S3 bucket.
The Neptune database reads data from the S3 endpoint defined in the Lambda request and loads the data into the graph database.
An Amazon EventBridge rule is scheduled to run every 5 minutes. This rule targets an AWS Step Functions state machine to create a new execution.
The Neptune Poller step function (state machine) replicates the data in the Neptune database to an OpenSearch Service cluster.
Note: The Neptune Poller step function is responsible for continually syncing new data after the initial data upload using Neptune Streams.

User can access the replicated data from the Neptune database with Amazon OpenSearch Service.
Note: A Lambda function is invoked to send a search request or query to an OpenSearch Service endpoint to get results.

Prerequisites

To implement this solution, you must have the following prerequisites:

An AWS account with local credentials is configured. For more information, check the documentation on configuration and credential file settings.
The latest version of the AWS Command Line Interface (AWS CLI).
An IAM user with Git credentials.
A Git client to clone the source code provided.
A Bash shell.

Docker installed on your localhost.

Terraform installed on your localhost.

Deploying the Terraform templates

The solution is available in this GitHub repository with the following structure:

data: Contains a sample dataset to be used with the solution for demonstration purposes. Information on fictional transactions, identities and devices is represented in files within the nodes/ folder, and relationships between them are represented in files in the edges/ folder.
terraform: This folder contains the Terraform modules to deploy the solution.
documents: This folder contains the architecture diagram image file of the solution.

Create a local directory called NeptuneOpenSearchDemo and clone the source code repository:

mkdir -p $HOME/NeptuneOpenSearchDemo

cd $HOME/NeptuneOpenSearchDemo

git clone https://github.com/aws-samples/neptune-fraud-detection-with-opensearch.git

Change directory into the Terraform directory:

cd $HOME/NeptuneOpenSearchDemo neptune-fraud-detection-with-opensearch /terraform

Make sure that the Docker daemon is running:

docker info

If the previous command outputs an error that is unable to connect to the Docker daemon, start Docker and run the command again.

Initialize the Terraform folder to install required providers:

terraform init

The solution is deployed on us-west-2 by default. The user can change this behavior by modifying the variable “region” in variables.tf file.

Deploy the AWS services:

terraform apply -auto-approve

Note: Deployment will take around 30 minutes due to the time necessary to provision the Neptune and OpenSearch Service clusters.

To retrieve the name of the S3 bucket to upload data to:

aws s3 ls | grep “neptunestream-loader.*d$”

Upload node data to the S3 bucket obtained in the previous step:

aws s3 cp $HOME/NeptuneOpenSearchDemo/neptune-fraud-detection-with-opensearch /data s3:// neptunestream-loader-us-west-2-123456789012 –recursive

Note: This is a sample dataset for demonstration purposes only created from the IEEE-CIS Fraud Detection dataset.

Test the solution

After the solution is deployed and the dataset is uploaded to S3, the dataset can be retrieved and explored through a Lambda function that sends a search request to the OpenSearch Service cluster.

Confirm the Lambda function that sends a request to OpenSearch was deployed correctly:

aws lambda get-function –function-name NeptuneStreamOpenSearchRequestLambda –-query ‘Configuration.[FunctionName, State]’

Invoke the Lambda function to see all records present in OpenSearch that are added from Neptune:

aws lambda invoke –function-name NeptuneStreamOpenSearchRequestLambda response.json

The results of the Lambda invocation are stored in the response.json file. This file contains the total number of records in the cluster and all records ingested up to that point. The solution stores records in the index amazon_neptune. An example of a node with device information looks like this:

Cleaning up

To avoid incurring future charges, clean up the resources deployed in the solution:

terraform destroy –auto-approve

The command will output information on resources being destroyed.

Destroy complete! Resources: 101 destroyed.

Conclusion

Fraud graphs are complementary to other techniques organizations can use to detect and prevent fraud. The solution presented in this blog post reduces the time financial analysts would take to access transactional data by automating data ingestion and replication. It also improves performance for systems with growing volumes of data when compared to executing a large number of insert statements or other API calls.