software-engineering Archives - Online Code Generator

Announcing Snapchange: An Open Source KVM-backed Snapshot Fuzzing Framework

Posted on May 10, 2023May 21, 2023 by Mark FeehilyComments Off

Fuzz testing or fuzzing is a commonly used technique for discovering bugs in software, and is useful in many different domains. However, the task of configuring a target to be fuzzed can be a laborious one, often involving refactoring large code bases to enable fuzz testing.

Today we are happy to announce Snapchange, a new open source project to make snapshot-based fuzzing much easier. Snapchange enables a target binary to be fuzzed with minimal modifications, providing useful introspection that aids in fuzzing. Snapchange is a Rust framework for building fuzzers that replay physical memory snapshots in order to increase efficiency and reduce complexity in fuzzing many types of targets. Snapchange utilizes the features of the Linux kernel’s built-in virtual machine manager known as kernel virtual machine or KVM. While Snapchange is agnostic to the target operating system, the included snapshot mechanism focuses on Linux-based targets for gathering the necessary debug information.

Snapchange started as an experiment by the AWS Find and Fix (F2) open source security research team to explore the potential of using KVM in enabling snapshot fuzzing. Snapshot fuzzing is a growing area of interest and experimentation among security researchers. Snapchange has now grown into a project that aims to provide a friendly experience for researchers and developers to experiment with snapshot fuzzing. Snapchange is one of a number of tools and techniques used by the F2 team in its research efforts aimed at creating a secure and trustworthy open source supply chain for AWS and its customers. We have found it sufficiently useful that we are sharing it with the broader research community.

Snapchange is available today under the Apache License 2.0 via GitHub. AWS F2 team is actively supporting Snapchange and has plans for new features, but we hope to engage the security research community to produce a more richly-featured and robust tool over the longer term. We welcome pull requests on GitHub and look forward to discussions that help enable future research via the project. In this blog post we’ll walk through a set of tutorials you’ll find in the repository to help provide a deeper understanding of Snapchange.

Note: Snapchange operates within a Linux operating system but requires direct access to underlying KVM primitives. Thus, it is compatible with EC2 bare metal instance types, which run without a hypervisor, but not with EC2 virtualized instances. While we provide an EC2 AMI to make it easier to get started by launching on a bare metal instance (more information on that provided below), users are free to run Snapchange in other environments that meet the basic hardware requirements.

What is snapshot fuzzing?

Fuzzing uncovers software issues by monitoring how the system behaves while processing data, especially data provided as an input to the system. Fuzzing attempts to answer the question: What happens when the data is structured in a way that is outside the scope of what the software expects to receive? If the software is bug-free, there should be no input, no matter how inappropriate or corrupt, that causes it to crash. All input data should be properly validated and either pass validation and be used, or be rejected with a well-defined error result. Any input data that causes the software to crash shows a flaw and a potential weakness in the software. For example, fuzzing an application that renders JPEGs could involve mutating a sample JPEG file and opening the mutated JPEG in the application. If the application crashes or otherwise behaves in unexpected ways, this mutated file might have uncovered an issue.

The chosen mutations, however, are not truly random. We typically guide a fuzzer using “coverage” techniques. Coverage measures what code path an input from the fuzzer has caused to be executed in the target, and is used to automatically guide a fuzzer to modify its subsequent inputs so that the execution path in the target is changed to include previously-untested portions of the target’s code. Information about the sections of code in the target that were previously executed are cached in an input corpus, and that information is used to guide new inputs to explore additional code paths. In this way, variations on the same inputs will be applied in the expectation of discovering more previously untested code sections in the target, until all parts of the target code which can be reached by a possible execution path have been tested.

A snapshot is a pairing of a physical memory dump of a running VM and its accompanying register state. Fuzzing with a snapshot enables granular execution in order to reach code blocks that are traditionally difficult to fuzz without the complexities of managing state within the target. The only information needed by Snapchange in order to continue the execution of the target in a virtual machine is the snapshot itself. Prior work exploring this technique include brownie, falkervisor, chocolate_milk, Nyx, and what the fuzz. Most of these other tools require booting into a custom hypervisor on bare metal or with a modified KVM and kernel module. Snapchange can be used in environments where booting into a custom hypervisor isn’t straightforward. As noted, it can also be used on EC2 on bare metal instances that boot without any hypervisor at all.

How Snapchange Works

Snapchange fuzzes a target by injecting mutated data in the virtual machine and provides a breakpoint-based hooking mechanism, real-time coverage reports in a variety of formats (such as Lighthouse and LCOV), and single-step traces useful for debugging. With Snapchange, you can fuzz a given physical memory snapshot across multiple CPU cores in parallel, while monitoring for crashing states such as a segmentation fault or a call to an Address Sanitizer report.

While Snapchange doesn’t care how a snapshot is obtained, it includes one method which uses a patched QEMU instance via the included qemu_snapshot utility. This snapshot is then used as the initial state of a KVM virtual machine to fuzz a target.

The fuzzing loop starts by initializing the memory of a whole KVM virtual machine with the physical memory and register state of the snapshot. Snapchange then gives the user the ability to write a mutated input in the loaded guest’s memory. The virtual machine is then executed until a crash, timeout, or reset event occurs. At this point, the virtual machine will revert back to a clean state. The guest memory is restored to the original snapshot’s memory in preparation for the next input case. In order to avoid writing the entire snapshot memory on every reset, only pages that were modified during execution are restored. This significantly reduces the amount of memory which needs to be restored, speeding up the fuzzing cycle, and allowing more time to be spent fuzzing the target.

This ability to arbitrarily reset guest memory enables precise choices when harnessing a fuzz target. With snapshots, the harnessing effort involves discovering where in memory the relevant input resides. For example, instead of having to rewrite a networked application to take input packets from command line or stdin, we can use a debugger to break immediately after a recv call. Pausing execution at this point, we can document the buffer that was read into, for example address 0x6000_0000_0100 , and take a snapshot of the system with this memory address in mind. Once the snapshot is loaded via Snapchange, we can write a mutated input packet to address 0x6000_0000_0100 and continue executing the target as if it were a real packet. This precisely mimics what would happen if a corrupt or malicious packet was read off the network in a real-world scenario.

Experimenting with Snapchange

Snapchange, along with several example targets, can be found on GitHub. Because Snapchange relies on KVM for executing a snapshot, Snapchange must be used on a machine that has KVM access. Currently, Snapchange only supports x64 hosts and snapshots. As previously noted, Snapchange can be used in Amazon EC2 on a wide variety of .metal instances based on Intel processors, for example, a c6i.metal instance. There is also a public AMI containing Snapchange, with the examples pre-built and pre-snapshotted. The pre-built AMI is ami-008dec48252956ad5 in the US-East-2 region. For more information about using an AMI, check out the Get started with Amazon EC2 Linux instances tutorial. You can also install Snapchange in your own environment if you have access to supported hardware.

This blog will go over the first example in the Snapchange repository to demonstrate some of the features provided. For a more step-by-step walk-through, check out the full tutorial in the README for the 01_getpid example here.

Example target

We’ll start with the first example in Snapchange to demonstrate some of its features.

There are two goals for this target:

The input data buffer must solve for the string fuzzmetosolveme!

The return value from getpid() must be modified to be 0xdeadbeef

// harness/example1.c

void fuzzme(char* data) {
int pid = getpid();

   // Correct solution: data == “fuzzmetosolveme!”, pid == 0xdeadbeef
   if (data[0] == ‘f’)
   if (data[1] == ‘u’)
   if (data[2] == ‘z’)
   if (data[3] == ‘z’)
   if (data[4] == ‘m’)
   if (data[5] == ‘e’)
   if (data[6] == ‘t’)
   if (data[7] == ‘o’)
   if (data[8] == ‘s’)
   if (data[9] == ‘o’)
   if (data[10] == ‘l’)
   if (data[11] == ‘v’)
   if (data[12] == ‘e’)
   if (data[13] == ‘m’)
   if (data[14] == ‘e’)
   if (data[15] == ‘!’) {
   pid = getpid();
   if (pid == 0xdeadbeef) {
            // BUG
   *(int*)0xcafecafe = 0x41414141;
   }
   }

return;
}

When taking the snapshot, we logged that the input buffer being fuzzed is located at 0x555555556004.

SNAPSHOT Data buffer: 0x555555556004

It is the fuzzer’s job to write an input test case to address 0x5555_5555_6004 to begin fuzzing. Let’s look at how Snapchange handles coverage with breakpoints.

Coverage Breakpoints

Snapchange gathers its coverage of a target using breakpoints. In the snapshot directory, an optional .covbps file containing virtual addresses in the guest can be created. Because the snapshot is static, we can use hard coded memory addresses as part of the fuzzing process. During initialization, a breakpoint is inserted into the guest memory at every address found in the coverage breakpoint file. If any coverage breakpoint is hit, it means the current input executed a new piece of the target for the first time. The input is saved into the input corpus for future use and the breakpoint is removed. This removal of coverage breakpoints when they are encountered, means that the fuzzer only pays for the cost of the coverage breakpoint once.

One approach using these coverage breakpoints is to trigger on new basic blocks from the control flow graph of a target. There are a few utility scripts included in Snapchange to gather these basic blocks using Binary Ninja, Ghidra, and radare2.

The example coverage breakpoint file of the basic blocks found in example1.bin is in snapshot/example1.bin.covbps

$ head ./snapshot/example1.bin.ghidra.covbps

0x555555555000
0x555555555014
0x555555555016
0x555555555020
0x555555555070
0x555555555080
0x555555555090
0x5555555550a0
0x5555555550b0
0x5555555550c0

Writing a fuzzer

To begin fuzzing with Snapchange, we can write the fuzzer specific for this target in Rust.

// src/fuzzer.rs

#[derive(Default)]
pub struct Example1Fuzzer;

impl Fuzzer for Example1Fuzzer {
type Input = Vec<u8>; // [0]
const START_ADDRESS: u64 = 0x5555_5555_5344;
const MAX_INPUT_LENGTH: usize = 16; // [1]
const MAX_MUTATIONS: u64 = 2; // [3]

fn set_input(&mut self, input: &Self::Input, fuzzvm: &mut FuzzVm<Self>) -> Result<()> {
// Write the mutated input
fuzzvm.write_bytes_dirty(VirtAddr(0x5555_5555_6004), CR3, input)?; // [2]

Ok(())
}
…
}

A few notes about this fuzzer:

The fuzzer uses input of type Vec<u8> ([0]). This tells Snapchange to provide the default mutation strategies for a vector of bytes.

Note: This is an abstract type, so the fuzzer can provide a custom mutator/generator if they choose.

The maximum length of a generated input will be 16 bytes ([1])
The fuzzer is passed a mutated Vec<u8> in set_input. This input is then written to the address of the buffer logged during the snapshot (0x5555_5555_6004) via the call to write_bytes_dirty ([2]).

Note: This address is from the printf(“SNAPSHOT Data buffer: %pn”, data); line in the harness

The fuzzer will apply, at most, two mutations per input case ([3])

Snapchange provides an entry point to the main command-line utility that takes an abstract Fuzzer, like the one we have written. This will be the entry point for our fuzzer as well.

// src/main.rs

fn main() {
snapchange_main::<fuzzer::Example1Fuzzer>().expect(“Error in Example 1”);
}

Building this we can verify that the project and snapshot directories are set up properly by attempting to translate the starting instruction pointer address from the snapshot. Snapchange provides a project translate command used for doing virtual to physical memory translations from the snapshot and attempting to disassemble the bytes found at the read physical address. We can disassemble from fuzzme function in the snapshot with the following:

$ cargo run -r — project translate fuzzme

With the confirmation that the project’s directory structure is set up properly, we can begin fuzzing!

Starting fuzzing!

Snapchange has a fuzz command which can execute across a configurable number of cores in parallel. To begin fuzzing, Snapchange will start a number of virtual machines with the physical memory found in the snapshot directory. Snapchange will then choose an input from the current corpus (or generate one if one doesn’t exist), mutate it with a variety of techniques, and then write it into the guest via the set_input() function we wrote. If any new coverage has been seen, the mutated input will be saved in the corpus for future use. If a crash is found, the crashing input will be saved for further analysis.

The example is looking for the password fuzzmetosolveme! by checking each byte in the input one at a time. This pattern creates a new location for coverage to find for each byte. If the mutation randomly finds the next byte in the password, that input is saved in the corpus to be used later to discover the next byte, until the entire password is uncovered.

We began fuzzing with 8 cores for this example.

$ cargo run -r — fuzz –cores 8

The fuzz terminal user interface (TUI) is brought up with several pieces of information used to monitor the fuzzing:

Execution time
Basic core statistics for number of executions per second overall and per core
Amount of coverage seen
Number of crashes seen
Average number of dirty pages needed to reset a guest
Number of cores currently alive
Basic coverage graph

The TUI also includes performance information about where time is being spent in the fuzzer as well as information about the reasons a virtual machine is exiting. This information is useful to have for understanding if the fuzzer is actually spending relevant time fuzzing or if the fuzzer is doing extraneous computation that is causing a performance degradation.

For example, this fuzz run is only spending 14% of the total execution time in the guest virtual machine fuzzing the target. For some targets, this could present an opportunity to improve the performance of the fuzzer. Ideally, we want the fuzzer to be working in the guest virtual machine as much as possible. This test case is so small, though, that this number is to be expected, but it is still useful to keep in mind for more complex targets.

Lastly, there is a running list of recently-hit coverage to present a quick glance at what the fuzzer has recently uncovered in the target.

When fuzzing, the current coverage state seen by the fuzzer is written to disk, in real time, in a variety of formats: raw addresses, module+offset for usage in tools like Lighthouse, and (if debug information is available) LCOV format used for graphically annotating source code with this coverage information. This allows the developer or researcher to review the coverage to understand what the fuzzer is actually accomplishing to help them iterate on the fuzzer for potentially better results.

LCOV coverage displayed mid-fuzz session

Coverage displayed using Lighthouse in Binary Ninja

After some time, the fuzzer finds the correct input string to solve the first part of the target. We can look at the current corpus of the fuzzer in ./snapshot/current_corpus.

$ xxd snapshot/current_corpus/c2b9b72428f4059c
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 66 75 7a 7a 6d 65 74 6f ┊ 73 6f 6c 76 65 6d 65 21 │fuzzmeto┊solveme!│
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

Snapchange hooks

With the password discovered, the second half of the target revolves around getting getpid() to return an arbitrary value. This value isn’t expected to be returned from getpid(), but we can use Snapchange’s introspection features to force this result to happen. Snapchange includes breakpoint callbacks as a technique to introspect and modify the guest, such as by patching functions. Here is one example of forcing getpid() to always return the value 0xdeadbeef for our fuzzer.

fn breakpoints(&self) -> Option<&[Breakpoint<Self>]> {
Some(&[
Breakpoint {
lookup: BreakpointLookup::SymbolOffset(“libc.so.6!__GI___getpid”, 0x0),
bp_type: BreakpointType::Repeated,
bp_hook: |fuzzvm: &mut FuzzVm<Self>, _input, _fuzzer| {
// Set the return value to 0xdeadbeef
fuzzvm.set_rax(0xdead_beef); // [0]

// Fake an immediate return from the function by setting RIP to the
// value popped from the stack (this assumes the function was entered
// via a `call`)
fuzzvm.fake_immediate_return()?; // [1]

// Continue execution
Ok(Execution::Continue) // [2]
},
}
])
}

The fuzzer sets a breakpoint on the address for the symbol libc.so.6!__GI___getpid. When the breakpoint is triggered, the bp_hook function is called with the guest virtual machine (fuzzvm) as an argument. The return value for the function is stored in register rax, so we can set the value of rax to 0xdeadbeef via fuzzvm.set_rax(0xdeadbeef) [0]. We want the function to immediately return and not continue executing getpid(), so we fake the returning of the function by calling fuzzvm.fake_immediate_return() [1] to set the instruction pointer to the value on the top of the stack and Continue execution of the guest at this point [2] (rather than forcing the guest to reset).

We aren’t restricted to user space breakpoints. We could also force getpid() to return 0xdeadbeef by patching the call in the kernel in __task_pid_nr_ns. At offset 0x83 in __task_pid_nr_ns, we patch the moment the PID is read from memory and returned to the user from the kernel.

/*
// Single step trace from `cargo run -r — trace ./snapshot/current_corpus/c2b9b72428f4059c`
INSTRUCTION 1162 0xffffffff810d0ed3 0x6aa48000 | __task_pid_nr_ns+0xb3
mov eax, dword ptr [rbp+0x50]
EAX:0x0
[RBP:0xffff88806c91c000+0x50=0xffff88806c91c050 size:UInt32->0xe6]]
[8b, 45, 50]
*/
Breakpoint {
lookup: BreakpointLookup::SymbolOffset(“__task_pid_nr_ns”, 0xb3),
bp_type: BreakpointType::Repeated,
bp_hook: |fuzzvm: &mut FuzzVm<Self>, _input, _fuzzer| {
// The instruction retrieving the PID is
// mov eax, dword ptr [rbp+0x50]
// Write the 0xdeadbeef value into the memory at `rbp + 0x50`

// Get the current `rax` value
let rbp = fuzzvm.rbp();
let val: u32 = 0xdeadbeef;

// Write the wanted 0xdeadbeef in the memory location read in the
// kernel
fuzzvm.write_bytes_dirty(VirtAddr(rbp + 0x50), CR3, &val.to_le_bytes())?;

// Continue execution
Ok(Execution::Continue)
},
},

With getpid patched, we can continue fuzzing the target and check the Crashes tab in the TUI.

This looks like we’ve detected a segmentation fault (SIGSEGV) for address 0xcafecafe from the bug found in the target:

// BUG
*(int*)0xcafecafe = 0x41414141;

Single Step Traces

With a crash in hand, Snapchange can give us a single step trace using the crash as an input.

$ cargo run -r — trace ./snapshot/crashes/SIGSEGV_addr_0xcafecafe_code_AddressNotMappedToObject/c2b9b72428f4059c

This will give the state of the system at the time of the reset as well as the single step trace of the execution path. Notice that the guest reset on the force_sig_fault kernel function. This function is hooked by Snapchange to monitor for crashing states.

The single step trace is written to disk containing all instructions executed as well as the register state during each instruction. The trace includes:

Decoded instruction
State of the involved registers and memory for the given instruction
Assembly bytes for the instruction (useful for patching)
Source code where this assembly originated (if debug information is available)

What’s next for Snapchange?

The team is excited to hear from you and the community at large. We have ideas for more features and other analysis that can aid in fuzzing efforts and are interested in hearing what features the community is looking for in their fuzzing workflows. We’d also love feedback from you about your experience writing fuzzers using Snapchange on Snapchange’s GitHub. If this blog has sparked your curiosity, check out the other real-world examples included in the Snapchange repository.

10 ways to build applications faster with Amazon CodeWhisperer

Posted on April 26, 2023May 21, 2023 by Mark FeehilyComments Off

Amazon CodeWhisperer is a powerful generative AI tool that gives me coding superpowers. Ever since I have incorporated CodeWhisperer into my workflow, I have become faster, smarter, and even more delighted when building applications. However, learning to use any generative AI tool effectively requires a beginner’s mindset and a willingness to embrace new ways of working.

Best practices for tapping into CodeWhisperer’s power are still emerging. But, as an early explorer, I’ve discovered several techniques that have allowed me to get the most out of this amazing tool. In this article, I’m excited to share these techniques with you, using practical examples to illustrate just how CodeWhisperer can enhance your programming workflow. I’ll explore:

Typing less
Generating functions using code
Generating functions using comments
Generating classes
Implementing algorithms
Writing unit tests
Creating sample data
Simplifying regular expressions
Learning third-party code libraries faster
Documenting code

Before we begin

If you would like to try these techniques for yourself, you will need to use a code editor with the AWS Toolkit extension installed. VS Code, AWS Cloud9, and most editors from JetBrains will work. Refer to the CodeWhisperer “Getting Started” resources for setup instructions.

CodeWhisperer will present suggestions automatically as you type. If you aren’t presented with a suggestion, you can always manually trigger a suggestion using the Option + C (Mac) or Alt + C (Windows) shortcut. CodeWhisperer will also sometimes present you with multiple suggestions to choose from. You can press the → and ← keys to cycle through all available suggestions.

The suggestions CodeWhisperer offers are non-deterministic, which means you may receive slightly different suggestions than the ones shown in this article. If you receive a suggestion that doesn’t suit your needs, try cycling through all available suggestions using the arrow keys or try rephrasing any code comment prompts slightly. Even punctuation changes can produce fresh suggestions.

The examples in this article are written in JavaScript and TypeScript. However, the techniques I’ll demonstrate are equally suited to many of the programming languages supported by CodeWhisperer. Now, onto the examples!

Typing less

At the most basic level, CodeWhisperer can be used as a supercharged auto-complete, significantly reducing your typing when writing individual lines or blocks of code. To use CodeWhisperer in this way, just write code as you normally would. CodeWhisperer will make automatic suggestions as you type. Accept suggestions when they’re helpful, and ignore suggestions when they’re not. In my experience, using CodeWhisperer in this way alone reduces my typing by 50% or more.

Generating functions

Automatically generating function bodies from signatures

Skilled programmers use clear, descriptive names for their functions and parameters. If you follow this practice in your own programming, CodeWhisperer will have all the context it needs to implement many functions for you.

After I enter the function signature below, CodeWhisperer generates the function body for me.

User input (JavaScript):

function standardDeviation(numbers)

Result:

function standardDeviation(numbers) {
let mean = numbers.reduce((a, b) => a + b) / numbers.length;
let variance =
numbers.reduce((a, b) => a + Math.pow(b – mean, 2)) / numbers.length;
return Math.sqrt(variance);
}

Generating functions from comments

Sometimes a good function signature alone isn’t enough to convey the desired behavior of a function. In those cases, you can use a detailed code comment as a prompt to CodeWhisperer. I recommend using your language’s doc comment delimiter when using this technique.

User Input (JavaScript):

/**
* Given a list that contains some numbers and strings, format them into a string
* in which the numbers are prepended with “#” and the strings are wrapped in
* double quotes.
*/

Result:

/**
* Given a list that contains some numbers and strings, format them into a string
* in which the numbers are prepended with “#” and the strings are wrapped in
* double quotes.
*/
function formatList(list) {
return list
.map((item) => {
if (typeof item === “number”) {
return `#${item}`;
} else {
return `”${item}”`;
}
})
.join(“, “);
}

Generating classes

You can generate full or partial class implementations from a single code comment. The comment should describe the key properties and behaviors of the class. Simple descriptions are often all that is required.

User input (TypeScript):

/**
* A folder has a name, children, and an optional parent. A child can’t be added
* twice.
*/

Result:

After I write the above block comment, CodeWhisperer provides multiple suggestions, including the following implementation, complete with documentation comments for each method!

/**
* A folder has a name, children, and an optional parent. A child can’t be added
* twice.
*/
export class Folder {
name: string;
children: Folder[];
parent?: Folder;

constructor(name: string, parent?: Folder) {
this.name = name;
this.children = [];
this.parent = parent;
}

/**
* Add a child to the folder.
* @param child The child to add.
* @returns True if the child was added, false if it was already present.
*/
addChild(child: Folder): boolean {
if (this.children.includes(child)) {
return false;
}
this.children.push(child);
child.parent = this;
return true;
}

/**
* Remove a child from the folder.
* @param child The child to remove.
* @returns True if the child was removed, false if it wasn’t present.
*/
removeChild(child: Folder): boolean {
const index = this.children.indexOf(child);
if (index === -1) {
return false;
}
this.children.splice(index, 1);
child.parent = undefined;
return true;
}
}

Implementing algorithms

CodeWhisperer enables you to implement well-known algorithms with ease. Simply tell CodeWhisperer which algorithm you want to apply to a particular coding problem, and it will oblige.

User input (JavaScript):

const actors = [ … ];

// Sort the actors by age using the merge sort algorithm.

Result:

const actors = [ … ];

// Sort the actors by age using the merge sort algorithm.
function mergeSort(arr) {
if (arr.length < 2) {
return arr;
}

const middle = Math.floor(arr.length / 2);
const left = arr.slice(0, middle);
const right = arr.slice(middle);

return merge(mergeSort(left), mergeSort(right));
}

function merge(left, right) {
const result = [];

while (left.length && right.length) {
if (left[0].age < right[0].age) {
result.push(left.shift());
} else {
result.push(right.shift());
}
}

while (left.length) {
result.push(left.shift());
}

while (right.length) {
result.push(right.shift());
}

return result;
}

console.log(mergeSort(actors));

Writing unit tests

CodeWhisperer accelerates the implementation of unit tests. Not only is it able to implement the logic for most of the explicit unit tests I ask for, but it often suggests unit tests for edge cases that hadn’t even occurred to me.

When I want CodeWhisperer to implement a unit test, I start with a declaration of the condition I’m testing and the expected result as in the following example.

User input (JavaScript):

describe(“Folder.addChild()”, () => {
it(“when adding an existing folder should not add duplicate”

Result:

describe(“Folder.addChild()”, () => {
it(“when adding an existing folder should not add duplicate”, () => {
const folder = new Folder(“root”);
const child = new Folder(“child”);
folder.addChild(child);
folder.addChild(child);
expect(folder.children.length).toBe(1);
});
});

After you’ve written a test or two, CodeWhisperer is likely to suggest more tests, even without explicit prompting.

Tip: You’ll notice that my test description includes 1) the function I’m testing, 2) the condition I’m testing, and 3) what I expect the result to be. Following this three-part pattern is a great way to ensure CodeWhisperer has the context it needs to implement high quality test logic.

Creating sample data

One of the delightful discoveries I’ve made is that CodeWhisperer can generate dummy data—or really any code that has a repeating pattern. As an illustration, for the following list I entered only the first “user” object as an example, and CodeWhisperer helpfully continued the pattern.

User input (JavaScript):

const fakeUsers = [
{ name: “User 1”, id: “user1” },

Result:

const fakeUsers = [
{ name: “User 1”, id: “user1” },
{ name: “User 2”, id: “user2” },
{ name: “User 3”, id: “user3” },
{ name: “User 4”, id: “user4” },
{ name: “User 5”, id: “user5” },
{ name: “User 6”, id: “user6” }
]

It even works for generating random but plausible values like the names seen below.

User input (JavaScript):

const fakeUsers = [
{ name: “Sally”, id: “user1” },

Result:

const fakeUsers = [
{ name: “Sally”, id: “user1” },
{ name: “Bob”, id: “user2” },
{ name: “Joe”, id: “user3” },
{ name: “Jane”, id: “user4” },
{ name: “Jill”, id: “user5” },
{ name: “Jack”, id: “user6” }
]

Simplifying regular expressions

While CodeWhisperer can’t translate natural language into custom regular expressions (yet), it does know a lot of commonly used regular expressions. This saves you time by putting these regular expressions at your fingertips whenever you need them.

Let’s look at one notoriously complex but common regular expression.

User input (JavaScript):

/**
* Function that validates whether a string is a valid email address.
*/

Result:

/**
* Function that validates whether a string is a valid email address.
*/
function validateEmail(email: string): boolean {
const re =
/^(([^<>()[]\.,;:s@”]+(.[^<>()[]\.,;:s@”]+)*)|(“.+”))@(([[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}])|(([a-zA-Z-0-9]+.)+[a-zA-Z]{2,}))$/;
return re.test(String(email).toLowerCase());
}

Learning third-party code libraries faster

CodeWhisperer helps accelerate my learning when I’m using an unfamiliar library, module, or framework. In the example below, I needed to write a utility function that would take a gzip compressed value coming from an API and translate it into a value that the rest of my code could understand. I knew the “pako” library would help me, but I had no idea how to use it. I just prompted CodeWhisperer with an import statement and a comment describing my goal.

import pako from “pako”;

/**
* Accepts a base64-encoded, gzip-compressed JSON string and returns
* a parsed JSON object.
*/

Result:

import pako from “pako”;

/**
* Accepts a base64-encoded, gzip-compressed JSON string and returns
* a parsed JSON object.
*/
export function decodeGzipJson(base64String) {
const compressed = atob(base64String);
const uncompressed = pako.inflate(compressed, { to: “string” });
return JSON.parse(uncompressed);
}

Documenting code

CodeWhisperer is capable of generating docstrings and comments for the code it generates, as well as for your existing code. For example, let’s say I want CodeWhisperer to document the matches() method of this FavoritesFilter TypeScript class I’ve implemented (I’ve omitted some implementation details for brevity).

class FavoritesFilter implements IAssetFilter {
…
matches(asset: Asset): boolean {
…
}
}

I can just type a doc comment delimiter (/** */) immediately above the method name and CodeWhisperer will generate the body of the doc comment for me.

Note: When using CodeWhisperer in this way you may have to manually trigger a suggestion using Option + C (Mac) or Alt + C (Windows).

class FavoritesFilter implements IAssetFilter {
…
/**
* Determines whether the asset matches the filter.
*/
matches(asset: Asset): boolean {
…
}
}

Conclusion

I hope the techniques above inspire ideas for how CodeWhisperer can make you a more productive coder. Install CodeWhisperer today to start using these time-saving techniques in your own projects. These examples only scratch the surface. As additional creative minds start applying CodeWhisperer to their daily workflows, I’m sure new techniques and best practices will continue to emerge. If you discover a novel approach that you find useful, post a comment to share what you’ve discovered. Perhaps your technique will make it into a future article and help others in the CodeWhisperer community enhance their superpowers.

Kris Schultz (he/him)

Kris Schultz has spent over 25 years bringing engaging user experiences to life by combining emerging technologies with world class design. In his role as 3D Specialist Solutions Architect, Kris helps customers leverage AWS services to power 3D applications of all sorts.

Maintaining Code Quality with Amazon CodeCatalyst Reports

Posted on February 22, 2023May 21, 2023 by Mark FeehilyComments Off

Amazon CodeCatalyst reports contain details about tests that occur during a workflow run. You can create tests such as unit tests, integration tests, configuration tests, and functional tests. You can use a test report to help troubleshoot a problem during a workflow.

Introduction

In prior posts in this series, I discussed reading The Unicorn Project, by Gene Kim, and how the main character, Maxine, struggles with a complicated Software Development Lifecycle (SDLC) after joining a new team. One of the challenges she encounters is the difficulties in shipping secure, functioning code without an automated testing mechanism. To quote Gene Kim, “Without automated testing, the more code we write, the more money it takes for us to test.”

Software Developers know that shipping vulnerable or non-functioning code to a production environment is to be avoided at all costs; the monetary impact is high and the toll it takes on team morale can be even greater. During the SDLC, developers need a way to easily identify and troubleshoot errors in their code.

In this post, I will focus on how developers can seamlessly run tests as a part of workflow actions as well as configure unit test and code coverage reports with Amazon CodeCatalyst. I will also outline how developers can access these reports to gain insights into their code quality.

Prerequisites

If you would like to follow along with this walkthrough, you will need to:

Have an AWS Builder ID for signing in to CodeCatalyst.
Belong to a CodeCatalyst space and have the Space administrator role assigned to you in that space. For more information, see Creating a space in CodeCatalyst, Managing members of your space, and Space administrator role.
Have an AWS account associated with your space and have the IAM role in that account. For more information about the role and role policy, see Creating a CodeCatalyst service role.

Walkthrough

As with the previous posts in the CodeCatalyst series, I am going to use the Modern Three-tier Web Application blueprint. Blueprints provide sample code and CI/CD workflows to help you get started easily across different combinations of programming languages and architectures. To follow along, you can re-use a project you created previously, or you can refer to a previous post that walks through creating a project using the Three-tier blueprint.

Once the project is deployed, CodeCatalyst opens the project overview. This view shows the content of the README file from the project’s source repository, workflow runs, pull requests, etc. The source repository and workflow are created for me by the project blueprint. To view the source code, I select Code → Source Repositories from the left-hand navigation bar. Then, I select the repository name link from the list of source repositories.

Figure 1. List of source repositories including Mythical Mysfits source code.

From here I can view details such as the number of branches, workflows, commits, pull requests and source code of this repo. In this walkthrough, I’m focused on the testing capabilities of CodeCatalyst. The project already includes unit tests that were created by the blueprint so I will start there.

From the Files list, navigate to web → src → components→ __tests__ → TheGrid.spec.js. This file contains the front-end unit tests which simply check if the strings “Good”, “Neutral”, “Evil” and “Lawful”, “Neutral”, “Chaotic” have rendered on the web page. Take a moment to examine the code. I will use these tests throughout the walkthrough.

Figure 2. Unit test for the front-end that test strings have been rendered properly.

Next, I navigate to the workflow that executes the unit tests. From the left-hand navigation bar, select CI/CD → Workflows. Then, find ApplicationDeploymentPipeline, expand Recent runs and select Run-xxxxx . The Visual tab shows a graphical representation of the underlying YAML file that makes up this workflow. It also provides details on what started the workflow run, when it started, how long it took to complete, the source repository and whether it succeeded.

Figure 3. The Deployment workflow open in the visual designer.

Workflows are comprised of a source and one or more actions. I examined test reports for the back-end in a prior post. Therefore, I will focus on the front-end tests here. Select the build_and_test_frontend action to view logs on what the action ran, its configuration details, and the reports it generated. I’m specifically interested in the Unit Test and Code Coverage reports under the Reports tab:

Figure 4. Reports tab showing line and branch coverage.

Select the report unitTests.xml (you may need to scroll). Here, you can see an overview of this specific report with metrics like pass rate, duration, test suites, and the test cases for those suites:

Figure 5. Detailed report for the front-end tests.

This report has passed all checks. To make this report more interesting, I’ll intentionally edit the unit test to make it fail. First, navigate back to the source repository and open web → src → components→ __tests__→TheGrid.spec.js. This test case is looking for the string “Good” so change it to say “Best” instead and commit the changes.

Figure 6. Front-End Unit Test Code Change.

This will automatically start a new workflow run. Navigating back to CI/CD → Workflows, you can see a new workflow run is in progress (takes ~7 minutes to complete).

Once complete, you can see that the build_and_test_frontend action failed. Opening the unitTests.xml report again, you can see that the report status is in a Failed state. Notice that the minimum pass rate for this test is 100%, meaning that if any test case in this unit test ever fails, the build fails completely.

There are ways to configure these minimums which will be explored when looking at Code Coverage reports. To see more details on the error message in this report, select the failed test case.

Figure 7. Failed Test Case Error Message.

As expected, this indicates that the test was looking for the string “Good” but instead, it found the string “Best”. Before continuing, I return to the TheGrid.spec.js file and change the string back to “Good”.

CodeCatalyst also allows me to specify code and branch coverage criteria. Coverage is a metric that can help you understand how much of your source was tested. This ensures source code is properly tested before shipping to a production environment. Coverage is not configured for the front-end, so I will examine the coverage of the back-end.

I select Reports on the left-hand navigation bar, and open the report called backend-coverage.xml. You can see details such as line coverage, number of lines covered, specific files that were scanned, etc.

Figure 8. Code Coverage Report Succeeded.

The Line coverage minimum is set to 70% but the current coverage is 80%, so it succeeds. I want to push the team to continue improving, so I will edit the workflow to raise the minimum threshold to 90%. Navigating back to CI/CD → Workflows → ApplicationDeploymentPipeline, select the Edit button. On the Visual tab, select build_backend. On the Outputs tab, scroll down to Success Criteria and change Line Coverage to 90%.

Figure 9. Configuring Code Coverage Success Criteria.

On the top-right, select Commit. This will push the changes to the repository and start a new workflow run. Once the run has finished, navigate back to the Code Coverage report. This time, you can see it reporting a failure to meet the minimum threshold for Line coverage.

Figure 10. Code Coverage Report Failed.

There are other success criteria options available to experiment with. To learn more about success criteria, see Configuring success criteria for tests.

Cleanup

If you have been following along with this workflow, you should delete the resources you deployed so you do not continue to incur charges. First, delete the two stacks that CDK deployed using the AWS CloudFormation console in the AWS account you associated when you launched the blueprint. These stacks will have names like mysfitsXXXXXWebStack and mysfitsXXXXXAppStack. Second, delete the project from CodeCatalyst by navigating to Project settings and choosing Delete project.

Summary

In this post, I demonstrated how Amazon CodeCatalyst can help developers quickly configure test cases, run unit/code coverage tests, and generate reports using CodeCatalyst’s workflow actions. You can use these reports to adhere to your code testing strategy as a software development team. I also outlined how you can use success criteria to influence the outcome of a build in your workflow. In the next post, I will demonstrate how to configure CodeCatalyst workflows and integrate Software Composition Analysis (SCA) reports. Stay tuned!

About the authors:

Imtranur Rahman

Imtranur Rahman is an experienced Sr. Solutions Architect in WWPS team with 14+ years of experience. Imtranur works with large AWS Global SI partners and helps them build their cloud strategy and broad adoption of Amazon’s cloud computing platform.Imtranur specializes in Containers, Dev/SecOps, GitOps, microservices based applications, hybrid application solutions, application modernization and loves innovating on behalf of his customers. He is highly customer obsessed and takes pride in providing the best solutions through his extensive expertise.

Wasay Mabood

Wasay is a Partner Solutions Architect based out of New York. He works primarily with AWS Partners on migration, training, and compliance efforts but also dabbles in web development. When he’s not working with customers, he enjoys window-shopping, lounging around at home, and experimenting with new ideas.

Bringing JavaScript to WebAssembly

Posted on February 10, 2023May 21, 2023 by Mark FeehilyComments Off

#625 — February 10, 2023

Read on the Web

It looked quiet at first but wow, what an epic week this turned out to be. There’s a lot to chew on here, and we even have a variety of bonus items at the very end of the issue. Enjoy!
__
Your editor, Peter Cooper

JavaScript Weekly

Speeding Up the JS Ecosystem: It’s ESLint’s Turn — Last year we featured an article from the same author about how he was finding, and fixing, low-hanging performance fruit in popular JavaScript projects. He’s back, and he’s found a lot of potential for savings in ESLint this time.

Marvin Hagemeister

The Future (and the Past) of the Web is Server Side Rendering — It’s fair to say the Deno folks have some skin in this game, but nonetheless this is a neat brief history of server-side rendering and why they feel it’s the right approach for modern web development.

Andy Jiang (Deno)

Monitoring Your NestJS Application with AppSignal — With AppSignal, you can monitor your NestJS app with ease and rely on OpenTelemetry to handle third-party instrumentations. AppSignal even provides helper functions to help you build comprehensive custom instrumentation. A box of ? included!

AppSignal sponsor

Ten Web Development Trends in 2023 — Following the State of JS survey results Robin takes a considered look at new web dev trends that we should be paying attention to this year, and why they matter.

Robin Wieruch

Bringing JavaScript to WebAssembly for Shopify Functions — As much as this is focused on a specific use case at Shopify, this is a fascinating look at how they’re integrating JavaScript and WebAssembly under tight constraints. They also talk about Javy, a JS to WebAssembly toolchain being built at Shopify that lets you run JS code on a WASM-embedded JS runtime.

Surma (Shopify)

Google Touts Web-Based Machine Learning with TensorFlow.js

Richard MacManus (The New Stack)

IN BRIEF:

? Time to celebrate — a recent survey allegedly found that JavaScript applications ‘have fewer flaws’ than Java and .NET ones. So there you go.

Honeypot’s highly anticipated ▶️ React.js documentary drops later today – it’ll probably be out by the time you read this.

Vanilla List is a directory of ‘vanilla’ JavaScript controls and plugins.

▶️ Evan You tells us what to expect in 2023 from Vue.js.

The Scala.js project is celebrating its ten year anniversary – it’s now a mature way to build Web projects using Scala, if you prefer.

? Vue.js Live is a JavaScript event taking place both in London and online on May 12 & 15. From the same folks as the also forthcoming JSNation conference.

A history of criticisms levelled at React.

RELEASES:

Eleventy / 11ty 2.0
↳ Popular Node.js static site generator.

pnpm 7.27 – The efficient package manager.

RxDB 14.0 – Offline-first, reactive database.

? Articles & Tutorials

Design Patterns in TypeScript — OO-inspired patterns aren’t for everyone or every use case, but this is a fantastic catalog of examples, complete with diagrams and explanations, if you need to learn to tell apart factory methods from decorators, facades, or proxies.

Refactoring Guru

Resumable React: How To Use React Inside Qwik — Building React apps without ever loading React in the user’s browser? “Sounds too good to be true? Let’s see how this works.”

Yoav Ganbar

Did You Know That You’re Already a Distributed Systems Developer?

Temporal Technologies sponsor

Build a Hacker News Client using Alpine.js — Alpine.js is a thin and elegant reactivity library that lets you add dynamic functionality to your site directly in markup. This is a short and sweet practical example of what you can quickly do with it.

Salai Vedha Viradhan

▶ TypeScript Speedrun: A Crash Course for Beginners — If you want to pick up TypeScript and would find a video guide useful, this is for you. Matt has become well known recently for his educational TypeScript tweets and videos, and this is another good one that flies through the basics. (23 minutes.)

Matt Pocock

Using Notion as a Headless CMS with Nuxt

Trent Brew

The Options API vs Composition API in Vue.js

Charles Allotey

? Code & Tools

Bookmarklet Editor: Easily Work on JavaScript Bookmarklets — Useful because who can remember the exact syntax for a bookmarklet? ? This also can instantly convert code to and from bookmarklet form and includes some examples in the help section (click the big ? to get all the details).

Marek Gibney

Breakpoints and console.log Is the Past, Time Travel Is the Future — 15x faster JavaScript debugging than with breakpoints and console.log, now with support for Vitest.

Wallaby.js sponsor

Yup 1.0: Super Simple Object Schema Validation — Define a schema, transform a value to match, assert the shape of an existing value, or both. Very extensive docs here.

Jason Quense

Material React Table: A Full-Featured React Table Component — Built upon Material UI 5 and TanStack Table 8. The docs include lots of interactive examples.

Kevin Van Cott

BlockNote: Notion-Style Block-Based Text Editor — Built on top of Prosemirror and Tiptap, this is for you if you like the way the Notion note-taking service’s text editor feels. There’s a live demo.

Yousef

TresJS: Build 3D Experiences with Vue.js — Create 3D scenes with Vue components and Three.js. Think React-three-fiber but Vue flavored.

Alvaro Sabu

depngn: Find Out if Dependencies Support a Given Node.js Version — A CLI tool that establishes whether or not the dependencies in your package.json will work against a specified version of Node.

OmbuLabs

Open-Source JS Form Libraries to Automate Your Form Workflow — Self-host SurveyJS to configure and modify multiple forms, convert them to fillable PDF files, and analyze collected data in interactive dashboards.

SurveyJS sponsor

Lawnmower: Build VR Scenes with Custom HTML Tags — A web component library that leans on Three.js and aims “to make building a basic VR website as easy to make as your first HTML site”.

Gareth Marland

Electron 23.0 Released — The popular cross platform JavaScript, HTML + CSS desktop app framework gets bumped up to Node 18.12.1, Chromium 110, and V8 11.0. Windows 7/8/8.1 support has also been dropped, so we might start to see those versions of Windows lose the support of a lot of Electron based apps soon.

Electron Core Team

Run: Run User-Provided Code in a Web Worker

SLASHD Analytics

? Jobs

Software Engineer (Backend) — Join our “kick ass” team. Our software team operates from 17 countries and we’re always looking for more exceptional engineers.

Sticker Mule

Find JavaScript Jobs with Hired — Hired makes job hunting easy-instead of chasing recruiters, companies approach you with salary details up front. Create a free profile now.

Hired

QUICK RELEASES:

vue-easytable 2.23
↳ A data table/grid control for Vue.js. (Demo.)

React-Custom-Scroll 5.0
↳ Customize the browser scroll bar. (Demo.)

react-jsonschema-form 5.1
↳ Component to build Web forms from JSON Schema.

AlaSQL.js 3.1
↳ JavaScript-based SQL database.

jest-puppeteer 7.0
↳ Run tests using Jest & Puppeteer.

MDX 2.3
↳ Markdown for the component era.

? The Bonus Round

✈️ Watching someone wrestle with Python and JavaScript to fly (virtual) planes with Microsoft Flight Simulator tickled me a lot.

A beautiful WebGL2-based fluid simulation. It’s even happy on mobile. Pretty!

Go-like channels in 10 lines of JavaTypeScript..?

? Misko Hevery: “useSignal() is the future of web frameworks and is a better abstraction than useState(), which is showing its age.” (source)

Mike Pennisi asks: when is an object property not a property?

Do you use Postgres at all? Check out Postgres Weekly – one of our sister newsletters. So much is going on in the Postgres space lately and it’s a great way to keep up.

Unlock the power of EC2 Graviton with GitLab CI/CD and EKS Runners

Posted on January 14, 2023May 21, 2023 by Mark FeehilyComments Off

Many AWS customers are using GitLab for their DevOps needs, including source control, and continuous integration and continuous delivery (CI/CD). Many of our customers are using GitLab SaaS (the hosted edition), while others are using GitLab Self-managed to meet their security and compliance requirements.

Customers can easily add runners to their GitLab instance to perform various CI/CD jobs. These jobs include compiling source code, building software packages or container images, performing unit and integration testing, etc.—even all the way to production deployment. For the SaaS edition, GitLab offers hosted runners, and customers can provide their own runners as well. Customers who run GitLab Self-managed must provide their own runners.

In this post, we’ll discuss how customers can maximize their CI/CD capabilities by managing their GitLab runner and executor fleet with Amazon Elastic Kubernetes Service (Amazon EKS). We’ll leverage both x86 and Graviton runners, allowing customers for the first time to build and test their applications both on x86 and on AWS Graviton, our most powerful, cost-effective, and sustainable instance family. In keeping with AWS’s philosophy of “pay only for what you use,” we’ll keep our Amazon Elastic Compute Cloud (Amazon EC2) instances as small as possible, and launch ephemeral runners on Spot instances. We’ll demonstrate building and testing a simple demo application on both architectures. Finally, we’ll build and deliver a multi-architecture container image that can run on Amazon EC2 instances or AWS Fargate, both on x86 and Graviton.

Figure 1. Managed GitLab runner architecture overview.

Let’s go through the components:

Runners

A runner is an application to which GitLab sends jobs that are defined in a CI/CD pipeline. The runner receives jobs from GitLab and executes them—either by itself, or by passing it to an executor (we’ll visit the executor in the next section).

In our design, we’ll be using a pair of self-hosted runners. One runner will accept jobs for the x86 CPU architecture, and the other will accept jobs for the arm64 (Graviton) CPU architecture. To help us route our jobs to the proper runner, we’ll apply some tags to each runner indicating the architecture for which it will be responsible. We’ll tag the x86 runner with x86, x86-64, and amd64, thereby reflecting the most common nicknames for the architecture, and we’ll tag the arm64 runner with arm64.

Currently, these runners must always be running so that they can receive jobs as they are created. Our runners only require a small amount of memory and CPU, so that we can run them on small EC2 instances to minimize cost. These include t4g.micro for Graviton builds, or t3.micro or t3a.micro for x86 builds.

To save money on these runners, consider purchasing a Savings Plan or Reserved Instances for them. Savings Plans and Reserved Instances can save you up to 72% over on-demand pricing, and there’s no minimum spend required to use them.

Kubernetes executors

In GitLab CI/CD, the executor’s job is to perform the actual build. The runner can create hundreds or thousands of executors as needed to meet current demand, subject to the concurrency limits that you specify. Executors are created only when needed, and they are ephemeral: once a job has finished running on an executor, the runner will terminate it.

In our design, we’ll use the Kubernetes executor that’s built into the GitLab runner. The Kubernetes executor simply schedules a new pod to run each job. Once the job completes, the pod terminates, thereby freeing the node to run other jobs.

The Kubernetes executor is highly customizable. We’ll configure each runner with a nodeSelector that makes sure that the jobs are scheduled only onto nodes that are running the specified CPU architecture. Other possible customizations include CPU and memory reservations, node and pod tolerations, service accounts, volume mounts, and much more.

Scaling worker nodes

For most customers, CI/CD jobs aren’t likely to be running all of the time. To save cost, we only want to run worker nodes when there’s a job to run.

To make this happen, we’ll turn to Karpenter. Karpenter provisions EC2 instances as soon as needed to fit newly-scheduled pods. If a new executor pod is scheduled, and there isn’t a qualified instance with enough capacity remaining on it, then Karpenter will quickly and automatically launch a new instance to fit the pod. Karpenter will also periodically scan the cluster and terminate idle nodes, thereby saving on costs. Karpenter can terminate a vacant node in as little as 30 seconds.

Karpenter can launch either Amazon EC2 on-demand or Spot instances depending on your needs. With Spot instances, you can save up to 90% over on-demand instance prices. Since CI/CD jobs often aren’t time-sensitive, Spot instances can be an excellent choice for GitLab execution pods. Karpenter will even automatically find the best Spot instance type to speed up the time it takes to launch an instance and minimize the likelihood of job interruption.

Deploying our solution

To deploy our solution, we’ll write a small application using the AWS Cloud Development Kit (AWS CDK) and the EKS Blueprints library. AWS CDK is an open-source software development framework to define your cloud application resources using familiar programming languages. EKS Blueprints is a library designed to make it simple to deploy complex Kubernetes resources to an Amazon EKS cluster with minimum coding.

The high-level infrastructure code – which can be found in our GitLab repo – is very simple. I’ve included comments to explain how it works.

// All CDK applications start with a new cdk.App object.
const app = new cdk.App();

// Create a new EKS cluster at v1.23. Run all non-DaemonSet pods in the
// `kube-system` (coredns, etc.) and `karpenter` namespaces in Fargate
// so that we don’t have to maintain EC2 instances for them.
const clusterProvider = new blueprints.GenericClusterProvider({
version: KubernetesVersion.V1_23,
fargateProfiles: {
main: {
selectors: [
{ namespace: ‘kube-system’ },
{ namespace: ‘karpenter’ },
]
}
},
clusterLogging: [
ClusterLoggingTypes.API,
ClusterLoggingTypes.AUDIT,
ClusterLoggingTypes.AUTHENTICATOR,
ClusterLoggingTypes.CONTROLLER_MANAGER,
ClusterLoggingTypes.SCHEDULER
]
});

// EKS Blueprints uses a Builder pattern.
blueprints.EksBlueprint.builder()
.clusterProvider(clusterProvider) // start with the Cluster Provider
.addOns(
// Use the EKS add-ons that manage coredns and the VPC CNI plugin
new blueprints.addons.CoreDnsAddOn(‘v1.8.7-eksbuild.3’),
new blueprints.addons.VpcCniAddOn(‘v1.12.0-eksbuild.1’),
// Install Karpenter
new blueprints.addons.KarpenterAddOn({
provisionerSpecs: {
// Karpenter examines scheduled pods for the following labels
// in their `nodeSelector` or `nodeAffinity` rules and routes
// the pods to the node with the best fit, provisioning a new
// node if necessary to meet the requirements.
//
// Allow either amd64 or arm64 nodes to be provisioned
‘kubernetes.io/arch’: [‘amd64’, ‘arm64’],
// Allow either Spot or On-Demand nodes to be provisioned
‘karpenter.sh/capacity-type’: [‘spot’, ‘on-demand’]
},
// Launch instances in the VPC private subnets
subnetTags: {
Name: ‘gitlab-runner-eks-demo/gitlab-runner-eks-demo-vpc/PrivateSubnet*’
},
// Apply security groups that match the following tags to the launched instances
securityGroupTags: {
‘kubernetes.io/cluster/gitlab-runner-eks-demo’: ‘owned’
}
}),
// Create a pair of a new GitLab runner deployments, one running on
// arm64 (Graviton) instance, the other on an x86_64 instance.
// We’ll show the definition of the GitLabRunner class below.
new GitLabRunner({
arch: CpuArch.ARM_64,
// If you’re using an on-premise GitLab installation, you’ll want
// to change the URL below.
gitlabUrl: ‘https://gitlab.com’,
// Kubernetes Secret containing the runner registration token
// (discussed later)
secretName: ‘gitlab-runner-secret’
}),
new GitLabRunner({
arch: CpuArch.X86_64,
gitlabUrl: ‘https://gitlab.com’,
secretName: ‘gitlab-runner-secret’
}),
)
.build(app,
// Stack name
‘gitlab-runner-eks-demo’);

The GitLabRunner class is a HelmAddOn subclass that takes a few parameters from the top-level application:

// The location and name of the GitLab Runner Helm chart
const CHART_REPO = ‘https://charts.gitlab.io’;
const HELM_CHART = ‘gitlab-runner’;

// The default namespace for the runner
const DEFAULT_NAMESPACE = ‘gitlab’;

// The default Helm chart version
const DEFAULT_VERSION = ‘0.40.1’;

export enum CpuArch {
ARM_64 = ‘arm64’,
X86_64 = ‘amd64’
}

// Configuration parameters
interface GitLabRunnerProps {
// The CPU architecture of the node on which the runner pod will reside
arch: CpuArch
// The GitLab API URL
gitlabUrl: string
// Kubernetes Secret containing the runner registration token (discussed later)
secretName: string
// Optional tags for the runner. These will be added to the default list
// corresponding to the runner’s CPU architecture.
tags?: string[]
// Optional Kubernetes namespace in which the runner will be installed
namespace?: string
// Optional Helm chart version
chartVersion?: string
}

export class GitLabRunner extends HelmAddOn {
private arch: CpuArch;
private gitlabUrl: string;
private secretName: string;
private tags: string[] = [];

constructor(props: GitLabRunnerProps) {
// Invoke the superclass (HelmAddOn) constructor
super({
name: `gitlab-runner-${props.arch}`,
chart: HELM_CHART,
repository: CHART_REPO,
namespace: props.namespace || DEFAULT_NAMESPACE,
version: props.chartVersion || DEFAULT_VERSION,
release: `gitlab-runner-${props.arch}`,
});

this.arch = props.arch;
this.gitlabUrl = props.gitlabUrl;
this.secretName = props.secretName;

// Set default runner tags
switch (this.arch) {
case CpuArch.X86_64:
this.tags.push(‘amd64’, ‘x86’, ‘x86-64’, ‘x86_64’);
break;
case CpuArch.ARM_64:
this.tags.push(‘arm64’);
break;
}
this.tags.push(…props.tags || []); // Add any custom tags
};

// `deploy` method required by the abstract class definition. Our implementation
// simply installs a Helm chart to the cluster with the proper values.
deploy(clusterInfo: ClusterInfo): void | Promise<Construct> {
const chart = this.addHelmChart(clusterInfo, this.getValues(), true);
return Promise.resolve(chart);
}

// Returns the values for the GitLab Runner Helm chart
private getValues(): Values {
return {
gitlabUrl: this.gitlabUrl,
runners: {
config: this.runnerConfig(), // runner config.toml file, from below
name: `demo-runner-${this.arch}`, // name as seen in GitLab UI
tags: uniq(this.tags).join(‘,’),
secret: this.secretName, // see below
},
// Labels to constrain the nodes where this runner can be placed
nodeSelector: {
‘kubernetes.io/arch’: this.arch,
‘karpenter.sh/capacity-type’: ‘on-demand’
},
// Default pod label
podLabels: {
‘gitlab-role’: ‘manager’
},
// Create all the necessary RBAC resources including the ServiceAccount
rbac: {
create: true
},
// Required resources (memory/CPU) for the runner pod. The runner
// is fairly lightweight as it’s a self-contained Golang app.
resources: {
requests: {
memory: ‘128Mi’,
cpu: ‘256m’
}
}
};
}

// This string contains the runner’s `config.toml` file including the
// Kubernetes executor’s configuration. Note the nodeSelector constraints
// (including the use of Spot capacity and the CPU architecture).
private runnerConfig(): string {
return `
[[runners]]
[runners.kubernetes]
namespace = “{{.Release.Namespace}}”
image = “ubuntu:16.04”
[runners.kubernetes.node_selector]
“kubernetes.io/arch” = “${this.arch}”
“kubernetes.io/os” = “linux”
“karpenter.sh/capacity-type” = “spot”
[runners.kubernetes.pod_labels]
gitlab-role = “runner”
`.trim();
}
}

For security reasons, we store the GitLab registration token in a Kubernetes Secret – never in our source code. For additional security, we recommend encrypting Secrets using an AWS Key Management Service (AWS KMS) key that you supply by specifying the encryption configuration when you create your Amazon EKS cluster. It’s a good practice to restrict access to this Secret via Kubernetes RBAC rules.

To create the Secret, run the following command:

# These two values must match the parameters supplied to the GitLabRunner constructor
NAMESPACE=gitlab
SECRET_NAME=gitlab-runner-secret
# The value of the registration token.
TOKEN=GRxxxxxxxxxxxxxxxxxxxxxx

kubectl -n $NAMESPACE create secret generic $SECRET_NAME
–from-literal=”runner-registration-token=$TOKEN”
–from-literal=”runner-token=”

Building a multi-architecture container image

Now that we’ve launched our GitLab runners and configured the executors, we can build and test a simple multi-architecture container image. If the tests pass, we can then upload it to our project’s GitLab container registry. Our application will be pretty simple: we’ll create a web server in Go that simply prints out “Hello World” and prints out the current architecture.

Find the source code of our sample app in our GitLab repo.

In GitLab, the CI/CD configuration lives in the .gitlab-ci.yml file at the root of the source repository. In this file, we declare a list of ordered build stages, and then we declare the specific jobs associated with each stage.

Our stages are:

The build stage, in which we compile our code, produce our architecture-specific images, and upload these images to the GitLab container registry. These uploaded images are tagged with a suffix indicating the architecture on which they were built. This job uses a matrix variable to run it in parallel against two different runners – one for each supported architecture. Furthermore, rather than using docker build to produce our images, we use Kaniko to build them. This lets us build our images in an unprivileged container environment and improve the security posture considerably.
The test stage, in which we test the code. As with the build stage, we use a matrix variable to run the tests in parallel in separate pods on each supported architecture.

The assembly stage, in which we create a multi-architecture image manifest from the two architecture-specific images. Then, we push the manifest into the image registry so that we can refer to it in future deployments.

Figure 2. Example CI/CD pipeline for multi-architecture images.

Here’s what our top-level configuration looks like:

variables:
# These are used by the runner to configure the Kubernetes executor, and define
# the values of spec.containers[].resources.limits.{memory,cpu} for the Pod(s).
KUBERNETES_MEMORY_REQUEST: 1Gi
KUBERNETES_CPU_REQUEST: 1

# List of stages for jobs, and their order of execution
stages:
– build
– test
– create-multiarch-manifest
Here’s what our build stage job looks like. Note the matrix of variables which are set in BUILD_ARCH as the two jobs are run in parallel:
build-job:
stage: build
parallel:
matrix: # This job is run twice, once on amd64 (x86), once on arm64
– BUILD_ARCH: amd64
– BUILD_ARCH: arm64
tags: [$BUILD_ARCH] # Associate the job with the appropriate runner
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [“”]
script:
– mkdir -p /kaniko/.docker
# Configure authentication data for Kaniko so it can push to the
# GitLab container registry
– echo “{“auths”:{“${CI_REGISTRY}”:{“auth”:”$(printf “%s:%s” “${CI_REGISTRY_USER}” “${CI_REGISTRY_PASSWORD}” | base64 | tr -d ‘n’)”}}}” > /kaniko/.docker/config.json
# Build the image and push to the registry. In this stage, we append the build
# architecture as a tag suffix.
– >-
/kaniko/executor
–context “${CI_PROJECT_DIR}”
–dockerfile “${CI_PROJECT_DIR}/Dockerfile”
–destination “${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}-${BUILD_ARCH}”

Here’s what our test stage job looks like. This time we use the image that we just produced. Our source code is copied into the application container. Then, we can run make test-api to execute the server test suite.

build-job:
stage: build
parallel:
matrix: # This job is run twice, once on amd64 (x86), once on arm64
– BUILD_ARCH: amd64
– BUILD_ARCH: arm64
tags: [$BUILD_ARCH] # Associate the job with the appropriate runner
image:
# Use the image we just built
name: “${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}-${BUILD_ARCH}”
script:
– make test-container

Finally, here’s what our assembly stage looks like. We use Podman to build the multi-architecture manifest and push it into the image registry. Traditionally we might have used docker buildx to do this, but using Podman lets us do this work in an unprivileged container for additional security.

create-manifest-job:
stage: create-multiarch-manifest
tags: [arm64]
image: public.ecr.aws/docker/library/fedora:36
script:
– yum -y install podman
– echo “${CI_REGISTRY_PASSWORD}” | podman login -u “${CI_REGISTRY_USER}” –password-stdin “${CI_REGISTRY}”
– COMPOSITE_IMAGE=${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}
– podman manifest create ${COMPOSITE_IMAGE}
– >-
for arch in arm64 amd64; do
podman manifest add ${COMPOSITE_IMAGE} docker://${COMPOSITE_IMAGE}-${arch};
done
– podman manifest inspect ${COMPOSITE_IMAGE}
# The composite image manifest omits the architecture from the tag suffix.
– podman manifest push ${COMPOSITE_IMAGE} docker://${COMPOSITE_IMAGE}

Trying it out

I’ve created a public test GitLab project containing the sample source code, and attached the runners to the project. We can see them at Settings > CI/CD > Runners:

Figure 3. GitLab runner configurations.

Here we can also see some pipeline executions, where some have succeeded, and others have failed.

Figure 4. GitLab sample pipeline executions.

We can also see the specific jobs associated with a pipeline execution:

Figure 5. GitLab sample job executions.

Finally, here are our container images:

Figure 6. GitLab sample container registry.

Conclusion

In this post, we’ve illustrated how you can quickly and easily construct multi-architecture container images with GitLab, Amazon EKS, Karpenter, and Amazon EC2, using both x86 and Graviton instance families. We indexed on using as many managed services as possible, maximizing security, and minimizing complexity and TCO. We dove deep on multiple facets of the process, and discussed how to save up to 90% of the solution’s cost by using Spot instances for CI/CD executions.

Find the sample code, including everything shown here today, in our GitLab repository.

Building multi-architecture images will unlock the value and performance of running your applications on AWS Graviton and give you increased flexibility over compute choice. We encourage you to get started today.

About the author:

Michael Fischer

Michael Fischer is a Principal Specialist Solutions Architect at Amazon Web Services. He focuses on helping customers build more cost-effectively and sustainably with AWS Graviton. Michael has an extensive background in systems programming, monitoring, and observability. His hobbies include world travel, diving, and playing the drums.

Building .NET 7 Applications with AWS CodeBuild

Posted on January 14, 2023May 21, 2023 by Mark FeehilyComments Off

AWS CodeBuild is a fully managed DevOps service for building and testing your applications. As a fully managed service, there is no infrastructure to manage and you pay only for the resources that you use when you are building your applications. CodeBuild provides a default build image that contains the current Long Term Support (LTS) version of the .NET SDK.

Microsoft released the latest version of .NET in November. This release, .NET 7, includes performance improvements and functionality, such as native ahead of time compilation. (Native AoT)..NET 7 is a Standard Term Support release of the .NET SDK. At this point CodeBuild’s default image does not support .NET 7. For customers that want to start using.NET 7 right away in their applications, CodeBuild provides two means of customizing your build environment so that you can take advantage of .NET 7.

The first option for customizing your build environment is to provide CodeBuild with a container image you create and maintain. With this method, customers can define the build environment exactly as they need by including any SDKs, runtimes, and tools in the container image. However, this approach requires customers to maintain the build environment themselves, including patching and updating the tools. This approach will not be covered in this blog post.

A second means of customizing your build environment is by using the install phase of the buildspec file. This method uses the default CodeBuild image, and adds additional functionality at the point that a build starts. This has the advantage that customers do not have the overhead of patching and maintaining the build image.

Complete documentation on the syntax of the buildspec file can be found here:

https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html

Your application’s buildspec.yml file contains all of the commands necessary to build your application and prepare it for deployment. For a typical .NET application, the buildspec file will look like this:

“`
version: 0.2
phases:
build:
commands:
– dotnet restore Net7TestApp.sln
– dotnet build Net7TestApp.sln
“`

Note: This build spec file contains only the commands to build the application, commands for packaging and storing build artifacts have been omitted for brevity.

In order to add the .NET 7 SDK to CodeBuild so that we can build your .NET 7 applications, we will leverage the install phase of the buildspec file. The install phase allows you to install any third-party libraries or SDKs prior to beginning your actual build.

“`
install:
commands:
– curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin –channel STS
“`

The above command downloads the Microsoft install script for .NET and uses that script to download and install the latest version of the .NET SDK, from the Standard Term Support channel. This script will download files and set environment variables within the containerized build environment. You can use this same command to automatically pull the latest Long Term Support version of the .NET SDK by changing the command argument STS to LTS.

Your updated buildspec file will look like this:

“`
version: 0.2
phases:
install:
commands:
– curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin –channel STS
build:
commands:
– dotnet restore Net7TestApp/Net7TestApp.sln
– dotnet build Net7TestApp/Net7TestApp.sln
“`

Once you check in your buildspec file, you can start a build via the CodeBuild console, and your .NET application will be built using the .NET 7 SDK.

As your build runs you will see output similar to this:

“`
Welcome to .NET 7.0!
———————
SDK Version: 7.0.100
Telemetry
———
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to ‘1’ or ‘true’ using your favorite shell.

Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
—————-
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run ‘dotnet dev-certs https –trust’ (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
—————-
Write your first app: https://aka.ms/dotnet-hello-world
Find out what’s new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use ‘dotnet –help’ to see available commands or visit: https://aka.ms/dotnet-cli
————————————————————————————–
Determining projects to restore…
Restored /codebuild/output/src095190443/src/git-codecommit.us-east-2.amazonaws.com/v1/repos/net7test/Net7TestApp/Net7TestApp/Net7TestApp.csproj (in 586 ms).
[Container] 2022/11/18 14:55:08 Running command dotnet build Net7TestApp/Net7TestApp.sln
MSBuild version 17.4.0+18d5aef85 for .NET
Determining projects to restore…
All projects are up-to-date for restore.
Net7TestApp -> /codebuild/output/src095190443/src/git-codecommit.us-east-2.amazonaws.com/v1/repos/net7test/Net7TestApp/Net7TestApp/bin/Debug/net7.0/Net7TestApp.dll
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:04.63
[Container] 2022/11/18 14:55:13 Phase complete: BUILD State: SUCCEEDED
[Container] 2022/11/18 14:55:13 Phase context status code: Message:
[Container] 2022/11/18 14:55:13 Entering phase POST_BUILD
[Container] 2022/11/18 14:55:13 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/11/18 14:55:13 Phase context status code: Message:
“`

Conclusion

Adding .NET 7 support to AWS CodeBuild is easily accomplished by adding a single line to your application’s buildspec.yml file, stored alongside your application source code. This change allows you to keep up to date with the latest versions of .NET while still taking advantage of the managed runtime provided by the CodeBuild service.

About the author:

Tom Moore

Tom Moore is a Sr. Specialist Solutions Architect at AWS, and specializes in helping customers migrate and modernize Microsoft .NET and Windows workloads into their AWS environment.

12+ Best Node.js Frameworks for Web App Development in 2022

Posted on June 14, 2022May 21, 2023 by Mark Feehily0

Node.js is getting increasingly popular among developers, to the point where some developers call Node.js their primary choice for backend development. In this article, we review the 12 best Node.js web frameworks that we rate according to their popularity and unique toolkits for time and cost-efficiency.

Is Node.js a web framework?

So is Node.js a web framework? The most common way of referring to it is as a web framework. Still, Node.js is a JavaScript execution environment – a server-side platform for JavaScript code execution and portability. Instead, web frameworks focus on building features. A lot of developers have built Node.js web frameworks, such as Nest.js, Express.js, and other toolkits, for Node.js applications, providing a unique experience for software developers.

What are Node.js web frameworks?

Every web application technology offers different types of frameworks, all supporting a specific use case in the development lifecycle. Node.js web frameworks come in three types – Full-Stack Model-View-Controller (MVC), MVC, and REST API web frameworks.

Node.js web framework features

API of Node.js is asynchronous. You can use the Node.js server to move after a data request, rather than waiting for the API to return the information.
The code execution process of Node.js is faster compared to the reverse backend framework.
Node.js runs on a single-threaded model.
With Node.js web framework developers never face buffering issues because it transfers information by parts.
It is supported by Google’s Node.js runtime environment.

Through these features, it is clear to understand why developers more often choose Node.js for Backend development. Let’s take a closer look at each Node.js web framework.

NestJS

Github repo: https://github.com/nestjs/nest
License: MIT
Github stars: 47400

NestJS is object-oriented and functional-reactive programming (FRP), widely used for developing enterprise-level dynamic and scalable web solutions, being well featured with extensive libraries.

NestJS is based on TypeScript as its core programming language, but is also highly compatible with a JavaScript subset and easily integrated with other frameworks such as ExpressJS through a command-line interface.

Why use NestJS:

Modern CLI
functional-reactive programming
Multiple easy-to-use external libraries
Straightforward Angular compatibility

NestJS has a clean and modular architecture pattern aiding developers to build scalable and maintainable applications with ease.

Pros of NestJS:

Powerful but super friendly to work with
Fast development
Easy to understand documentation
Angular style syntax for the backend

NodeJS ecosystem
Typescript
Its easy to understand since it follows angular syntax
Good architecture
Integrates with Narwhal Extensions
Typescript makes it well integrated in vscode
Graphql support easy
Agnosticism
Easily integrate with others external extensions

ExpressJS

Github repo: https://github.com/expressjs/express
License: MIT
Github stars: 57200

ExpressJS is minimalistic, asynchronous, fast, and powerful and was launched in 2010. It’s beginner-friendly thanks to a low learning curve that requires only a basic understanding of the Node.js environment and programming skills. ExpressJS optimises client-to-server requests and observed user interaction via an API very quickly, and also helps you manage high-speed I/O operations.

Why use ExpressJS:

Enhanced content coordination
MVC architecture pattern
HTTP helpers
Asynchronous programming to support multiple independent operations

ExpressJS offers templating, robust routing, security and error handling, making it suitable for building enterprise or browser-based applications.

Pros of ExpressJS :

Simple
NodeJS
Javascript
High performance
Robust routing
Middlewares
Open source
Great community
Hybrid web applications
Well documented
Light weight

Meteor

Github repo: https://github.com/meteor/meteor
License: MIT
Github stars: 42900

Meteor is an open-source framework that was launched in 2012 that works best for teams who want to develop in a single language, being a full-featured Node.js web framework. Meteor is ideal for modern real-time applications as it facilitates instant data transfer between server and client.

Why use Meteor:

Cross-platform web framework
Rapid prototyping using the CLI
Extensive community support and open-source code
End-to-end solution
Seamless integration with other frameworks

The Meteor is an excellent option for those who are familiar with Javascript and prefer it. It’s a great one for both web and mobile app development as well. Meteor is great for applications that require a lot of updates that need to be sent out, even in a live environment.

Pros of Meteor :

Real-time
Full stack, one language
Best app dev platform available today
Data synchronization
Javascript
Focus on your product not the plumbing
Hot code pushes
Open source
Live page updates
Latency compensation
Ultra-simple development environment
Great for beginners
Smart Packages

KoaJS

Github repo: https://github.com/koajs/koa
License: MIT
Github stars: 32700

Koa has been called the next-generation Node.js web framework, and it’s one of the best of the bunch. Koa uses a stack-based approach to handling HTTP mediators, which makes it a great option for easy API development. Koa is similar to ExpressJS, so it’s fairly easy to switch from either one. Despite the same features and flexibility, Koa reduces the complexity of writing code even more.

Why use Koa:

Multi-level customisation
Considered a lightweight version of ExpressJS
Supplied with cascading middleware ( user experience personalisation)
Node mismatch normalization
Cleans caches and supports content and proxy negotiation

Use Koa when performance is the main focus of your web application. Koa is ahead of ExpressJS in some scenarios, so you can use it for large-scale projects.

Pros of Koa :

Async/Await
JavaScript
REST API

socket.io

Github repo:https://github.com/socketio/socket.io
License: MIT
Github stars: 55900

The socket is a Javascript library that works most effectively for real-time web applications. The socket is used when communication between real-time web clients and servers needs to be efficiently bidirectional.

Why use socket.io:

Binary support
Multiplexing support
Reliability
Auto-reconnection support
Auto-correction and error detection

The socket is a great choice when building real-time applications like video conferencing, chat rooms and multiplayer games with servers being required to send data out before it’s requested from the client-side.

Pros of socket :

Real-time
Event-based communication
NodeJS
WebSockets
Open source
Binary streaming
No internet dependency
Large community

TotalJS

Github repo: https://github.com/totaljs/
License: MIT
Github stars: n/a

TotalJS is a web framework that offers a CMS-like user experience and has almost all the functionality you need in a Node.js environment. The framework is a full open-source framework that provides developers with the ultimate flexibility. There are various options available for the framework, e.g. CMS, and HelpDesk. Through these options, your application will have more integration possibilities with the REST service and hyper-fast, low-maintenance, stable applications.

TotalJS is most well-known for its real-time, high-precision tracking in modern applications.

Pros of TotalJS:

Tracking in real-time
API Testing
Automatic project discovery
Compatibility with multiple databases
Flexibility to work with different frontend frameworks
Fast development and low cost of maintenance

SailsJS

Github repo: https://github.com/balderdashy/sails
License: MIT
Github stars: 22247

SailsJS is similar to the MVC architect pattern of web frameworks such as Ruby on Rails, and it supports modernized data-centric development. Compatible with all databases, at the same time it flexibly integrates Javascript frameworks. SailsJS is the most relevant framework for building high-quality custom applications. Its special code-writing policy helps reduce the code needed, allowing you to integrate npm modules while remaining flexible and open source.

Pros of SailsJS:

REST API auto-generation
Multiple security policies
Frontend agnosticism
Object Relational Mapping for framework databases compatibility
Supports ExpressJS integration for HTTP requests and socket.io for WebSockets

FeathersJS

Github repo: https://github.com/feathersjs/feathers
License: MIT
Github stars: 14000

FeathersJS is gaining popularity between website and application developers because it provides flexibility in development with react native as well as Node.js. It is a framework of microservices because it operates with more than one database, providing real-time functionality. FeathersJS makes it easier for web developers to code concretely and understandably.

Pros of FeathersJS:

Reusable services
Modern CLI
Automated RESTful APIs
Authentication and authorization plugins by default
Lightweight

FeathersJS natively supports all frontend technologies, and its database-agnostic is best performed in a Node.js environment because the web framework supports Javascript and Typescript. It allows you to create production-ready applications and real-time applications, and also REST APIs in just a few days.

hapi.dev

Github repo: https://github.com/hapijs/hapi
License: MIT
Github stars: 13900

Hapi is an open-source framework for web applications. It is well-known for proxy server development as well as REST APIs and some other desktop applications since the framework is robust and security-rich. It has a wealth of built-in plugins, therefore this means you don’t have to worry about running non-official middleware.

Pros of Hapi:

Extensive and scalable applications
Low overhead
Secure default settings
Rich ecosystem
Quick and easy bug fixes
Compatible with multiple databases
Compatible with Rest API and HTTPS proxy applications
Caching, authentication and input validation by default

AdonisJS

Github repo: https://github.com/adonisjs/core
License: MIT
Github stars: 12600

AdonisJS is a Model-View-Controller Node.js web framework based on a structural template repeating Laravel. The framework decreases the time required for development by focusing on core details such as out of the box web socket support, development speed and performance, lifecycle dependency management, and built-in modules for data validation, mailing, and authentication. Command-based coding structure and the interface is easy for developers to understand. The Node.js web framework uses the concepts of dependency injections through IoC or control inversion. It offers developers an organized structure for accessing all the components of the framework.

Pros of AdonisJS:

Organised template with folder structure
Easy user input validation.
Ability to write custom functional testing scripts
Support for Lucid object-relational mapping.
Threat protection such as cross-site forgery protection

Loopback

Github repo: https://github.com/loopbackio/loopback-next
License: MIT
Github stars: 4200

Loopback provides the best connection with any Node.js web framework and can be integrated with multiple API services. You can best use the platform to build REST APIs with minimal lead time. Loopback offers outstanding flexibility, interfacing with a broad range of browsers, devices, databases, and services. Framework’s structured code helps support application modules and speed of development. Loopback has the best documentation, allowing even beginners to work with it.

Pros of Loopback:

Comprehensive support for networked applications
The built-in client API explorer
High extensibility
Multiple database support
Clean and modular code
Full-stack development
Data storage, third-party access, and user management

Loopback is designed solely for creating powerful end-to-end APIs and handling requests for them.

DerbyJS

Github repo: https://github.com/derbyjs/derby
License: MIT
Github stars: 4622

DerbyJS is a full-stack web application development platform powered by Node.js technology. This framework uses the Model-View-Controller architecture with an easy-to-write nomenclature for coding. This framework is great for building real-time web applications since it allows essentially the same code to work on Node.js and in the browser. That way, you don’t have to worry about writing separate codes for the view part. DerbyJS decreases the delay in content delivery by rendering a client-side view on the server. Performing this makes the application SEO-friendly and improves the user experience.

Pros of DerbyJS:

Support for Racer Engine
Real-time conversion for data synchronization
Offline use and conflict resolution support

Version control
Client-side and server-side code sharing
Rendering client-side views on the server-side

Conclusion

Node.js web frameworks make application development easier with their enormous possibilities for the advancement of web and mobile application development. Under the conditions of increasingly growing technologies, a thorough investigation of project requirements and accessibility of resources is the key to choosing the right web framework that will provide the greatest results.

The post 12+ Best Node.js Frameworks for Web App Development in 2022 appeared first on Flatlogic Blog.

A pattern for dealing with #legacy code in c#

Posted on March 18, 2022May 21, 2023 by Mark Feehily0

static string legacy_code(int input)
{
// some magic process
const int magicNumber = 7;

var intermediaryValue = input + magicNumber;

return “The answer is ” + intermediaryValue;
}

When dealing with a project more than a few years old, the issue of legacy code crops up time and time again. In this case, you have a function that’s called from lots of different client applications, so you can’t change it without breaking the client apps.

I’m using the code example above to keep the illustration simple, but you have to imagine that this function “legacy_code(int)”, in reality, could be hundreds of lines long, with lots of quirks and complexities. So you really don’t want to duplicate it.

Now, imagine, that as an output, I want to have just the intermediary value, not the string “The answer is …”. My client could parse the number out of the string, but that’s a horrible extra step to put on the client.

Otherwise you could create “legacy_code_internal()” that returns the int, and legacy_code() calls legacy_code_internal() and adds the string. This is the most common approach, but can end up with a rat’s nest of _internal() functions.

Here’s another approach – you can tell me what you think :

static string legacy_code(int input, Action<int> intermediary = null)
{
// some magic process
const int magicNumber = 7;

var intermediaryValue = input + magicNumber;

if (intermediary != null) intermediary(intermediaryValue);

return “The answer is ” + intermediaryValue;
}

Here, we can pass an optional function into the legacy_code function, that if present, will return the intermediaryValue as an int, without interfering with how the code is called by existing clients.

A new client looking to use the new functionality could call;

int intermediaryValue = 0;
var answer = legacy_code(4, i => intermediaryValue = i);
Console.WriteLine(answer);
Console.WriteLine(intermediaryValue);

This approach could return more than one object, but this could get very messy.

Bringing a little slice of Developer Tools to Seq in 2022.1

Posted on March 14, 2022May 21, 2023 by Mark Feehily0

If your experience is anything like mine, debugging in production is as much about organizing information – clues, leads, sometimes frustrating dead-ends – as it is about unearthing it. Working through an issue, I need pen and paper, sticky notes, whiteboards and spreadsheets as much as I depend on searching logs.

Seq is never going to replace my favorite Lamy pen (a chrome Pico I’ve hung onto for absolutely years ?), but in 2022.1 there are two small improvements coming that should help with the information overload:

Variables as an alternative to copy-and-paste for keeping track of useful values, and
A redesigned (and searchable!) history makes it easier to jump back to an earlier search or query.

Search history doesn’t need much explanation (it’s now a tool window you can pop out in the signal bar), but variables are actually pretty interesting.

When GUIDs start to look alike…

The basic scenario looks like this:

Why does request cbc6f181ba66498092c78bdd4106e3f1 time out, while request c43df41457dd443a83f1bcf94353a3c7 succeeds?

Instead of copying those values back and forth to a text editor window, inevitably forgetting which is on the clipboard at any given moment, Seq 2022.1 supports assigning them to variable names:

select ‘cbc6f181ba66498092c78bdd4106e3f1’ into $failingRequest

You can use the resulting variable names in any expression where you would otherwise use the value:

The design of variables in Seq takes inspiration from the ubiquitous browser developer tools console: variables are somewhere to stash and inspect useful pieces of information while you work on a problem.

This could be a little confusing if you were expecting variables to behave more like those in procedural languages such as Transact-SQL or PL/pgSQL. Seq doesn’t support procedural programming, so there’s no way to, say, reassign a variable on each iteration of a loop. If you’re here looking for something along those lines, check out let bindings instead.

Quick variables

The green check-mark menu beside event properties can now be used to assign the value of a property to a new variable:

Seq will generate a variable name based on the property name. Variables can’t be directly renamed, but you can give a variable a more meaningful name by selecting its value into a new one:

select $requestId0 into $slowRequest

Variables in signals, alerts, and dashboards

Variables are tracked client-side, and passed back to the server by the Seq UI whenever they appear in a search or query. The value you assign to a variable doesn’t live any longer than your browser session.

This means that variable values aren’t available when signals are indexed, when alerts are checked, or when your dashboard is rendered on an overhead display. To avoid silently failing with incorrect results, Seq will reject all variable references that appear in these places:

Wrapping up the tour

So that’s it! Variables are a simple feature, but also part of an exciting direction that we’re going to explore further.

You can find out more about variables – the scenarios we’ve thought about, and some of the implementation details – in the Variables RFC.

Or, to jump in and try them against your next production debugging puzzle, pull the datalust/seq:preview Docker image, or download the 2022.1.*-pre MSI from the Seq downloads page. We can’t wait to hear how you go!

.NET IdempotentAPI 1.0.0 Release Candidate

Posted on February 4, 2022May 21, 2023 by Mark Feehily0

Introduction

A distributed system consists of multiple components located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system. Fault-tolerant applications can continue operating despite the system, hardware, and network faults of one or more components.

Idempotency in Web APIs ensures that the API works correctly (as designed) even when consumers (clients) send the same request multiple times. To simplify the integration of Idempotency in an API project, we could use the IdempotentAPI open-source NuGet library. IdempotentAPI implements an ASP.NET Core attribute (filter) to handle the HTTP write operations (POST and PATCH) to affect only once for the given request data and idempotency key.

In July 2021, we saw how the IdempotentAPI v0.1.0-beta in .NET Nakama (2021, July 4) provides an easy way to develop idempotent Web APIs in .NET Core. Since then, with the community’s help, several issues and improvements have been identified and implemented. The complete journey of the IdempotentAPI is available in the CHANGELOG.md file.

Now, the IdempotentAPI 1.0.0-RC-01 is available with many improvements ?✨. In the following sections, we will see the complete features, details regarding the improvements, the available NuGet packages, and instructions to start using the IdempotentAPI library quickly.

Features

⭐ Simple: Support idempotency in your APIs easily with three simple steps 1️⃣2️⃣3️⃣.
? Validations: Performs validation of the request’s hash-key to ensure that the cached response is returned for the same combination of Idempotency Key and Request to prevent accidental misuse.
? Use it anywhere!: IdempotentAPI targets .NET Standard 2.0. So, we can use it in any compatible .NET implementation (.NET Framework, .NET Core, etc.). Click here to see the minimum .NET implementation versions that support each .NET Standard version.
⚙ Configurable: Customize the idempotency in your needs.

Configuration Options (see the GitHub repository for more details)
Logging Level configuration.

? Caching Implementation based on your needs.

? DistributedCache: A build-in caching based on the standard IDistributedCache interface.
? FusionCache: A high-performance and robust cache with an optional distributed 2nd layer and advanced features.
… or you could use your implementation ?

Improvement Details

Improving Concurrent Requests Handling

The standard IDistributedCache interface doesn’t support a command to GetOrSet a cached value with atomicity. However, it defines the Get and Set methods. In our previous implementation, we used these two methods without locking (i.e. without grouping into a single logical operation). As a result, we had an issue with concurrent requests with the same idempotency key. The problem was that the controller action could be executed multiple times.

As we can observe in Figure 1, this issue happens when the API Server receives a second request (with the same idempotency key) before we flag the first idempotency key as Inflight (i.e., execution in progress). Thus, racing conditions occur when setting idempotency key as Inflight.

Figure 1. – The issue of executing the controller more than once when concurrent requests with the same idempotency key are performed.

To overcome this issue, we defined the IIdempotencyCache interface and implemented the GetOrSet method, which performs a lock (locally) for each idempotency key (see code below). In Figure 2, we can see how we used the GetOrSet method to execute the controller action only once on concurrent requests with the same idempotency key.

The idea is to use GetOrSet method to set an Inflight object with a dynamic unique id per request when the Get method returns Null (it doesn’t have a value) as a single logical operation. The second call of the GetOrSet will wait for the first call to complete. Thus, only the execution that receives its unique id can continue with the execution of the controller action.

Figure 2. – Concurrent requests with the same idempotency key, execute the controller action only once.

public byte[] GetOrSet(
string key,
byte[] defaultValue,
object? options = null,
CancellationToken token = default)
{
if (key is null)
{
throw new ArgumentNullException(nameof(key));
}

if (options is not null && options is not DistributedCacheEntryOptions)
{
throw new ArgumentNullException(nameof(options));
}

using (var valuelocker = new ValueLocker(key))
{
byte[] cachedData = _distributedCache.Get(key);
if (cachedData is null)
{
_distributedCache.Set(key, defaultValue, (DistributedCacheEntryOptions?)options);
return defaultValue;
}
else
{
return cachedData;
}
}
}

Caching as Implementation Detail

To overcome the concurrent requests with the same idempotency key issue, we defined the IIdempotencyCache interface and implemented the GetOrSet method. This is implemented in our build-in DistributedCache caching project, which is based on the standard IDistributedCache interface.

Our implementation provides basic caching functionality. However, by defining the IIdempotencyCache interface, our IdempotencyAPI logic becomes independent from the caching implementation. Thus, we can support other caching implementations with advanced features, such as the FusionCache.

FusionCache is a high-performance and robust caching .NET library with an optional distributed 2nd layer with advanced features, such as fail-safe mechanism, cache stampede prevention, fine-grained soft/hard timeouts with background factory completion, extensive customizable logging, and more.

BinaryFormatter is Obsolete

The BinaryFormatter serialization methods become obsolete from ASP .NET Core 5.0. In the IdempotentAPI project, the BinaryFormatter was used in the Utils.cs class for serialization and deserialization. As a result, our library was not working in .NET Core 5.0 and later versions unless we enabled the BinaryFormatterSerialization option in the .csproj file.

The recommended action based on the .NET documentation is to stop using BinaryFormatter and use a JSON or XML serializer. In our case, we used the Newtonsoft JsonSerializer, which can include type information when serializing JSON, and read this type of information when deserializing JSON to create the target object with the original types.

In the following JSON example, we can see how the type of information is included in the data.

{
“$type”: “System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]], System.Private.CoreLib”,
“Request.Method”: “POST”,
“Response.StatusCode”: 200,
“Response.Headers”: {
“$type”: “System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]], System.Private.CoreLib]], System.Private.CoreLib”,
“myHeader1”: {
“$type”: “System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]], System.Private.CoreLib”,
“$values”: [
“value1-1”,
“value1-2”
]
},
“myHeader2”: {
“$type”: “System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]], System.Private.CoreLib”,
“$values”: [
“value2-1”,
“value2-1”
]
}
}
}

Quick Start

Step 1: Register the Caching Storage

Storing-caching data is necessary for idempotency. Therefore, the IdempotentAPI library needs an implementation of the IIdempotencyCache to be registered in the Program.cs or Startup.cs file depending on the used style (.NET 6.0 or older). The IIdempotencyCache defines the caching storage service for the idempotency needs.

Currently, we support the following two implementations (see the following table). However, you can use your implementation ?. Both implementations support the IDistributedCache either as primary caching storage (requiring registration) or secondary (optional registration).

Thus, we can define our caching storage service in the IDistributedCache, such as in Memory, SQL Server, Redis, NCache, etc. See the Distributed caching in the ASP.NET Core article for more details about the available framework-provided implementations.

IdempotentAPI.Cache
Implementation
Support Concurrent Requests
Primary Cache
2nd-Level Cache
Advanced Features

DistributedCache (Default)
✔
IDistributedCache
❌
❌

FusionCache
✔
Memory Cache

✔(IDistributedCache)
✔

Choice 1 (Default): IdempotentAPI.Cache.DistributedCache

Install the IdempotentAPI.Cache.DistributedCache via the NuGet UI or the NuGet package manager console.

// Register an implementation of the IDistributedCache.
// For this example, we are using a Memory Cache.
services.AddDistributedMemoryCache();

// Register the IdempotentAPI.Cache.DistributedCache.
services.AddIdempotentAPIUsingDistributedCache();

Choice 2: Registering: IdempotentAPI.Cache.FusionCache

Install the IdempotentAPI.Cache.FusionCache via the NuGet UI or the NuGet package manager console. To use the advanced FusionCache features (2nd-level cache, Fail-Safe, Soft/Hard timeouts, etc.), configure the FusionCacheEntryOptions based on your needs (for more details, visit the FusionCache repository).

// Register the IdempotentAPI.Cache.FusionCache.
// Optionally: Configure the FusionCacheEntryOptions.
services.AddIdempotentAPIUsingFusionCache();

Tip: To use the 2nd-level cache, we should register an implementation for the IDistributedCache and register the FusionCache Serialization (NewtonsoftJson or SystemTextJson). For example, check the following code:

// Register an implementation of the IDistributedCache.
// For this example, we are using Redis.
services.AddStackExchangeRedisCache(options =>
{
options.Configuration = “YOUR CONNECTION STRING HERE, FOR EXAMPLE:localhost:6379”;
});

// Register the FusionCache Serialization (e.g. NewtonsoftJson).
// This is needed for the a 2nd-level cache.
services.AddFusionCacheNewtonsoftJsonSerializer();

// Register the IdempotentAPI.Cache.FusionCache.
// Optionally: Configure the FusionCacheEntryOptions.
services.AddIdempotentAPIUsingFusionCache();

Step 2: Decorate Response Classes as Serializable

The response Data Transfer Objects (DTOs) need to be serialized before caching. For that reason, we will have to decorate the relative DTOs as [Serializable]. For example, see the code below.

using System;

namespace WebApi_3_1.DTOs
{
[Serializable]
public class SimpleResponse
{
public int Id { get; set; }
public string Message { get; set; }
public DateTime CreatedOn { get; set; }
}
}

Step 3: Set Controller Operations as Idempotent

In your Controller class, add the following using statement. Then choose which operations should be Idempotent by setting the [Idempotent()] attribute, either on the controller’s class or each action separately. The following two sections describe these two cases. First, however, we should define the Consumes and Produces attributes on the controller in both cases.

using IdempotentAPI.Filters;

Using the Idempotent Attribute on a Controller’s Class

By using the Idempotent attribute on the API controller’s class, all POST and PATCH actions will work as idempotent operations (requiring the IdempotencyKey header).

[ApiController]
[Route(“[controller]“)]
[Consumes(“application/json”)] // We should define this.
[Produces(“application/json”)] // We should define this.
[Idempotent(Enabled = true)]
public class SimpleController : ControllerBase
{
// …
}

Using the Idempotent Attribute on a Controller’s Action

By using the Idempotent attribute on each action (HTTP POST or PATCH), we can choose which of them should be Idempotent. In addition, we could use the Idempotent attribute to set different options per action.

[HttpPost]
[Idempotent(ExpireHours = 48)]
public IActionResult Post([FromBody] SimpleRequest simpleRequest)
{
// …
}

NuGet Packages

Package Name
Description

IdempotentAPI
The implementation of the IdempotentAPI library.

IdempotentAPI.Cache
Defines the caching abstraction (IIdempotencyCache) that IdempotentAPI is based.

IdempotentAPI.Cache.DistributedCache
The default caching implementation, based on the standard IDistributedCache interface.

IdempotentAPI.Cache.FusionCache
Supports caching via the FusionCache third-party library.

Summary

The IdempotentAPI 1.0.0-RC-01 is available with many improvements ?✨. With the community’s help, several issues and improvements have been identified and implemented. I want to take this opportunity to thank @apchenjun, @fjsosa, @lvzhuye, @RichardGreen-IS2, and @william-keller for your support, ideas, and time to improve this library.

Any help in coding, suggestions, questions, giving a GitHub Star, etc., are welcome ?. If you are using this library, don’t hesitate to contact me. I would be happy to know your use case ?.