CUDA, short for Compute Unified Device Architecture, is Nvidia's parallel computing platform that allows developers and engineers to build incredibly powerful computation systems and programs through the use of GPUs, known as Graphics Processing Units. With the development of CUDA, developers can now easily use their Nvidia GPUs for computationally-expensive tasks.

Difference between GPU and CPU for parallel processing

You might be asking yourself why CUDA and GPU programming matters. The reason why is because while CPUs are great for general-purpose computing, GPUs are good for computationally-expensive workloads. If you need to programmatically add 2 very large vectors or multiply matrices, a modern GPU can handle the operations in parallel whereas a CPU would need to process the operations serially albeit in parallel if the processing is multi-threaded.

This post serves as a basic tutorial for getting CUDA running on a GPU and running a custom built kernel that will add 2 vectors.

For this exercise, the readers should have a somewhat modern GPU. An RTX 5050 or even a GTX 1080 will work fine in a 22.04 version of Ubuntu Linux.

For this guide we are using Ubuntu 22.04. If you have a different version of Ubuntu or distribution of Linux, you will need to adjust your installation step.

Installing Nvidia drivers and CUDA:

The first thing that you should do on your ubuntu 22.04 machine is install the Nvidia drivers and CUDA packages so that your programs can dispatch workloads to the GPU.

This can be done by running the following commands. If you run something besides Ubuntu 22.04, then adjust the commands to whatever version of Ubuntu or other distribution that you are running.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update

sudo apt install cuda

sudo reboot

Once this runs and the machine reboots, run the following:

nvidia-smi

nvcc --version

You should get an output that looks similar to the following:

In my case, I have an RTX 3090 installed.

Running the add vectors CUDA kernel:

In a add_vectors.cu file, insert the following

#include <stdio.h>
#include <stdlib.h>
#include <cuda_runtime.h>

__global__ void add(float *a, float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}

void check(cudaError_t err, const char *msg) {
    if (err != cudaSuccess) {
        printf("Error: %s - %s\n", msg, cudaGetErrorString(err));
        exit(1);
    }
}

int main() {
    int n = 1024;
    size_t size = n * sizeof(float);
    
    float *a = (float*)malloc(size);
    float *b = (float*)malloc(size);
    float *c = (float*)malloc(size);
    
    for (int i = 0; i < n; i++) {
        a[i] = i;
        b[i] = i * 2;
    }
    
    float *d_a, *d_b, *d_c;
    check(cudaMalloc(&d_a, size), "malloc a");
    check(cudaMalloc(&d_b, size), "malloc b");
    check(cudaMalloc(&d_c, size), "malloc c");
    
    check(cudaMemcpy(d_a, a, size, cudaMemcpyHostToDevice), "copy a");
    check(cudaMemcpy(d_b, b, size, cudaMemcpyHostToDevice), "copy b");
    
    int threads = 256;
    int blocks = (n + threads - 1) / threads;
    
    add<<<blocks, threads>>>(d_a, d_b, d_c, n);
    check(cudaGetLastError(), "kernel launch");
    check(cudaDeviceSynchronize(), "sync");
    
    check(cudaMemcpy(c, d_c, size, cudaMemcpyDeviceToHost), "copy result");
    
    for (int i = 0; i < 10; i++) {
        printf("%.0f + %.0f = %.0f\n", a[i], b[i], c[i]);
    }
    
    cudaFree(d_a);
    cudaFree(d_b);
    cudaFree(d_c);
    free(a);
    free(b);
    free(c);
    
    return 0;
}

Running the kernel:

Now that we have a basic CUDA file, we can compile and run it by running the following commands:

nvcc -o vector_add vector_add.cu
./vector_add

After running this program, you should see an output that matches the following:

And now you have run your first add vectors CUDA kernel! If you want to dive deeper you can check dive into this kernel and driver program in the next blog post!

Strengths of notebooks

Simplified Development Environment:

The notebooks allow you to develop and execute python code in a simplified manner making it easy to hit the ground running. Here is an example of a hello world cell:

With this example, we have a code cell, a place where we write code, and can press shift+enter to run it, and we see the output. All I needed to do was create a colab notebook file in my Google drive and was up-and-running quickly.

This is the main benefit of running these notebooks. You can easily create them, open them, and code immediately with them and get results quickly.

Smooth Output And Visualizations:

In addition to showing the text output of the cell, we can easily visualize data:

This data visualization is very powerful, because we immediately see it from running the code. We don’t have to run a python script from the command line that saves a plot to an image file and then open the image file to see the data. It is all just there in the notebook UI.

Reproducibility:

If we want to easily get results from the notebook, but we close it or cleared it, we can just restart the runtime and rerun the cells in order to set up the environment. Clearing the runtime means we restart the python backend process so that we no longer have variables that were created and modified in the previous runtime.

Weakness of notebooks

Low Version Control:

While the notebooks allow you to get up and running quickly, it can be quite difficult to be able to share the results with other engineers. This means the engineer needs to find a way to bring the notebook to other engineers. If you run your own server, you need to commit it to a Git repo or FTP it to another engineer if they need to run it, which complicates the development process. Colab is easier because you can just share the drive and your peers can copy the notebook and run it themselves, but any changes will require manual merge conflict resolution which can be quite painful.

Code Execution Order:

Code execution order refers to the process of running the individual cells for a notebook. When we are exploring and working with variables and data, if we don’t run the notebook cells from top to bottom, we can get conflicts in the variables for a runtime. This means that at the end of development, users need to clear the runtime and run the entire notebook from beginning to end to make sure everything is correct. It gets easy to screw up the runtime, so care needs to be given for that.

Not Suitable for Production:

When developing code and scripts in the notebook, it becomes quite difficult to push that code to a production environment that needs to provide services to users. The reason why is because the format of the notebook is meant for ease of use, not power and functionality. You could theoretically run a production environment from a notebook, but it would be quite fragile and painful.

We have just explored the tradeoffs of developing in a notebook. Now let’s explore the dedicated strengths and weaknesses of the dedicated project:

Strengths of dedicated projects

Reliable and Scalable Execution:

With a dedicated python, c++, or other language project, we can write compile and run scripts to launch a project properly in a more-scalable manner. This is incredibly important when building systems that people need to access frequently.

High Version Control:

Version control is a cornerstone of any mature software product. By using a project-based data application, it eases the ability for teams and engineers to check in their project’s progress. This becomes incredibly important as projects involve more engineers that need to work together as seamlessly as possible.

Increased Modularity:

With a project, it can be built in a way such that other projects or services can import and integrate easily with a project. This is great for a development system where a lot of services are being developed at the same time and they use one another.

Weakness of dedicated projects

More Complex Management:

With a dedicated project, it takes quite a bit of effort to build out the qualities we would expect from a well-designed and developed project such as reliability, scalability, and elasticity. It is possible to run a python script or the runtime from the command-line, but as engineers build features into the project, it increases the complexity of the project, and eventually it will become necessary to spend time to build out the building, running, and deployment systems for the project.

Less Interactive:

We saw before with the notebook that it is quite easy to work with a notebook. Just type the code and run a given cell. With the dedicated project, there is more friction to execute the program. We might have build/run scripts and output directories. We won’t be able to see easy visualizations without writing to a file on disk and opening the file.

More Complicated To Develop:

One issue with projects is that in order to code for them, you need to know enough about the project’s source code to know what changes should be made to achieve a desirable result. With the notebook, you just follow the code in the file until you need to make a change, but with the project, you need to dig through the source code references to find what you are looking for.


Now that we have explored the tradeoffs for both approaches to development, I would like to describe the ideal users and roles that would naturally gravitate towards each tool:

Notebook Developers: The Data Explorers

Data Scientist: This role is the major user of notebooks. Data scientists perform analysis on data for the purposes of fulfilling feature-engineering and model development/validation/optimization.

Data Analyst: This role is another major user of notebooks. Data analysts take existing data to derive insights about business metrics in order to understand optimal business strategies and to visualize these details.

Educator and Student: These roles make effective use of notebooks in order to better teach and learn course material. Notebooks allow people to learn and teach in a more interactive manner.

Project Developers: The Data System Builders

Software Engineer: This role primarily develops using projects. The reason why is that they need to build systems that scale properly with a standardized development flow. These systems could be data-processing systems, but they could also be general purpose systems outside of data.

Data Engineer: This role uses projects to develop data-processing pipelines. They primarily use projects because they are essentially software engineers with a focus on data processing. They focus more on the ETL pipelines and infrastructure.

DevOps and MLOps Engineer: This role is essentially a combination between the software/data engineer and operations engineer. This role also works on data-processing systems, but they also work on making sure systems fulfill their SLOs and SLAs and also maintaining the pipelines that run.

Educator and Student: These roles along with the notebook user also use projects for learning and growth as well. They might implement highly-technical assignments using projects that are not conducive to being developed in a notebook.

This concludes the more specific users for both notebooks and projects, but there are a couple roles that I want to mention that I think will use both notebooks and projects:

Machine Learning Engineer: This role is primarily focused on building data-intensive systems. This engineer should be prepared to bridge the role of data scientists and software/MLOps engineers.

Tech Lead: This role serves the purpose of leading teams and organizations. Usually this contributor starts with one of the previously mentioned roles, but has gained enough competency, influence, and trust to be able to lead the building of entire services and products.

In conclusion, both notebooks and development projects have their roles and purposes. If you need fast and prototype development, then notebooks are your best bet. If you need systems, a project will be your goto.

This is a guide for getting deepseek-r1 up and running on your system. Make sure that you have a system with a GPU(s) that have a total of around 16 GB of VRAM to make it feasible to run the model.

Before starting, make sure you have the relevant drivers for your GPUs. I am running Nvidia currently, so I have the drivers and CUDA toolkit installed.

1: You must first install a platform like ollama in order to pull the model and start inferencing on it. Follow these steps to install ollama:

For Mac and Ubuntu and Windows Subsystem For Linux, do the following:

curl -fsSL https://ollama.com/install.sh | sh

This will install ollama.

2: Next, you need to run ollama’s backend in order to inference on it:

On the various platforms, you can run:

ollama serve

and that should allow you to create another terminal to run deepseek.

On Linux systems running systemd, you can run:

sudo systemctl start ollama

After doing this you should see something like the following:

when you do ollama serve, or

when running systemctl to launch ollama service.

Once you have these, we can finally start inferencing!

3: Pull and run the deepseek model

For this step we are going to run the model locally. There are several models with varying amounts of parameters. The goal is the pick the best bang-for-the-buck model to run on your hardware. Here is a ranking of the models and their sizes:

1.5b version (smallest)

8b version

14b version

32b version

70b version

671b version (biggest)

There are a lot of different variations of r1, so here is a general breakdown to the best of my knowledge:

Use 1.5b, 8b, or 14b if you have around 16 GB of VRAM on your system. These are the economy models that will get the job done, but they are not the smartest.

Use 32b if you have around 32 GB of VRAM. You might actually need more VRAM, so running a 32GB GPU will probably cut it close.

Use 70b if you have something like an A100, and you probably need to run several state-of-the-art cards to run the 671b parameter model.

These are just guesstimates based off my experience using the 14b and smaller models on my GTX 1080s. I will show you how to determine which model is best for your system now.

4: Running the model

For actually running the model, open a terminal and run the following:

ollama run deepseek-r1:8b

This will kick off the process for ollama to pull the model and run it locally. To determine if your hardware is capable, run the folllowing in another terminal assuming you have Nvidia GPUs:

nvidia-smi

You will get an output like the following:

Notice that in my case I have dual GTX 1080s running, and they currently have the 14b model loaded. This model is the highest I can run with without overloading my GPUs.

If you have multiple GPUs, run the following command to use both in parallel for inferencing:

CUDA_VISIBLE_DEVICES=0,1 ollama run deepseek-r1:14b

Once you have completed these steps, you should be able to send a message and now you have the best AI running locally!

For an example, I gave it the following prompt: “Write a simple C++ hello world program” and it gave me the following output:

Enjoy!

Cloud computing infrastructure today forces companies to choose between the convenience of public clouds like AWS and the control of private solutions like OpenStack. Each path requires tradeoffs - either sacrificing infrastructure control and cost optimization for ease of use, or investing heavily in complex private cloud maintenance.

InfraMatrix Global Systems(IGS) is designed to bring the ease of use of the public cloud with the finely-tuned control of the private cloud together in an environment that can run on any amount of hardware that a company or individual owns. Our hyperscaler provides a seamless cloud experience that runs on your own hardware and scales from small deployments to large enterprise infrastructure. The reason we are currently working on this hyperscaler is because we are building infrastructure software products, and we want to build a system that will scale to any amount of hardware in a simplified manner.

The source code and guide for running IGS is available here. Please note that this system is entirely experimental and not production-ready yet.

Architecture:

Before diving into the supported features, it would be helpful to first understand the current version’s system design and architecture. This is the current architecture of the hyperscaler:

In this diagram, we see an IGS node that represents a server with several different components. We have various hardware resources like memory, NICs, disks, and GPUs. For the server’s operating system, we are using Ubuntu and that serves as the main coordinator for us being able to reach the hardware devices. We have a data plane which is what IGS represents currently, and that program talks to a hypervisor, in this case KVM, using QEMU for interfacing it and it talks to kernel in order to provision currently supported hardware, which are NICs and disks, currently.

As you can see, this is a monolithic software architecture making the hyperscaler not yet suitable for an enterprise production environment. The reason why it is not suitable is because a hyperscaler should be able to work from a single interface that will handle the communication between any amount of hardware. This monolith means that for every server that an organization has, it needs to install the hyperscaler on it, which is not feasible.

Subsystems:

Moving on we have 3 main subsystems that the hyperscaler has implemented, and they are compute, networking, and storage. Here is a brief overview of each and the technologies associated with them:

Compute - This serves as the main subsystem for providing VMs, containers, and serverless resources to applications. Currently, we are using KVM with QEMU to provision and launch the VMs.

Networking - This serves as the communication subsystem for compute resources to communicate with each other and external networks. Without this subsystem, we would not be able to push updates, install software, or run distributed systems. Currently, we are solely using QEMU’s SLiRP networking functionality to enable basic internet connectivity. A major problem with using SLiRP is that we can’t effectively server public key authentication for SSH, so the only way to communicate with the VMs running is effectively through serial, which can be a major issue.

Storage - This serves as the main memory subsystem for compute resources to save their state and data properly. Without this subsystem, we would not be able to make persistent changes or enable databases for services that are running on the node. Currently, IGS only supports provisioning and partitioning disks on the host and plugging partitions into VMs.

These are the main subsystems of IGS, currently. You may notice that there is an abundant reliance on existing systems that are mature like Linux and KVM. These mature OSSs provide us the ability to do whatever we want with our hardware, and it makes this project’s undertaking possible. Perhaps eventually it would make sense to write our own OS, hypervisor, networking/storage stacks, but these provide a ton of easy-to-use features for us now.

Main Functionalities:

Given that this project is quite immature, it only supports a couple main features:

Compute - On the compute side, we currently support creating, deleting, starting, stopping, and connecting to VMs over serial.

Networking - On the networking side, we currently support connecting VMs to internet

Storage - On the storage side, we currently support adding and removing, partitioning disks, and placing partitions into VMs.

These are baseline minimum functionalities for a hyperscaler Minimum-Viable-Product(MVP), but it is a good start. The system is quite buggy and thus not suitable for production environments yet, but it will be a lot closer once version 1.00 is released. This is a good segue into what to expect for version 1.

Version 1.00 Expected Features:

Version 1.00 is going to be a large release for the hyperscaler and make a lot of progress towards making it production ready. These are the improvements that will be made to the hyperscaler and its subsystems:

Testing - Create integration tests to validate all paths for the hyperscaler.

Compute - Provide a cleaner way to produce a golden VM image, be it serving it on the internet or autogeneration scripts that will produce the image in an automated manner instead of having to manually create it on the node.

Networking - Use Open vSwitch to connect VMs directly to Linux’s networking stack in a cleaner manner than using SLiRP to enable critical features like SSH and enable Security infrastructure in version 2.00.

Security - Enable SSH only communication. Isolate VM networking interfaces.

Observability - Enable Prometheus and Grafana metrics for capturing performance data for the node and the subsystems.

This post is a walkthrough of how we can use Linux networking interfaces and tcpdump to observe the traffic going across our host. For this example, we are going to show different ways to use tcpdump to observe traffic:

The first example is to just dump all the traffic going through our physical NIC. First we need to figure out what our external-facing NIC is, and we do the following:

ip route show

We should see a default entry that looks like the following:

This tells us our NIC for accessing the internet is enp14s0. Next run the following command:

sudo tcpdump -i enp14s0

We should see all of our traffic running across the NIC. This is a straight dump of all the traffic and it helps us understand what our network is doing. For the next example, open 2 terminals and run these in separate terminals:

sudo tcpdump -i enp14s0 -n icmp

and in another terminal run:

ping 192.168.1.29

Go back to your tcpdump terminal and should see traffic that looks like this:

We can run tcpdump and specify a specific protocol like ICMP that we want to capture the traffic on. This makes it really useful for observing and debugging network configurations as ICMP is used for validating and testing network functionality. Finally, run the following:

sudo tcpdump -i enp14s0 -n -v port 443

and in another terminal run:

curl https://www.thehypervisor.blog > /dev/null

Go to the tcpdump terminal, and you should see something like this:

This shows that we can specify the specific networking port that we want to capture traffic on as well. In this case, we are tracking port 443 because that is the port used for HTTPS traffic, and curl allows us to initiate an HTTPS connection using the network transport protocol: TCP.

In summary, we can use tcpdump for observing and debugging issues at any level of our networking stack, which corresponds to the OSI model. Network software engineers use this tool frequently to verify that their software and networking configurations work properly.

There are several projects that implement this technology. A few of them are HAProxy, nginx, Envoy, and Traefik. They all have their tradeoffs, but today I want to show the power of Traefik.

First, it will be helpful to discuss some recurring issues with a lot of reverse-proxies.


1. A lot of manual and static configuration is required to configure and maintain reverse-proxies.These two points can make developing distributed systems challenging. The reason why is because these systems need to be told what services running on a given host need to be connected to and fed traffic. Adding services may require complex configuration changes, and may also require the system to be taken down to bring on more services running on a host.

2. Enabling security protocols like SSL can be painful for the services added to the proxy. On load-balancers, there may be a large subsystem designed just for handling the certificates that a service handles, which means making direct changes to the host environment, requiring complex maintenance that can disrupt the services.

Traefik does a beautiful job of handling these at the expense of a little bit more over-head than non-cloud optimized reverse-proxies like HAProxy and nginx.


Traefik has a very simple system for enabling a self-managing service easily, which we will do now. It runs on top of docker, and you manage it by making changes to a container management system like kubernetes or docker compose. This guide uses docker compose.

The requirements for running Traefik are relatively straightforward. Assuming you are running an ubuntu server, you can perform the following 3 steps to prep your system to run Traefik in a containerized environment:

1: Install ca-certificates and curl:

sudo apt-get update && sudo apt-get install -y ca-certificates curl

2: Install docker and docker compose by following the guide. I would recommend using the apt method, but use whatever works best for you: https://docs.docker.com/desktop/setup/install/linux/ubuntu/

3: Install the docker packages:

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Once you have performed these steps, you are all set to run Traefik.


There are a couple more steps for actually running Traefik, though. Those steps are to create a traefik and docker-compose configurations:

1: Create the traefik configuration:

The goal of this step is to set up a repository or directory such that it can pull and run a Traefik docker container in a proxy network and connect it to a service running on the same host.


In a base directory or repository, run the following:

mkdir -p config

touch config/acme.json
chmod 600 config/acme.json

touch config/traefik.yml
chmod 644 config/traefik.yml

and then place the following content in traefik.yml:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: proxy

certificatesResolvers:
  letsencrypt:
    acme:
      email: *your_email*
      storage: acme.json
      httpChallenge:
        entryPoint: web

…and make sure to replace your_email in the file with the email that you want to coordinate with letsencrypt.

After this has been completed, create a docker-compose.yml file in the same directory as the config directory, and place the following content in it:

networks:

  proxy:
    external: true

services:

  traefik:
    image: traefik:v2.10
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/traefik.yml:/traefik.yml:ro
      - ./config/acme.json:/acme.json
    networks:
      - proxy

  site:
    build: .
    image: site
    volumes:
      - .:/site/app
      - /site/app/node_modules
      - /site/app/.next
      - ./site/public:/usr/src/app/public
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myservice-http.rule=PathPrefix(`/`)"
      - "traefik.http.routers.myservice-http.entrypoints=web"
      - "traefik.http.routers.myservice-https.rule=PathPrefix(`/`)"
      - "traefik.http.routers.myservice-https.entrypoints=websecure"
      - "traefik.http.routers.myservice-https.tls=true"
      #- "traefik.http.routers.nextjs-https.tls.certresolver=letsencrypt"
      - "traefik.http.services.nextjs.loadbalancer.server.port=3000"

This docker compose configuration launches a Traefik container in the proxy network and passes the configuration that we just created to it. We also give it ports 80 and 443 from the host, because this example is running a web application and it needs HTTP and HTTPS ports. Notice that the certresolve label is commented out. The reason why is because we aren’t trying to forward traffic to a specific site to the app, we are just forwarding all traffec based on the http.rule and the https.rules give by the ‘/’ above it.


After that we run the following to launch the configuration:

docker network create proxy
docker build --force-rm -t site .
docker compose up

Assuming that everything is set up properly, we should see that Traefik launches and handles everything from the configuration setup to coordinating with letsencrypt for enabling TLS for the traffic that we receive. We should see an output that looks similar to the following:

Just to summarize, the beauty of Traefik is that we have a simple system for running and handling web applications and servers. Traefik enables me to host however many services I want, but in my case, it automatically handles my website’s encryption. This comes at the tradeoff of high-performance, but if your usecase is focused more on the features, then Traefik is an incredibly powerful tool that makes building and running a website simple.

The source code for this example can be downloaded and ran by doing the following:

git clone https://github.com/leonematt/research
cd research/cloud/load-balancers/traefik/example
./install.sh
./run.sh