How to add serverless functions with Django in 6 mins (with HTMX and AWS Lambda) 🧠

How do serverless functions help me?

Answer: for any longer running tasks (e.g., preparing PDFs, sending emails, machine learning tasks).

If you don't run long-running functions outside your main server, these tasks will make your server unresponsive to new requests from users -> A rubbish user experience.

Some people use background workers to create long-running tasks (e.g., Celery, Django-Q, Huey). For me, serverless functions are easier to write, simpler to test, and much easier to debug. They're also infinitely scalable (which is cool). ♾️

How can I learn serverless functions in 6 minutes?

There'll be 4 sections. After this, you'll know how to use serverless functions in your Django app to do your long-running tasks in a fast and maximally simple way.

Here's an optional video tutorial (with me 👋) that follows the written guide below.

🏁 Your finished product - using Django, serverless functions (with AWS Lambda), and HTMX - will look like this:

Section 1: Setup your Django app

Install the necessary packages:

pip install django requests

Create the Django project and a new app:

django-admin startproject core .
python manage.py startapp sim

Add our new app to the INSTALLED_APPS in core/settings.py

# settings.py
INSTALLED_APPS = [
    # ...
    'sim',
    # ...
]

Update the ALLOWED_HOSTS in core/settings.app:

ALLOWED_HOSTS = [
    "host.docker.internal",
    "127.0.0.1"
]

Section 2: Add content to your app

2.1 Add your views

# sim/views.py
from django.shortcuts import render
from django.http import JsonResponse
import requests

def dashboard(request):
    return render(request, 'dashboard.html')


def process_task(request):
    # We will update this view later with a link to our serverless function.
    return JsonResponse({'status': 'success'})


def save_result(request):
    # We will update this view later to save the result data.
    print("Task completed! Time to save the data to our database.")
    return JsonResponse({'status': 'success'})

2.2 Add your URLs

Create a urls.py in the sim directory:

# sim/urls.py
from django.urls import path
from .views import dashboard, process_task, save_result

urlpatterns = [
    path('dashboard/', dashboard, name='dashboard'),
    path('process-task/', process_task, name='process_task'),
    path('save-result/', save_result, name='save_result'),
]

Update the project's core/urls.py to include our new paths:

# core/urls.py
from django.contrib import admin
from django.urls import include, path


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('sim.urls')),
]

2.3 Add your models

Add the below to models.py:

# sim/models.py
from django.db import models


class TaskResult(models.Model):
    name = models.CharField(max_length=50)
    status = models.CharField(max_length=10)
    created_at = models.DateTimeField(auto_now_add=True)

Make and run your migrations to create the database

python manage.py makemigrations
python manage.py migrate

2.4 Add your templates

Now we'll create a dashboard where users can initiate their mind upload. We'll use HTMX because it's elegant and excellent.

Side note: If you want to learn more about how HTMX (it's powerful and simple to use), check out these mini-guides I wrote on HTMX (all free as usual):

How to create a Django form (using HTMX) in 90 seconds 🐎

Add database search with Django and HTMX 🕵️

How to create ChatGPT with Django and HTMX in 4 minutes 🦾

Create a folder sim/templates
Add the below dasboard.html into sim/templates

<!DOCTYPE html>
<html>
<head>
    <title>Dashboard</title>
    <script src="https://unpkg.com/htmx.org"></script>
</head>
<body>
    <h1>Welcome to the Mind Upload Dashboard</h1>
    <button hx-post="/process-task/" hx-trigger="click" hx-swap="outerHTML">Initiate Mind Upload</button>
    <div id="task-status">
        <!&#45;&#45; We'll dynamically update this section to display the task status &#45;&#45;>
    </div>
</body>
</html>

2.5 Run your server to check your progress

In your terminal, enter:

python manage.py runserver

Head over to http://127.0.0.1:8000/dashboard/ to see your mind upload dashboard. It should like:

Section 3: Add your serverless function

3.1 Install SAM

To run our serverless function locally, we'll install AWS's SAM tool (Serverless Application Model) to create a model of the AWS environment locally.

Install SAM:

pip install --upgrade aws-sam-cli
sam --version

Initiate your SAM project:

sam init

Select 'AWS Quick Start Templates'
Select 'Hello World' Example
Yes to Python and zip
No to 'AWS X-Ray tracing'
No to 'Amazon CloudWatch Application Insights'.
Name your new application 'sam-app'.

Here's a minimal explanation of the folders that sam init creates:
- hello_world: Folder containing your serverless function (AKA your lambda function).
- events: Folder containing a sample JSON requests to test your hello_world serverless function.

3.2 Update your serverless function API

Replace sam-app/template.yaml with:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 10
    MemorySize: 128

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: post

3.3 Build your serverless function

Run the below to package your local serverless function to run it locally (and later deploy to AWS):

cd sam-app
sam build

You should see a "Build Succeeded" message.
Note: you'll need to run sam build to rebuild your serverless function whenever you make a change.

3.4 Run your serverless function

In the sam-app directory, run:

sam local invoke

You should see the 200 status code below. If not, first check out the troubleshooting below.

{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}

Troubleshooting `sam local invoke`

I hit the below error when running the serverless function above:

"Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running?"

Here's how you can tackle it:
1. Install Docker: Although Docker is cumbersome, it's worth it to use AWS SAM to test your serverless functions locally. Install Docker Desktop and run it. Restart your terminal after installing Docker.

Error stays after installing docker?: Go to Docker's Advanced Settings (or the equivalent) and activate the default Docker socket. This worked for me with macOS (Solution from here).

3.5 Edit your serverless function to do your processing and send the result to your app

Replace all the existing code in sam-app/hello_world/app.py with:

import json
import requests
import time


def lambda_handler(event, context):
    data = json.loads(event['body'])
    task_id = data['task_id']

    time.sleep(3)  # Simulate a long-running function (e.g., uploading your brain state).
    print(f'Completed task_id {task_id}')

    response = requests.post(
        'http://host.docker.internal:8000/save-result/',
        headers={'Authorization': 'secret_123'},
        json={'task_id': task_id},
    )  # We send the result data to our django server.

    if response.status_code == 200:
        return {
            "statusCode": response.status_code, "body": json.dumps(
                {"message": f"Completed task_id {task_id}"}
            ),
        }
    else:
        return {"statusCode": response.status_code, "body": response.content}

Section 4: Connect your Serverless Function to your Django app

4.1 Connect your views to your serverless function:

Connect your process_task view to the serverless function URL.
Update the save_result view to mark the task as completed and save the result data:

# sim/views.py
from django.shortcuts import render
from django.views.decorators.csrf import csrf_exempt
from django.http import JsonResponse, HttpResponse
from .models import TaskResult
import requests
import json


def dashboard(request):
    tasks = TaskResult.objects.all().order_by('-created_at')
    return render(request, 'dashboard.html', {'tasks': tasks})


def process_task(request):
    task = TaskResult.objects.create(
        name=request.POST['name'], status='pending'
    )

    serverless_function_url = 'http://127.0.0.1:3000/hello'
    response = requests.post(
        serverless_function_url, json={'task_id': task.id},
    )
    print(f'{response.content = }')

    if response.status_code == 200:
        print(f'Sent task_id {task.id} for processing')
        return HttpResponse(status=response.status_code)
    else:
        task.status = 'failed'
        task.save()
        return HttpResponse(status=response.status_code)


@csrf_exempt
def save_result(request):
    api_key = request.headers.get('Authorization')
    if api_key != 'secret_123':  # Replace this with an environment variable in production.
        return JsonResponse({'status': 'failure', 'error': 'Forbidden'}, status=403)

    data = json.loads(request.body.decode('utf-8'))
    task_id = data['task_id']
    print(f'Received: {task_id = }')

    try:
        task = TaskResult.objects.get(id=task_id)
        task.status = "completed"
        task.result_data = {"message": f"task completed with task_id {task_id}"}
        task.save()
        return JsonResponse({'status': 'success'}, status=200)
    except TaskResult.DoesNotExist:
        return JsonResponse({'status': 'task does not exist'}, status=404)

4.2 Call your views with your serverless function

Update your sample event in sam-app/events/event.json(replace all of the code) to:

{
    "body": "{\"task_id\": 1}",
    "httpMethod": "POST"
}

Rebuild your function. Run it with your new sample event (make sure you're in the sam-app folder)

sam build
sam local invoke --event events/event.json

You should see something like the below. The status is "task does not exist" because there's no task yet with an id of 1 in our Django database:

Event = {'body': {'task_id': 1}, 'httpMethod': 'POST'}
Received task_id 1
Completed task_id 1
END RequestId: ac4d9b48-9675-4513-881b-d5a89d801035
REPORT RequestId: ac4d9b48-9675-4513-881b-d5a89d801035  Init Duration: 0.76 ms  Duration: 3895.27 ms    Billed Duration: 3896 ms        Memory Size: 128 MB     Max Memory Used: 128 MB     
{"status_code": 404, "body": "{\"status\": \"task does not exist\"}"}

4.3 Update your frontend

We'll update the template in sim/templates/dashboard.html to:
a) show all task results
b) poll the server for task status updates (using HTMX).
c) add some styling to make it look better 🥋

<!DOCTYPE html>
<html xmlns:hx-on="http://www.w3.org/1999/xhtml">
<head>
    <title>Dashboard</title>
    <script src="https://unpkg.com/htmx.org"></script>

</head>
<body>

<h1>Welcome to your Mind Uploader</h1>
<form id="process-task" hx-select="#process-task" hx-post="/process-task/" hx-swap="none"  hx-on::before-request="this.reset()">
    {% csrf_token %}
    <input type="text" name="name" placeholder="Person to upload" required>
    <button type="submit"  >
        Initiate Mind Upload
    </button>
</form>

<div id="task-status"
     hx-get="/dashboard/" hx-trigger="every 1s"
     hx-select="#task-status" hx-swap="outerHTML">
    <div id="update-time">
        {% if tasks %}
        Last updated: <span id="now-text">{% now "H:i:s" %}</span>
        {% endif %}
    </div>

    <!&#45;&#45; This section will dynamically update to display the task status &#45;&#45;>
    <section id="tasks">
        {% for task in tasks %}
        <div class="task" id="task-{{ task.id }}">
            <div>
                <div>{{ task.name }}</div>
                <div>(Brain #{{task.id}})</div>
                <span class="status-{{ task.status|lower }}">
                    {{ task.status }}
                    {% if task.status == "completed" %}
                         🧠

                    {% elif task.status == "pending" %}
                        ⏳
                    {% endif %}
                </span>
                <span>Created {{ task.created_at|date:"H:i:s"  }}</span>
            </div>

        </div>
        {% endfor %}
    </section>
</div>

</body>
<style>
    body {
        font-family: 'Arial', sans-serif;
        margin: 0;
        padding: 0;
        background-color: #f0f2f5;
        color: #333;
    }
    h1 {
        background-color: #ffffff;
        padding: 20px;
        margin-bottom: 20px;
        text-align: center;
        border-bottom: 2px solid #f0f2f5;
    }
    form {
        text-align: center;
        margin-bottom: 20px;
    }
    input {
        padding: 10px;
        border-radius: 5px;
        border: 1px solid #ccc;
        margin-right: 10px;
    }
    button {
        padding: 10px 20px;
        font-size: 16px;
        background-color: #e0e1e2;
        color: #333;
        border: none;
        border-radius: 5px;
        cursor: pointer;
    }
    button:hover {
        background-color: #ccd0d4;
    }
    #update-time {
        background-color: #ffffff;
        text-align: center;
        padding-top: 10px;
    }
    #now-text {
        color: #6587b6;
    }
    #tasks {
        background-color: #ffffff;
        padding: 20px;
        border-radius: 5px;
        box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
        display: grid;
        grid-template-columns: repeat(auto-fill, minmax(150px, 1fr));
        gap: 20px;
    }
    .task {
        padding: 20px;
        background-color: #f8f9fa;
        border-radius: 5px;
        box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
        display: flex;
        flex-direction: column;
        align-items: center;
        text-align: center;
    }
    .status-completed {
        color: #28a745;
    }
    .status-pending {
        color: #ffbf00;
    }
</style>
</html>

4.4 Full run

Run your Django server in one terminal:

python manage.py runserver

And run your serverless environment in another terminal:

cd sam-app
sam local start-api

Visit http://127.0.0.1:8000/dashboard/ and submit some data. Your final product should look like this:

Congratulations 🎉 You can now use serverless functions with Django

The next step is to deploy your serverless functions with your Django app. This is quite simple - I'll plan to write a guide on this as well.

Two cliffhanger comments:

The two standard ways of deploying serverless functions with AWS are: i) using an API gateway, which fits what we've been doing so far, or ii) using the AWS SDK (which works well, but isn't as easy to test locally)
If you are deploying your codebase on push, you'll want to automatically build your lambda function and deploy it during this process. I would recommend doing this with a simple script in your CI provider.

Reader questions from Youtube

Is Celery an alternative of serverless functions?

Yes. Celery is an alternative to using serverless functions.

Running Celery (other similar alternatives are Django-Q and Huey) essentially involves running a different server in the background, with a queue of jobs to do. The program runs in a loop, running the top item from the queue. Many people use these background task queues.

However, I'd recommend serverless functions instead. Serverless functions are easier to write, simpler to test, and much easier to debug (+ also infinitely scalable ♾️).

The last point of easier debugging is most important. It's common to have issues debugging strange asynchronous task queue operations in Celery or other background task queues. Serverless functions are simpler.

Would using threads (threadpools) be another option instead of serverless functions?

No. You probably wouldn't use threads as a background worker instead of serverless functions.

Threads are suitable for speeding up your tasks, such as running things in parallel in the same application context, rather than handling background tasks or distributed workloads.

In contrast, serverless functions are great for running tasks in the background.

P.S - Photon Designer

I'm building Photon Designer - an entirely visual editor for building Django frontend at the speed that light hits your eyes. Photon Designer outputs neat, clean Django templates.