3 steps to upload files properly with Django (and HTMX) 📁

Most Django tutorials give bad advice about how to upload files.

Even the official docs (which are generally great) describe uploading files server-side.

This is a bad idea for a real app. If a bunch of people upload big files at the same time, your server will freeze.

The Better way: We'll use Django, but upload files straight from the user's web browser (with HTMX).

There are 3 steps:

Create a temporary upload url for the user
Connect our file hosting service
Let users can upload directly to that link (without going through our server)

Let's go! 🚀 Optional video tutorial featuring me below:

Setup: Create Django app

Create a new virtual env, and install Django pip install django
Create a new Django project named core and app named sim

django-admin startproject core .
python manage.py startapp sim

Include your new app in your INSTALLED_APPS in settings.py.

# settings.py

INSTALLED_APPS = [
    ...
    'sim',
    ...
]

1. Connect our file hosting service (AWS S3)

1.1 Create/login

Create/login to an AWS account at http://console.aws.amazon.com. There is a free tier.

1.2 Create an IAM user with S3 access

Visit the IAM service (Click on the IAM service or visit https://us-east-1.console.aws.amazon.com/iam/)
Create an IAM user to access your account (IAM > Users > Create user). Use any username.
On 'Set Permissions':
Click -> 'Attach policies directly'
Find and select -> AmazonS3FullAccess (full S3 read/write access to your IAM role).
- I recommended searching for AmazonS3FullAccess, rather than scrolling through the 1131 available permissions policies.
Click to 'Create user'

1.3 Get your AWS access keys for your new user

Click on your newly created user. (My user's username is "Robert")
Under, Access keys, click Create access key
Click any use case
Ignore the Set description tag
Click Create access key and leave this page open. We'll need the key shortly.

1.4 Store your AWS access keys in Django

Create a file called .env into your project at core/.env
Add your AWS access keys. Don't wrap the contents with speech marks.

AWS_ACCESS_KEY_ID=<123-your-key>
AWS_SECRET_ACCESS_KEY=<123-another-key>
AWS_DEFAULT_REGION=eu-central-1

(Optionally change the region to your local region) - Click 'Done' on the AWS Access keys page

Note: these keys allow you to access your aws account. Don't upload them to github. If you want to share your repo, add your .env file to a .gitignore file to make git ignore the file with your keys.

1.5 Create an S3 bucket to store your uploads

Go to the S3 service at https://s3.console.aws.amazon.com/
Click the button 'Create bucket'
Enter a globally unique name for your bucket.
The other settings don't matter for us. Click to 'Create bucket'

1.6 Update your S3 bucket permissions to allow our upload

Go to your new bucket's permission (Click on it -> 'Permissions')
Scroll down to the "Cross-origin resource sharing (CORS)" section and click "Edit".
Paste the below :

[
    {
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
        "AllowedOrigins": ["*"],
        "ExposeHeaders": []
    }
]

Save the changes.

2. Create a temporary upload url for the user

2.1 Generate the temporary upload url

Install boto3. This is the python package to interact with AWS. It is named after the Boto River Dolphin 🐬

pip install boto3

Create a file called services.py and add the below to generate the temporary upload (a presigned URL)

# sim/services.py
import os
import boto3


def generate_presigned_post(bucket_name, filename, expiration=600):
    """
    Docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/generate_presigned_post.html#generate-presigned-post
    """
    s3_client = boto3.client(
        's3',
        aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
        aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY')
    )
    return s3_client.generate_presigned_post(
        Bucket=bucket_name, Key=filename,
        ExpiresIn=expiration,
    )

2.2 Create your views

Add the name of the S3 bucket that you created earlier into your views

# sim/views.py
from django.shortcuts import render
from .services import generate_presigned_post


def uploader(request):
    """
    We render the page with or without data for a presigned post upload.
    """
    if request.method == 'GET':
        return render(request, 'uploader.html')

    elif request.method == 'POST':
        file = request.FILES["file"]
        name = file.name
        content_type = file.content_type

        # Add any Django form validation here to check the file is valid, correct size, type, etc.

        bucket_name = '' # Todo: Change to your bucket name
        presigned_data = generate_presigned_post(bucket_name, name)

        context = {
            'url': presigned_data['url'],'fields': presigned_data['fields'], 'path': f'to {bucket_name}/{name}'
        }
        print(f'{context = }')

        return render(request, 'uploader.html', context)

2.3 Add your URLs

Update your core/urls.py:

from django.contrib import admin
from django.urls import include, path


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('sim.urls')),
]

Create urls.py in the sim directory containing:

from django.urls import path
from . import views

urlpatterns = [
    path('', views.uploader, name='uploader'),
]

3. Allow the user to upload files to the temporary upload url (without going through our server)

3.1 Load your AWS keys into your Django app

Install python-dotenv

pip install python-dotenv

Add these lines to the top of your core/settings.py to load the AWS keys from your .env file into your Django app as environment variables when you run your Django server.

from pathlib import Path
from dotenv import load_dotenv
import os


load_dotenv()


# The below are optional checks that we've connected the keys.
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')

if not AWS_ACCESS_KEY_ID:
    print(f'Missing your AWS_ACCESS_KEY_ID')
if not AWS_SECRET_ACCESS_KEY:
    print(f'Missing your AWS_SECRET_ACCESS_KEY')

3.2 Update your views with your AWS data and add your frontend

Create a folder sim/templates
Add the below uploader.html into sim/templates.

<!DOCTYPE html>
<html>
<head>
    <title>File Upload</title>
    <script src="https://unpkg.com/htmx.org@1.6.1"></script>
</head>
<body>

{% csrf_token %}

<form hx-post="{{ url }}" hx-trigger="submit"
      hx-encoding="multipart/form-data"
      id="form"
>
    {% for field, value in fields.items %}
    <input type="hidden" name="{{ field }}" value="{{ value }}">
    {% endfor %}

    <input type="file" name="file" id="file-input"
           hx-post="{% url 'uploader' %}"
           hx-include="[name='csrfmiddlewaretoken']"
           hx-target="#form"
           hx-select="#form"
           hx-swap="outerHTML"
           hx-trigger="change"
           hx-preserve="true"
    >

    <div id="progress-display" >
        <progress id='progress' value='0' max='100'></progress>
        <div id="percent-wrapper">
            <input id='percent' value="0" type="number" contenteditable="false" disabled/>
            <span>%</span>
        </div>
    </div>

    <button type="submit" id="upload-btn">
        <span id="start-upload">Upload</span>
        <span id="loader">Uploading...</span>
    </button>

    <div id="upload-path">
        Uploaded {{ path }}
    </div>
</form>

<script>
    htmx.on('htmx:xhr:progress', function (evt) {
        if (evt.detail.elt.id === 'form') {
            console.log("evt.detail = ", evt.detail)
            const progressValue = evt.detail.loaded / evt.detail.total * 100;
            htmx.find('#progress').setAttribute('value', progressValue);
            htmx.find('#percent').setAttribute('value', Math.round(progressValue));
        }
    });

    htmx.on('htmx:beforeRequest', (evt) => {
        if (evt.detail.elt.id === 'form') {
            htmx.find('#start-upload').style.display = 'none';
            htmx.find('#loader').style.display = 'block';
            htmx.find('#upload-path').style.visibility = 'hidden';
            htmx.find('#percent-wrapper').style.visibility = 'visible';
        }
    });

    htmx.on('htmx:configRequest', (evt) => {
        if (evt.detail.elt.id === 'form') {
            event.detail.headers = []; // We clear the headers due to a bug in htmx: https://github.com/bigskysoftware/htmx/issues/779#issuecomment-1019373147
        }
    });

    htmx.on('htmx:afterOnLoad', (evt) => {
        if (evt.detail.elt.id === 'form') {
            htmx.find('#loader').style.display = 'none';
            htmx.find('#upload-path').style.visibility = 'visible';
            htmx.find('#start-upload').style.display = 'block';
        }
    });
</script>


<style>
    /* General styling */
    body {
        font-family: Arial, sans-serif;
        margin: auto;
        height: 100vh;
    }

    /* Form styling */
    #form {
        margin: auto;
        display: flex;
        flex-direction: column;
        gap: 20px;
        width: 300px;
        padding: 50px;
    }

    #progress-display {
        display: flex;
        gap: 10px;
    }

    #percent-wrapper{
        width: 90px;
        visibility: hidden;
    }

    #percent {
        border: none;
        text-align: right;
        width: 40px;
        background: transparent;
    }

    #file-input {
        padding: 10px;
        border: 1px solid #ccc;
        border-radius: 5px;
    }

    #upload-btn {
        background-color: #007bff;
        color: white;
        padding: 10px 20px;
        border: none;
        border-radius: 5px;
        cursor: pointer;
    }

    #upload-btn:hover {
        background-color: #0056b3;
    }

    #upload-path {
        visibility: hidden;
    }

    /* Progress bar */
    #progress {
        width: 100%;
        height: 20px;
        border: none;
        border-radius: 10px;
        background: #f3f3f3;
    }

    #progress[value]::-webkit-progress-bar {
        border-radius: 10px;
        background: #f3f3f3;
    }

    #progress[value]::-webkit-progress-value {
        border-radius: 10px;
        background: #007bff;
    }

    #progress[value]::-moz-progress-bar {
        border-radius: 10px;
        background: #007bff;
    }

    /* Loader and Success messages */
    #loader{
        display: none;
    }

</style>

</body>
</html>

Run your Django server

python manage.py runserver

Visit http://127.0.0.1:8000 (or whatever local url you're using) to upload your files 🗳️

Michael on Youtube asked me "Are we uploading the file twice with this approach?" The answer is no. Click to see my explanation

The answer is no: we only upload the file once to S3.

When we select the file, we send a request to our server view. The view a) generates a temporary s3 post url, and b) re-renders our html, inserting the url. This means that our re-rendered HTML now contains the temporary s3 post url: the re-render adds this url to the form at form hx-post="{{ url }}". This was initially blank in our html.

Then, when we submit the form, the form uploads the file to this url.

Oscar on Twitter asked me more about uploading the file: "How does the upload process work more clearly?"

Thanks Oscar 👍
To answer, for each upload, we:
1. Send data about the file to s3. This creates upload permission for the specific file (the presigned data) and an upload url
2. Upload the file with the presigned data to the upload url

Finished 🎉 But what about other long-running tasks?

Congratulations! You've now implemented client-side upload with Django and HTMX. Your app is likely to be much faster and nicer to use.

Because you're smart, you might now be thinking:

"Alright, so I've got this cool way to upload files without making my server slow down.

What about other big jobs, though? Like making PDFs, sending a ton of emails, or machine learning. Would they also block the server?"

The answer is yes. All long-running tasks can block up your server.

And here's one way I like to solve that: Upload your brain - How to add serverless functions to Django in 6 minutes (with HTMX and AWS Lambda) 🧠

P.S Photon Designer

I'm building Photon Designer - an entirely visual editor for building Django frontend at the speed that light hits your eyes💡