3 steps to upload files properly with Django (and HTMX) 📁

Most Django tutorials give bad advice about how to upload files.

Even the official docs (which are generally great) describe uploading files server-side.

This is a bad idea for a real app. If a bunch of people upload big files at the same time, your server will freeze.

The Better way: We'll use Django, but upload files straight from the user's web browser (with HTMX).

There are 3 steps:

  1. Create a temporary upload url for the user
  2. Connect our file hosting service
  3. Let users can upload directly to that link (without going through our server)

Let's go! 🚀 Optional video tutorial featuring me below:

Setup: Create Django app

  • Create a new virtual env, and install Django pip install django
  • Create a new Django project named core and app named sim
django-admin startproject core .
python manage.py startapp sim
  • Include your new app in your INSTALLED_APPS in settings.py.
# settings.py

INSTALLED_APPS = [
    ...
    'sim',
    ...
]

1. Connect our file hosting service (AWS S3)

1.1 Create/login

1.2 Create an IAM user with S3 access

  • Visit the IAM service (Click on the IAM service or visit https://us-east-1.console.aws.amazon.com/iam/)

  • Create an IAM user to access your account (IAM > Users > Create user). Use any username.

  • On 'Set Permissions':

  • Click -> 'Attach policies directly'
  • Find and select -> AmazonS3FullAccess (full S3 read/write access to your IAM role).
    • I recommended searching for AmazonS3FullAccess, rather than scrolling through the 1131 available permissions policies. set-S3-permissions-for-your-IAM-role
  • Click to 'Create user'

1.3 Get your AWS access keys for your new user

  • Click on your newly created user. (My user's username is "Robert")
  • Under, Access keys, click Create access key
  • Click any use case
  • Ignore the Set description tag
  • Click Create access key and leave this page open. We'll need the key shortly.

1.4 Store your AWS access keys in Django

  • Create a file called .env into your project at core/.env
  • Add your AWS access keys. Don't wrap the contents with speech marks.
AWS_ACCESS_KEY_ID=123123-eefse-your-key
AWS_SECRET_ACCESS_KEY=123123-eefse-another-key
  • Click 'Done' on the AWS Access keys page

Note: these keys allow you to access your aws account. Don't upload them to github. If you want to share your repo, add your .env file to a .gitignore file to make git ignore the file with your keys.

1.5 Create an S3 bucket to store your uploads

  • Go to the S3 service at https://s3.console.aws.amazon.com/
  • Click the button 'Create bucket'
  • Enter a globally unique name for your bucket.
  • The other settings don't matter for us. Click to 'Create bucket'

1.6 Update your S3 bucket permissions to allow our upload

  • Go to your new bucket's permission (Click on it -> 'Permissions')
  • Scroll down to the "Cross-origin resource sharing (CORS)" section and click "Edit".
  • Paste the below :
[
    {
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
        "AllowedOrigins": ["*"],
        "ExposeHeaders": []
    }
]
  • Save the changes.

2. Create a temporary upload url for the user

2.1 Generate the temporary upload url

  • Install boto3. This is the python package to interact with AWS. It is named after the Boto River Dolphin 🐬
pip install boto3
  • Create a file called services.py and add the below to generate the temporary upload (a presigned URL)
# sim/services.py
import os
import boto3


def generate_presigned_post(bucket_name, filename, expiration=600):
    """
    Docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/generate_presigned_post.html#generate-presigned-post
    """
    s3_client = boto3.client(
        's3',
        aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
        aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY')
    )
    return s3_client.generate_presigned_post(
        Bucket=bucket_name, Key=filename,
        ExpiresIn=expiration,
    )

2.2 Create your views

  • Add the name of the S3 bucket that you created earlier into your views
# sim/views.py
from django.shortcuts import render
from .services import generate_presigned_post


def uploader(request):
    """
    We render the page with or without data for a presigned post upload.
    """
    if request.method == 'GET':
        return render(request, 'uploader.html')

    elif request.method == 'POST':
        file = request.FILES["file"]
        name = file.name
        content_type = file.content_type

        bucket_name = '' # Todo: Change to your bucket name
        presigned_data = generate_presigned_post(bucket_name, name)

        context = {
            'url': presigned_data['url'],'fields': presigned_data['fields'], 'path': f'to {bucket_name}/{name}'
        }
        print(f'{context = }')

        return render(request, 'uploader.html', context)

1.3 Add your URLs

  • Update your core/urls.py:
from django.contrib import admin
from django.urls import include, path


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('sim.urls')),
]
  • Create urls.py in the sim directory containing:
from django.urls import path
from . import views

urlpatterns = [
    path('', views.uploader, name='uploader'),
]

3. Allow the user to upload files to the temporary upload url (without going through our server)

3.1 Load your AWS keys into your Django app

  • Install python-dotenv
pip install python-dotenv
  • Add these lines to the top of your core/settings.py to load the AWS keys from your .env file into your Django app as environment variables when you run your Django server.
from pathlib import Path
from dotenv import load_dotenv
import os


load_dotenv()


# The below are optional checks that we've connected the keys.
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')

if not AWS_ACCESS_KEY_ID:
    print(f'Missing your AWS_ACCESS_KEY_ID')
if not AWS_SECRET_ACCESS_KEY:
    print(f'Missing your AWS_SECRET_ACCESS_KEY')

3.2 Update your views with your AWS data

3.2 Add your frontend

  • Create a folder sim/templates
  • Add the below uploader.html into sim/templates.
<!DOCTYPE html>
<html>
<head>
    <title>File Upload</title>
    <script src="https://unpkg.com/htmx.org@1.6.1"></script>
</head>
<body>

{% csrf_token %}

<form hx-post="{{ url }}" hx-trigger="submit"
      hx-encoding="multipart/form-data"
      id="form"
>
    {% for field, value in fields.items %}
    <input type="hidden" name="{{ field }}" value="{{ value }}">
    {% endfor %}

    <input type="file" name="file" id="file-input"
           hx-post="{% url 'uploader' %}"
           hx-include="[name='csrfmiddlewaretoken']"
           hx-target="#form"
           hx-select="#form"
           hx-swap="outerHTML"
           hx-trigger="change"
           hx-preserve="true"
    >

    <div id="progress-display" >
        <progress id='progress' value='0' max='100'></progress>
        <div id="percent-wrapper">
            <input id='percent' value="0" type="number" contenteditable="false" disabled/>
            <span>%</span>
        </div>
    </div>

    <button type="submit" id="upload-btn">
        <span id="start-upload">Upload</span>
        <span id="loader">Uploading...</span>
    </button>

    <div id="upload-path">
        Uploaded {{ path }}
    </div>
</form>

<script>
    htmx.on('htmx:xhr:progress', function (evt) {
        if (evt.detail.elt.id === 'form') {
            console.log("evt.detail = ", evt.detail)
            const progressValue = evt.detail.loaded / evt.detail.total * 100;
            htmx.find('#progress').setAttribute('value', progressValue);
            htmx.find('#percent').setAttribute('value', Math.round(progressValue));
        }
    });

    htmx.on('htmx:beforeRequest', (evt) => {
        if (evt.detail.elt.id === 'form') {
            htmx.find('#start-upload').style.display = 'none';
            htmx.find('#loader').style.display = 'block';
            htmx.find('#upload-path').style.visibility = 'hidden';
            htmx.find('#percent-wrapper').style.visibility = 'visible';
        }
    });

    htmx.on('htmx:configRequest', (evt) => {
        if (evt.detail.elt.id === 'form') {
            event.detail.headers = []; // We clear the headers due to a bug in htmx: https://github.com/bigskysoftware/htmx/issues/779#issuecomment-1019373147
        }
    });

    htmx.on('htmx:afterOnLoad', (evt) => {
        if (evt.detail.elt.id === 'form') {
            htmx.find('#loader').style.display = 'none';
            htmx.find('#upload-path').style.visibility = 'visible';
            htmx.find('#start-upload').style.display = 'block';
        }
    });
</script>


<style>
    /* General styling */
    body {
        font-family: Arial, sans-serif;
        margin: auto;
        height: 100vh;
    }

    /* Form styling */
    #form {
        margin: auto;
        display: flex;
        flex-direction: column;
        gap: 20px;
        width: 300px;
        padding: 50px;
    }

    #progress-display {
        display: flex;
        gap: 10px;
    }

    #percent-wrapper{
        width: 90px;
        visibility: hidden;
    }

    #percent {
        border: none;
        text-align: right;
        width: 40px;
        background: transparent;
    }

    #file-input {
        padding: 10px;
        border: 1px solid #ccc;
        border-radius: 5px;
    }

    #upload-btn {
        background-color: #007bff;
        color: white;
        padding: 10px 20px;
        border: none;
        border-radius: 5px;
        cursor: pointer;
    }

    #upload-btn:hover {
        background-color: #0056b3;
    }

    #upload-path {
        visibility: hidden;
    }

    /* Progress bar */
    #progress {
        width: 100%;
        height: 20px;
        border: none;
        border-radius: 10px;
        background: #f3f3f3;
    }

    #progress[value]::-webkit-progress-bar {
        border-radius: 10px;
        background: #f3f3f3;
    }

    #progress[value]::-webkit-progress-value {
        border-radius: 10px;
        background: #007bff;
    }

    #progress[value]::-moz-progress-bar {
        border-radius: 10px;
        background: #007bff;
    }

    /* Loader and Success messages */
    #loader{
        display: none;
    }

</style>

</body>
</html>
  • Run your Django server
python manage.py runserver
  • Visit http://127.0.0.1:8000 (or whatever local url you're using) to upload your files 🗳️


Michael on Youtube asked me "Are we uploading the file twice with this approach?" The answer is no. Click to see my explanation The answer is no: we only upload the file once to S3.

When we select the file, we send a request to our server view. The view a) generates a temporary s3 post url, and b) re-renders our html, inserting the url. This means that our re-rendered HTML now contains the temporary s3 post url: the re-render adds this url to the form at form hx-post="{{ url }}". This was initially blank in our html.

Then, when we submit the form, the form uploads the file to this url.


Finished 🎉 But what about other long-running tasks?

Congratulations! You've now implemented client-side upload with Django and HTMX. Your app is likely to be much faster and nicer to use.

Because you're smart, you might now be thinking:

"Alright, so I've got this cool way to upload files without making my server slow down.

What about other big jobs, though? Like making PDFs, sending a ton of emails, or machine learning. Would they also block the server?"

The answer is yes. All long-running tasks can block up your server.

And here's one way I like to solve that: Upload your brain - How to add serverless functions to Django in 6 minutes (with HTMX and AWS Lambda) 🧠

P.S Photon Designer

I'm building Photon Designer - an entirely visual editor for building Django frontend at the speed that light hits your eyes💡

Let's get visual.

Do you want to create beautiful django frontends effortlessly?
Click below to book your spot on our early access mailing list (as well as early adopter prices).
Made with care by Tom Dekan
© 2023 Photon Designer