Have you ever wondered how to build your personal URL shortener service like bitly or tinyurl 🤓 ?
In this tutorial, we are going to build a simple URL shortener using FastAPI, MongoDB and Python. A URL Shortener is a simple tool that can be used to shorten long links, which makes it easier to share and remember in most cases. URL shorteners can ultimately help foster a brand's growth through increased number of clicks.
Core Objective of this article
FastAPI is gaining massive traction amongst Python developers because of its incredible performance, rich features, and extensive documentation. The crux of this article is to teach you how to build useful apps like this using FastAPI and MongoDB as the database.
How does a URL Shortener work ?
The working principle of a URL shortener is actually less complex than you think 😉. You simply have a logic that generates a random shortcode which identifies the long URL, this shortcode is used to generate a short URL which is stored in the database with the long URL. So when a request is made to that short URL, it redirects to the original long URL. More advanced features like site analytics, custom shortcodes and monitoring of clicks can also be implemented to facilitate seamless user experience. So, Let's get our hands dirty,folk !
Prerequisites
- A good grasp of Python and FastAPI fundamentals, refer to the FastAPI documentation HERE,
- Basic understanding of HTTP request and response lifecycle,
- Basic knowledge of NoSQL databases (i.e. MongoDB) , refer to the official MongoDB documentation HERE
- An IDE or code editor i.e. VSCode.
Getting Started
STEP 1 : Setting up the MongoDB database
For the purpose of this tutorial, we shall be using Mongo Atlas. Mongo Atlas is a fully-managed cloud database service provided by mongodb for rapidly building scalable and secure applications. The free tier cluster provides up to 512MB with a shared RAM, You can read more on using the Free tier cluster HERE
You can alternatively set up MongoDB locally following the official docs
To setup Mongo Atlas for the purpose of this tutorial :-
- Create a new MongoDB account at mongodb.com
- Create a new project and name it 'fastapi-url-shortener'
- Create a free cluster on Mongo Atlas, select your preferred cloud provider and region. A new cluster takes about 1 - 3 minutes to fully provision :-
- Click on connect to configure the connection to the Mongo Atlas cluster and fetch the connection parameters that will be required by FastAPI mongo client
- Fill out the information for the next steps regarding the database user and connection security, Choose your connection method as 'Connect to application' which will help us connect from our FastAPI app. Finally, copy the required connection string :-
NOTE :- The connection string will be used to connect from the FastAPI app, keep it safe and replace the "password" with the password you created for that database user.
STEP 2 :- Setting up the Python environment
You are required to have Python (> 3.6) to proceed with this step.
(1) Create and activate a virtual environment 'env' in your root directory using venv . This virtual environment will be used to properly manage the project dependencies.
$ python -m venv env
$ source env/Scripts/activate
This might change based on the OS, kindly refer to the venv docs
(2) Installing required packages :- The following packages will be used during the development of this app :-
- fastapi
- uvicorn (For running FastAPI on an ASGI server),
- pydantic (For robust data validation and building request models),
- mongoengine (Mongo client and Object document mapper for interacting with the mongo database from Python),
- validators (For validating URLs),
- python-decouple (For managing environment variables),
- shortuuid (For generating random shortcodes),
- bson,
- jinja2 (For integrating HTML templates)
(env) $ pip install fastapi uvicorn pydantic mongoengine validators shortuuid python-decouple dnspython bson jinja2
(3) Directory / Folder structure
Next, create the following files and folders
📦env
📦urlshortener
┣ 📂routes
┃ ┣ 📜shorten.py
┣ 📜redirect.py
┃ ┗ 📜__init__.py
┣ 📂templates
┃ ┗ 📜index.html
┣ 📂utils
┃ ┣ 📜helpers.py
┃ ┗ 📜__init__.py
┣ 📜.env
┣ 📜main.py
┣ 📜models.py
┗ 📜schemas.py
The main.py file contains the main fastapi app, the routes folder contains routes linked to the main app via FastAPI APIRouter, the schemas.py file contains pydantic data validation schemas.
Refer to the FastAPI documentation to understand how to properly structure a FastAPI application
(4) Finalizing setup We will run a simple FastAPI server here which returns "Hello World". Import the required libraries in the main.py file and connect to the shorten.py route using the APIRouter :-
## main.py
from fastapi import FastAPI, HTTPException, Depends, status, Request
import schemas
from routes.shorten import router as ShortenRouter
from fastapi.responses import JSONResponse, HTMLResponse
from fastapi.templating import Jinja2Templates
app = FastAPI()
# Templates
templates = Jinja2Templates(directory = "templates")
@app.get('/', response_class = HTMLResponse)
async def index(request : Request):
return templates.TemplateResponse("index.html", {"request" : request})
# Connect the shorten.py route
app.include_router(ShortenRouter, tags = ["Test"], prefix = "/api/v1/shorten")
Write the route logic for displaying Hello world in the shorten.py file :-
from fastapi import APIRouter, Depends, HTTPException, Header
from typing import Optional
import schemas
router = APIRouter()
@router.get('/', response_model = str)
def test():
return "Hello world, FastAPI is running on PORT 8000. 🤩"
Write a simple template in templates/index.html :-
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>FastAPI and MongoDB URL Shortener</title>
<style></style>
</head>
<body>
<section class = "main">
<div class = "app__container">
<h1 class = "app__header">FastAPI URL Shortener</h1>
</div>
</section>
<script></script>
</body>
</html>
To run the uvicorn server :-
$ uvicorn main:app --reload
This runs the server on the default PORT 8000 and displays "Hello world, FastAPI is running on PORT 8000. 🤩" in the browser at localhost:8000/api/v1/shorten
You can also view the HTML template at localhost:8000 which displays the "FastAPI URL Shortener" header.
Great work so far 😃
STEP 3 :- Connecting to Mongo Atlas
We need to connect to the Mongo Atlas database to store and fetch data, this will be achieved using mongoengine (An Object-Document mapper for interacting with MongoDB databases, similar to SQLAlchemy for SQL databases).
To connect to Mongo Atlas :- Add the MONGO_URI you copied and BASE_URL to the .env file
#.env
BASE_URL = http://localhost:8000/
MONGO_URI = mongodb+srv://<yourusername>:<yourpassword>@cluster0.7wc8l.mongodb.net/<dbname>?retryWrites=true&w=majority
Use event handlers in FastAPI to connect to the database when the app starts. python-decouple will be used to read the MONGO_URI from the .env file created above. Add the following code to the main.py file :-
## main.py
import uvicorn
from fastapi import FastAPI, HTTPException, Depends, status, Request
from fastapi.middleware.cors import CORSMiddleware
from routes.shorten import router as ShortenRouter
from routes.redirect import router as RedirectRouter
from mongoengine import connect, disconnect, errors
from fastapi.responses import JSONResponse, HTMLResponse
from fastapi.templating import Jinja2Templates
from decouple import config
MONGO_URI = config('MONGO_URI')
print(MONGO_URI)
app = FastAPI()
# Templates
templates = Jinja2Templates(directory = "templates")
@app.get("/", response_class = HTMLResponse)
async def index(request : Request):
return templates.TemplateResponse("index.html", {"request" : request})
app.include_router(ShortenRouter, tags = ["Shorten long URL"], prefix = "/api/v1/shorten")
app.include_router(RedirectRouter, tags = ["Redirect to Short URL"])
@app.on_event("startup")
async def create_db_client():
try:
connect(host = MONGO_URI)
print("Successfully connected to the Mongo Atlas database.")
except Exception as e:
print(e)
print("An error occurred while connecting to Mongo Atlas database.")
@app.on_event("shutdown")
async def shutdown_db_client():
pass
The create_db_client() connects to the Mongo Atlas database when the app starts. This initiated connection will be used to handle all MongoDB operations. When you run the server again, you should see this in your CLI.
STEP 4 :- Creating the database and data validation schemas
The database schema basically defines a blueprint for how the data will be structured in the Mongo Atlas database. This schema will be created using mongoengine ODM. The data validation schema will be created using pydantic to define the validation schema for the request models.
(1) Creating the database schema using mongoengine Add this in the models.py file :-
from mongoengine import Document, StringField, IntField, DateTimeField, ObjectIdField
from bson import ObjectId
from datetime import datetime
class Url(Document):
id = ObjectId()
longUrl = StringField(required = True)
shortCode = StringField(required = True, unique = True)
shortUrl = StringField(required = True)
visitorCount = IntField(default = 0)
createdAt = DateTimeField(default = datetime.now())
The database schema Url inherits from the mongoengine Document and defines the blueprint for the required fields that will be stored in the db (id, longUrl, shortCode, shortUrl, visitorCount, createdAt) and their respective data-type. Check out the mongoengine documentation for a more comprehensive explanation.
(2) Creating the data validation schema using pydantic Add this in the schemas.py file :-
from typing import Optional
from pydantic import BaseModel, validator, constr
from datetime import datetime
from bson import ObjectId
import validators
class UrlSchema(BaseModel):
longUrl : str
customCode : Optional[constr(max_length = 8)] = None
@validator('longUrl')
def validate_url(cls, v):
if not validators.url(v):
raise ValueError("Long URL is invalid.")
return v
This basically defines the data validation schema for the request model using pydantic. The shorten endpoint is going to receive the longUrl and a customCode (optionally) and then perform the required logic to generate a random shortcode if a customCode is not supplied, the shortcode is then used to generate the shortened URL and these values are stored in the database.
The @validator('longUrl') decorator is used to define a custom validation method for the longUrl to check if it is a valid URL using the 'validators' package. The constr(max_length = 8) is also used to ensure that the passed customCode contains a maximum of 8 characters. Great work so far !, we are almost there. Now it is going to be a stroll in the park 😉.
STEP 5 :- Writing the logic for shortening the URL and saving to MongoDB
This is the logic for shortening the long URL, Add this to routes/shorten.py file :-
from fastapi import APIRouter, Depends, HTTPException, Header
from schemas import UrlSchema
from models import Url
import os
import shortuuid
from decouple import config
router = APIRouter()
@router.post('/', response_model = dict)
async def test(url : UrlSchema):
# convert pydantic schema object to python dict
url = dict(url)
# If a customCode is supplied, that will be the shortCode, else generate a random
#shortCode
if (url["customCode"]):
shortCode = url["customCode"]
else:
shortCode = shortuuid.ShortUUID().random(length = 8)
# Generate short URL by joining BASE_URL to shortCode
shortUrl = os.path.join(config("BASE_URL"), shortCode)
# Raise an exception if a record in the database already uses that shortCode
urlExists = Url.objects(shortCode = shortCode)
if len(urlExists) != 0:
raise HTTPException(status_code = 400, detail = "Short code is invalid, It has been used.")
try:
# Save the new Url object to the Mongo Atlas database
url = Url(
longUrl = url["longUrl"],
shortCode = shortCode,
shortUrl = shortUrl
)
url.save()
return {
"message" : "Successfully shortened URL.",
"shortUrl" : shortUrl,
"longUrl" : url["longUrl"]
}
except Exception as e:
print(e)
raise HTTPException(status_code = 500, detail = "An unknown error occurred.")
An 8-character long shortcode is generated if no customCode is supplied and used to generate the shortUrl using os.path.join(config("BASE_URL"), shortCode) where BASE_URL is configured in the .env file.
The url.save() method is used to create the new Url document in MongoDB
Testing this Endpoint :- Reload the uvicorn server. We shall test this endpoint using POSTMAN, this can also be tested using the interactive swagger docs at localhost:8000/docs
The endpoint is at /api/v1/shorten as defined in the main.py route prefix. We shall send a POST request with the required fields.
#CURL
curl --location --request POST 'http://localhost:8000/api/v1/shorten' \
--header 'Content-Type: application/json' \
--data-raw '{
"longUrl" : "https://meta.stackexchange.com/questions/118594/data-explorer-truncates-links-after-380-characters",
"customCode" : "test"
}'
It successfully returns the response if you pass in valid values, It returns a nice error conversely.
STEP 6 :- Writing the URL redirection logic
This is the final step for the API, now we want to redirect a user to the longUrl when the short URL is entered in the browser. So basically, what we will do is accept the short URL, read the shortCode from the path parameter and query the database for the document that matches that shortcode and then redirect to the longUrl for that document. We shall use RedirectResponse method from starlette.responses to redirect to the original longUrl.
Add this to routes/redirect.py :-
from fastapi import APIRouter, Depends, HTTPException, Header
from starlette.responses import RedirectResponse
from models import Url
from decouple import config
router = APIRouter()
@router.get("/{short_code}")
async def redirect_url(short_code : str):
# Query the database for the document that matches the short_code from the path param
url = Url.objects(
shortCode = short_code
)
# 404 ERROR if no url is found
if not url:
raise HTTPException(status_code= 404, detail = "URL not found !")
else:
# Increment visitor count by 1
url.update_one(upsert = True, inc__visitorCount = 1)
url = url[0].to_mongo().to_dict()
print(url["shortUrl"])
# Redirect to the longUrl
response = RedirectResponse(url = url["longUrl"])
return response
- The short_code is extracted from the path parameter and the database is queried to find the document that contains that shortCode, then a redirect is made to the longUrl for that particular document. It returns a 404 error if no document matches that short_code.
- The visitorCount is also incremented to monitor the number of times the URL has been clicked. So if you go to localhost:8000/test in your browser, it redirects you to the longUrl - meta.stackexchange.com/questions/118594/dat...
Great, we are finally done building the API endpoints. You deserve a bottle of cold beer for coming this far 😅😋 .
STEP 7 :- Connecting to a UI via Jinja2Templates (OPTIONAL)
This is a simple HTML template for shortening the URL from a form input and copying the shortened URL, The codes are available in the github gist below. The template basically contains a form which posts the data via an XMLHttpRequest to the /api/v1/shorten endpoint to shorten the Long URL, and returns the shortened URL.
View the github gist to copy the codes HERE
Conclusion
In this article, we built a URL shortener using FastAPI and MongoDB via the mongoengine ODM (Object Document Mapper). The complete codes for this article are available on Github and the app is publicly available on Heroku at shortlinq.herokuapp.com
References
- FastAPI Documentation - fastapi.tiangolo.com
- Mongoengine Documentation - docs.mongoengine.org
Thanks for reading ❤️, Kindly drop any questions/suggestions in the comment section.