Text of realpython.com › build-a-content-aggregator-python as Styx crawled it on Nov 20, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

Build a Content Aggregator in Python – Real Python

Build a Content Aggregator in Python by Ricky White Updated Nov 03, 2021 Reading time estimate 35m intermediate projects Table of Contents Remove ads In this project-based tutorial, you’ll build a content aggregator from scratch using Python and the popular framework Django . With so much content coming out online daily, it can be time consuming to go to multiple sites and sources to consume information about your favorite subjects.

This is why content aggregators are so popular and powerful, as you can use them to view all the latest news and content in one place. Whether you’re looking for a portfolio project or ways in which to extend future projects beyond simple CRUD capabilities, this tutorial will have something for you. In this tutorial, you’ll learn: How to work with RSS feeds How to create a Django custom management command How to run your custom command automatically on a schedule How to use unit tests to test your Django app’s functionality Click the link below to download the code for this project and follow along as you build your own content aggregator: Get Source Code: Click here to get the source code you’ll use to build a content aggregator with Django and Python in this tutorial.

Demo: What You’ll Build You’ll build your own podcast content aggregator in Python named pyCasts! by following this tutorial from start to finish. The application will be a single web page displaying the latest Python podcast episodes from The Real Python Podcast and the Talk Python to Me Podcast . After completing this tutorial, you can practice what you’ve learned by adding more podcast feeds to the application.

Here’s a quick demo video of how it will look in action: There are many moving parts behind the scenes that make this work efficiently and effectively in an automated fashion. You’ll learn about all of them in this tutorial. Get ready to delve in. Remove ads Project Overview In order to be able to display the content to the end user, you have several steps to follow: Set up the project Build the podcast model Create the homepage view Parse a podcast RSS feed Create a Django custom command Add additional feeds Schedule tasks with django-apscheduler You’ll walk through each of these over the course of this tutorial.

Now you’ll look at which technologies and frameworks you’re going to use for the above steps. In order to fetch the podcast RSS feeds into your application and parse them, you’ll learn how to use the feedparser library. You’ll use this library to extract only the newest episode data from the feed, which you’ll marshall into an Episode model and save to the database with the Django ORM. You could add this code to a script and manually run it periodically, but that would defeat the point of having an aggregator to save time.

Instead, you’ll learn how to use a built-in Django tool called a custom management command . To parse and save the data, you’ll run your code from within Django itself. With help from the django-apscheduler library, you’ll set a schedule for your function calls, which are also called jobs . You can then use the Django admin panel to view which jobs ran and when. This will ensure that the fetching and parsing of your feed will happen automatically without needing admin intervention.

You’ll then use the Django template engine to show the user the queried context—in other words, the latest episodes. Prerequisites To get the most out of this tutorial, you should be comfortable with the following concepts and technologies: Python basics Virtual environment setup and usage A basic level of HTML and CSS Django fundamentals, such as its folder structure , URL routing , migrations , and how to create a project and app You might also find it helpful to have some experience with Bootstrap 4 .

If you don’t have all the prerequisite knowledge before starting this tutorial, that’s okay! In fact, you might learn more by going ahead and just getting started. You can always stop and review the resources linked above if you get stuck. Step 1: Setting Up Your Project By the end of this step, you’ll have set up your environment, installed your dependencies, and finished getting Django up and running. Begin by making your project directory and then changing directory into it: Language: Shell $ mkdir pycasts $ cd pycasts Now that you’re inside your project directory, you should create your virtual environment and activate it.

Use whatever tool makes you happiest to do this. This example uses venv : Language: Shell $ python3 -m venv .venv $ source .venv/bin/activate (.venv) $ python -m pip install --upgrade pip With your environment now activated and pip upgraded, you’ll need to install the required dependencies to complete the project. You can find a requirements.txt file in the downloadable source code for this tutorial: Get Source Code: Click here to get the source code you’ll use to build a content aggregator with Django and Python in this tutorial.

Open the source_code_setup/ folder and install the pinned dependencies. Be sure to replace the <path_to_requirements.txt> with the actual path of your downloaded file: Language: Shell (.venv) $ python -m pip install -r <path_to_requirements.txt> You should now have Django, feedparser , django-apscheduler , and their sub-dependencies installed. Now that you have all the tools you need to get up and running, you can set up Django and start building.

To complete this step of the build, you need to do the following four things: Create your Django project in the current working directory, /pycasts Create a podcasts Django app Run initial migrations Create a superuser As you’re familiar with Django already, you won’t explore each of these steps in detail. You can go ahead and run the following commands: Language: Shell (.venv) $ django-admin startproject content_aggregator . (.venv) $ python manage.py startapp podcasts (.venv) $ python manage.py makemigrations && python manage.py migrate (.venv) $ python manage.py createsuperuser If it turns out that you do need to dig a little deeper to understand any of these terminal commands, you can check out Get Started With Django Part 1 .

Once you’ve followed Django’s prompts and finished creating your superuser account, you have one more change to make before testing that the app works. Although the application will run without it, don’t forget to add your new podcasts app to the settings.py file: Language: Python # content_aggregator/settings.py # ... INSTALLED_APPS = [ "django.contrib.admin" , "django.contrib.auth" , "django.contrib.contenttypes" , "django.contrib.sessions" , "django.contrib.messages" , "django.contrib.staticfiles" , # My Apps "podcasts.apps.PodcastsConfig" , ] You listed your new app in INSTALLED_APPS as "podcasts.apps.PodcastsConfig" .

Note: If you’re curious about why you’ve used the verbose podcasts.apps.PodcastsConfig instead of podcasts , then you can read more about configuring applications in the official Django docs. The TLDR; version is that although using the app name, podcasts , should work fine for this small application, it’s considered best practice to use the full AppConfig name. The time has come to take your new Django project for a spin.

Start the Django server: Language: Shell (.venv) $ python manage.py runserver Navigate to localhost:8000 in your browser and you should see Django’s default success page: Now that you’ve set up your project and you have Django working, continue to the next step. Remove ads Step 2: Building Your Podcast Model At this point, you should have your environment set up, your dependencies installed, and Django successfully running.

By the end of this step, you’ll have defined and tested a model for podcast episodes and migrated the model to the database. Your Episode model shouldn’t only reflect the information you’d like to capture as the developer. It should also reflect the information that the user would lik

The rest of this page is on the live site.