Getting Started with Back-end Development

This page will show you how to contribute to MetaGrid's back-end. You'll learn about the technologies used, the file structure scheme, and the style guide. Resources are also provided to get new contributors up to speed in the technologies.



Formatter and Linter



File Structure


Adapted from Cookiecutter Django's file structure scheme

├── .envs
│ ├── .django
│ ├── .keykloak
│ ├── .postgres
├── config
│ ├── settings
│ │ ├──
│ │ ├──
│ │ ├──
│ │ └──
│ ├──
│ └──
├── docker
│ ├── local
│ └── production
├── metagrid
│ ├──
│ ├── api_proxy
│ ├── cart
│ ├── mysites
│ ├── projects
│ └── users
├── requirements
│ ├── base.txt
│ ├── local.txt
│ └── production.txt
├── .coveragerc
├── .dockerignore
├── .editorconfig
├── .gitignore
├── docker-compose.yml
├── mkdocs.yml
├── pyproject.toml
├── setup.cfg
  • .envs/ - stores environment variables for each microservice found in the docker-compose files, for use in local development.
  • config/ - stores Django configuration files
    • settings/ - stores Django settings files
      • - base setting shared across local development and production environments
      • - extends with local development environment settings
      • - extends with production environment settings
      • - extends with test environment settings
    • - provides URL mapping to Django views
    • - interface between application server to connect with Django. Used primarily in deployment.
  • docker/ - stores files used by each microservice found in the docker-compose files, including DockerFiles, start scripts, etc, separated by environment and service
  • docs/ - stores documentation files for the project
  • metagrid/ - stores Django apps
    • - used to pre-populate the postgres database with project related groups, facets etc. If modified, then you must run the script for the database to be updated.
    • api_proxy/ - handles the proxy communication between the backend and outside sources like esgf, citations etc.
    • cart/ - handles the data cart related models and views
    • mysites/ - only hosts data migrations for internal Django apps and external third-party apps, not a typical standalone Django app
    • projects/ - handles project related models and views, including reading in the initial project data for the database.
    • users/ - handles user related data such as accounts and profiles
  • .coveragerc - configuration file for (used by pytest-cov)
  • .dockerignore - files and folders to ignore when building docker containers
  • .editorconfig - configuration file for unifying coding style for different editors and IDEs
  • - the production build of docker-compose
  • docker-compose.yml - the local development build of docker-compose
  • - a command-line utility that lets you interact with this Django project in various ways, such as starting the project or running migrations
  • mkdocs.yml - configuration file for MkDocs
  • pyproject.toml - configuration for system dependencies written in the TOML format. In MetaGrid, it is specifically used to configure Black which does not support setup.cfg
  • setup.cfg - configuration for system dependencies such as flake8 and mypy. setup.cfg is preferred over using individual config files because it centralizes configuration
  • - a script which updates the database using For example, if a new project has come out, or an existing one has had changes or been removed, the would need to be updated and then this script is run to migrate the django database.

Django Apps

What are Django Apps?

General Practices

  • Django apps should by tightly focused on its task rather than a combination of multiple tasks
  • Keep Django apps small. If an app becomes too complex, break it up.
  • App's name should be a plural version of the app's main model (e.g. users for User model)
    • Use valid, PEP8 complain and importable Python package names, which are short, all-lowercase names
    • Use underscores for readability, but avoid in most cases.

Starting an App

Run the command to start an app

cd metagrid
docker compose -p metagrid_backend_dev run --rm django python startapp <app_name>

Register the app under INSTALLED_APPS

    # Your apps

App File Structure

Below is an example of how MetaGrid scaffolds Django apps.

└── metagrid
    └── cart
        ├── migrations
        ├── tests
        │   ├──
        │   ├──
        │   ├──
        │   ├──
        │   ├──
        │   └──

* denotes file is auto-generated after running python startapp <APP_NAME>

  • \migrations - stores Django migration files for the app's models
  • \tests - stores test related files
    • - stores factories, which are fixtures based on your Django models for testing purposes
  •* - used to display your models in the Django admin panel. You can also customize your admin panel
  •* - created to help the user include any application configuration for the app. Using this, you can configure some of the attributes of the application
  •* - stores Django models
  • - stores Django REST Framework serializers
  •* - stores Django REST Framework views, generic views, and viewsets
    • If becomes too large from storing views and viewsets for example, you can seperate viewsets in a

Building REST APIs in an App

After adding an app, you can start building REST APIs for the frontend. Here's the typical flow for creating an API.

  1. Add models
  2. (If needed) Link user-related models to User model through OnetoOneField
    • Make sure to update the User model's save() method (e.g. add Cart.objects.create(user=self)). When a new User object is saved, it will map a new Cart object automatically
    • You need to create and run a data migration to create new Cart objects that map to any existing User objects. Make sure to have a reverse operation for RunPython in case you need to reverse the migration
  3. Add model serializers and/or serializers
  4. Add viewsets, generic class-based views, and/or views,
    • Viewsets and generic class-based views are the preferred way to write APIs because it abstracts a lot of boilerplate code, making code simpler and cleaner
    • If you need custom behaviors in a view and viewsets/generic class-based views aren't cutting it, then use the APIView class
  5. Register your viewsets to the router in and/or add your views/generic class-based views to the list
  6. Run pytest and open htmlcov/index.html to view code coverage
  7. Write tests for 100% coverage

Testing the App

  1. Delete the autogenerated file
  2. Create a tests folder to store all your test files
  3. If needed, add,,,
    • The convention is to add an associated test_ file for a file that needs to be tested

Creating and Applying Model Migrations (Database Version Control)

What are migrations?

Migrations are Django’s way of propagating changes you make to your models (adding a field, deleting a model, etc.) into your database schema. They’re designed to be mostly automatic, but you’ll need to know when to make migrations, when to run them, and the common problems you might run into.

Style Guide

MetaGrid's back-end follows the Black code style. Please read through Black's code style guide.

Style guide and linting rules are enforced in CI test builds..

Useful Django Commands

Run a command inside the docker container:

docker compose -p metagrid_backend_dev run --rm django [command]

Django migrations

Make migrations

python makemigrations your_app_name
  • You specify the app using your_app_name, or omit to run on all

Make data migration

  • Useful for changing the data in the database itself, in conjunction with the schema if you want
python makemigrations --empty your_app_name

Show migrations

python showmigrations

Run migrations

python migrate your_app_name
  • You specify the app using your_app_name, or omit to run on all

Creating a Superuser

Useful for logging into Django Admin page to manage the database

python createsuperuser

Show URLs

Produce a tab-separated list of (url_pattern, view_function, name) tuples. Useful for writing tests and testing REST APIs using curl, Postman, etc.

python show_urls

Run the Enhanced Django Shell

Useful for prototyping and testing code

Run the enhanced django shell:

python shell_plus

Run Tests and Produce Coverage Report

To run the tests, check your test coverage, and generate an HTML coverage report:

# Optional, stop existing Django containers so tests can run without conflicts
docker compose -f docker-compose.yml down
# Runs the tests
docker compose -p metagrid_backend_dev run --rm django pytest

Note: Run commands above within the 'metagrid/backend' directory.

The HTML coverage report is located here: htmlcov/index.html.

Format Code

Format with black

black .

Sort imports with isort

isort .

Type Checks

mypy metagrid


flake8 .

New Contributor Resources

Here are some resources to get you up to speed with the technologies.



Tutorials and Cheatsheets