Getting Started with Back-end Development

This page will show you how to contribute to MetaGrid's back-end. You'll learn about the technologies used, the file structure scheme, and the style guide. Resources are also provided to get new contributors up to speed in the technologies.

Technologies

Core

Formatter and Linter

Formatter: Black
Linter: Flake8
Static Type Checker: mypy

Testing/QA

Test Runner: pytest
Code Coverage: Coverage.py (through pytest-cov package)

DevOps

File Structure

Root

Adapted from Cookiecutter Django's file structure scheme

backend
├── .envs
│ ├── .django
│ ├── .keykloak
│ ├── .postgres
├── config
│ ├── settings
│ │ ├── base.py
│ │ ├── local.py
│ │ ├── production.py
│ │ └── test.py
│ ├── urls.py
│ └── wsgi.py
├── docker
│ ├── local
│ └── production
├── metagrid
│ ├── initial_projects_data.py
│ ├── api_proxy
│ ├── cart
│ ├── mysites
│ ├── projects
│ └── users
├── requirements
│ ├── base.txt
│ ├── local.txt
│ └── production.txt
├── .coveragerc
├── .dockerignore
├── .editorconfig
├── .gitignore
├── docker-compose.prod.yml
├── docker-compose.yml
├── manage.py
├── mkdocs.yml
├── pyproject.toml
├── setup.cfg
└── updateProjects.sh

.envs/ - stores environment variables for each microservice found in the docker-compose files, for use in local development.
config/ - stores Django configuration files
- settings/ - stores Django settings files
  - base.py - base setting shared across local development and production environments
  - local.py - extends base.py with local development environment settings
  - production.py - extends base.py with production environment settings
  - test.py - extends base.py with test environment settings
- urls.py - provides URL mapping to Django views
- wsgi.py - interface between application server to connect with Django. Used primarily in deployment.
docker/ - stores files used by each microservice found in the docker-compose files, including DockerFiles, start scripts, etc, separated by environment and service
docs/ - stores documentation files for the project
metagrid/ - stores Django apps
- initial_projects_data.py - used to pre-populate the postgres database with project related groups, facets etc. If modified, then you must run the updateProjects.sh script for the database to be updated.
- api_proxy/ - handles the proxy communication between the backend and outside sources like esgf, citations etc.
- cart/ - handles the data cart related models and views
- mysites/ - only hosts data migrations for internal Django apps and external third-party apps, not a typical standalone Django app
- projects/ - handles project related models and views, including reading in the initial project data for the database.
- users/ - handles user related data such as accounts and profiles
.coveragerc - configuration file for coverage.py (used by pytest-cov)
.dockerignore - files and folders to ignore when building docker containers
.editorconfig - configuration file for unifying coding style for different editors and IDEs
docker-compose.prod.yml - the production build of docker-compose
docker-compose.yml - the local development build of docker-compose
manage.py - a command-line utility that lets you interact with this Django project in various ways, such as starting the project or running migrations
mkdocs.yml - configuration file for MkDocs
pyproject.toml - configuration for system dependencies written in the TOML format. In MetaGrid, it is specifically used to configure Black which does not support setup.cfg
setup.cfg - configuration for system dependencies such as flake8 and mypy. setup.cfg is preferred over using individual config files because it centralizes configuration
updateProjects.sh - a script which updates the database using initial_project_data.py. For example, if a new project has come out, or an existing one has had changes or been removed, the intial_project_data.py would need to be updated and then this script is run to migrate the django database.

Django Apps

What are Django Apps?

General Practices

Django apps should by tightly focused on its task rather than a combination of multiple tasks
Keep Django apps small. If an app becomes too complex, break it up.
App's name should be a plural version of the app's main model (e.g. users for User model)
- Use valid, PEP8 complain and importable Python package names, which are short, all-lowercase names
- Use underscores for readability, but avoid in most cases.

Starting an App

Run the command to start an app

cd metagrid
docker-compose -p metagrid_backend_dev run --rm django python manage.py startapp <app_name>

Register the app under INSTALLED_APPS

INSTALLED_APPS =[
    ...
    # Your apps
    "metagrid.users",
    "metagrid.app_name"
]

App File Structure

Below is an example of how MetaGrid scaffolds Django apps.

backend
└── metagrid
    └── cart
        ├── migrations
        ├── tests
        │   ├── __init__.py
        │   ├── factories.py
        │   ├── test_models.py
        │   ├── test_serializers.py
        │   ├── test_urls.py
        │   └── test_views.py
        ├── __init__.py
        ├── admin.py
        ├── app.py
        ├── models.py
        ├── serializers.py
        └── views.py

* denotes file is auto-generated after running python manage.py startapp <APP_NAME>

\migrations - stores Django migration files for the app's models
\tests - stores test related files
- factories.py - stores factories, which are fixtures based on your Django models for testing purposes
admin.py* - used to display your models in the Django admin panel. You can also customize your admin panel
app.py* - created to help the user include any application configuration for the app. Using this, you can configure some of the attributes of the application
models.py* - stores Django models
serializers.py - stores Django REST Framework serializers
views.py* - stores Django REST Framework views, generic views, and viewsets
- If views.py becomes too large from storing views and viewsets for example, you can seperate viewsets in a viewsets.py

Building REST APIs in an App

After adding an app, you can start building REST APIs for the frontend. Here's the typical flow for creating an API.

Add models
(If needed) Link user-related models to User model through OnetoOneField
- Make sure to update the User model's save() method (e.g. add Cart.objects.create(user=self)). When a new User object is saved, it will map a new Cart object automatically
- You need to create and run a data migration to create new Cart objects that map to any existing User objects. Make sure to have a reverse operation for RunPython in case you need to reverse the migration
Add model serializers and/or serializers
Add viewsets, generic class-based views, and/or views,
- Viewsets and generic class-based views are the preferred way to write APIs because it abstracts a lot of boilerplate code, making code simpler and cleaner
- If you need custom behaviors in a view and viewsets/generic class-based views aren't cutting it, then use the APIView class
Register your viewsets to the router in urls.py and/or add your views/generic class-based views to the urls.py list
Run pytest and open htmlcov/index.html to view code coverage
Write tests for 100% coverage

Testing the App

Delete the autogenerated tests.py file
Create a tests folder to store all your test files
If needed, add test_models.py, test_serializers.py, test_views.py, test_urls.py
- The convention is to add an associated test_ file for a file that needs to be tested

Creating and Applying Model Migrations (Database Version Control)

What are migrations?

Migrations are Django’s way of propagating changes you make to your models (adding a field, deleting a model, etc.) into your database schema. They’re designed to be mostly automatic, but you’ll need to know when to make migrations, when to run them, and the common problems you might run into.

Style Guide

MetaGrid's back-end follows the Black code style. Please read through Black's code style guide.

Style guide and linting rules are enforced in CI test builds..

Useful Django Commands

Run a command inside the docker container:

docker-compose -p metagrid_backend_dev run --rm django [command]

Django migrations

Make migrations

python manage.py makemigrations your_app_name

You specify the app using your_app_name, or omit to run on all

Make data migration

https://docs.djangoproject.com/en/3.1/topics/migrations/#data-migrations
Useful for changing the data in the database itself, in conjunction with the schema if you want

python manage.py makemigrations --empty your_app_name

Show migrations

python manage.py showmigrations

Run migrations

python manage.py migrate your_app_name

You specify the app using your_app_name, or omit to run on all

Creating a Superuser

Useful for logging into Django Admin page to manage the database

python manage.py createsuperuser

Show URLs

Produce a tab-separated list of (url_pattern, view_function, name) tuples. Useful for writing tests and testing REST APIs using curl, Postman, etc.

python manage.py show_urls

Run the Enhanced Django Shell

Useful for prototyping and testing code

Run the enhanced django shell:

python manage.py shell_plus

Run Tests and Produce Coverage Report

To run the tests, check your test coverage, and generate an HTML coverage report:

# Optional, stop existing Django containers so tests can run without conflicts
docker-compose -f docker-compose.yml down
# Runs the tests
docker-compose -p metagrid_backend_dev run --rm django pytest

Note: Run commands above within the 'metagrid/backend' directory.

The HTML coverage report is located here: htmlcov/index.html.

Format Code

Format with black

black .

Sort imports with isort

isort .

Type Checks

mypy metagrid

Linting

flake8 .