datatracker/docker

Datatracker Development in Docker

Getting started

1. Set up Docker on your preferred platform. On Windows, it is highly recommended to use the WSL 2 (Windows Subsystem for Linux) backend.

See the IETF Tools Windows Dev guide on how to get started when using Windows.

2. On Linux, you must install Docker Compose manually and not install Docker Desktop. On Mac and Windows install Docker Desktop which already includes Docker Compose.

3. If you have a copy of the datatracker code checked out already, simply cd to the top-level directory.

   If not, check out a datatracker branch as usual. We'll check out main below, but you can use any branch:

   git clone https://github.com/ietf-tools/datatracker.git
   cd datatracker
   git checkout main
   

4. Follow the instructions for your preferred editor:

   • Visual Studio Code
   • Other Editors / Generic

Using Visual Studio Code

This project includes a devcontainer configuration which automates the setup of the development environment with all the required dependencies.

Initial Setup

1. Launch VS Code
2. Under the Extensions tab, ensure you have the Dev Containers (ms-vscode-remote.remote-containers) extension installed.

• On Linux, note that the Snap installation of VS Code is incompatible with this plugin.
• On Windows, you also need the WSL (ms-vscode-remote.remote-wsl) extension to take advantage of the WSL 2 (Windows Subsystem for Linux) native integration.

2. Open the top-level directory of the datatracker code you fetched above.
3. A prompt inviting you to reopen the project in containers will appear in the bottom-right corner. Click the Reopen in Container button. If you missed the prompt, you can press F1, start typing reopen in container task and launch it.
4. VS Code will relaunch in the dev environment and create the containers automatically.
5. You may get several warnings prompting you to reload the window as extensions get installed for the first time. Wait for the initialization script to complete before doing so. (Wait for the message Done! to appear in the terminal panel.)

Subsequent Launch

To return to your dev environment created above, simply open VS Code and select File > Open Recent and select the datatracker folder with the [Dev Container] suffix.

You can also open the datatracker project folder and click the Reopen in container button when prompted. If you missed the prompt, you can press F1, start typing reopen in container task and launch it.

Usage

• Under the Run and Debug tab, you can run the server with the debugger attached using Run Server (F5). Once the server is ready to accept connections, you'll be prompted to open in a browser. Navigate to http://localhost:8000 in your preferred browser.

  An alternate profile Run Server with Debug Toolbar is also available from the dropdown menu, which displays various tools on top of the webpage. However, note that this configuration has a significant performance impact.

  To add a Breakpoint, simply click to the left of the line gutter you wish to stop at. You can also add Conditional Breakpoints and Logpoint by right-clicking at the same location.

  

  While running in debug mode (F5), the following toolbar is shown at the top of the editor:

  

  See this tutorial on how to use the debugging tools for Django in VS Code.

• An integrated terminal is available with various shell options (zsh, bash, fish, etc.). Use the New Terminal button located at the right side of the Terminal panel. You can have as many as needed running in parallel and you can use split to display multiple at once.

  

• The pgAdmin web interface, a PostgreSQL DB browser / management UI, is available at http://localhost:8000/pgadmin/.

• Under the Task Explorer tab, a list of available preconfigured tasks is displayed. (You may need to expand the tree to src > vscode to see it.) These are common scritps you can run (e.g. run tests, fetch assets, etc.).

  

• From the command palette (F1), the command Run Test Task allows you to choose between running all tests or just the javascript tests.

• The Ports panel, found in the Terminal area, shows the ports currently mapped to your host and if they are currently listening.

  

Using Other Editors / Generic

1. From the terminal, in the top-level directory of the datatracker project:

   On Linux / macOS:

   ./docker/run # or whatever path you need
   

   Note that you can pass the -r flag to run to force a rebuild of the containers. This is useful if you switched branches and that the existing containers still contain configurations from the old branch. You should also use this if you don't regularly keep up with main and your containers reflect a much older version of the branch.

   On Windows (using Powershell):

   Copy-Item "docker/docker-compose.extend.yml" -Destination "docker/docker-compose.extend-custom.yml"
   (Get-Content -path docker/docker-compose.extend-custom.yml -Raw) -replace 'CUSTOM_PORT','8000' | Set-Content -Path docker/docker-compose.extend-custom.yml
   docker compose -f docker-compose.yml -f docker/docker-compose.extend-custom.yml up -d
   docker compose exec app /bin/sh /docker-init.sh
   

2. Wait for the containers to initialize. Upon completion, you will be dropped into a shell from which you can start the datatracker and execute related commands as usual, for example

   ietf/manage.py runserver 8001
   

   to start the datatracker.

   Once the datatracker has started, you should be able to open http://localhost:8000 in a browser and see the landing page.

   Note that unlike the VS Code setup, a debug SMTP server is launched automatically. Any email will be discarded and logged to the shell.

Exit Environment

To exit the dev environment, simply enter command exit in the shell.

The containers will automatically be shut down on Linux / macOS.

On Windows, type the command

docker compose down

to terminate the containers.

Clean and Rebuild DB from latest image

To delete the active DB container, its volume and get the latest image / DB dump, simply run the following command:

On Linux / macOS:

cd docker
./cleandb

On Windows:

docker compose down -v
docker compose pull db
docker compose build --no-cache db

Clean all

To delete all containers for this project, its associated images and purge any remaining dangling images, simply run the following command:

On Linux / macOS:

cd docker
./cleanall

On Windows:

docker compose down -v --rmi all
docker image prune

Updating an older environment

If you already have a clone, such as from a previous codesprint, and are updating that clone, before starting the datatracker from the updated image:

• rm ietf/settings_local.py # The startup script will put a new one, appropriate to the current release, in place
• Execute the Clean all sequence above.

Accessing PostgreSQL Port

The port is exposed but not automatically mapped to 5432 to avoid potential conflicts with the host. To get the mapped port, run the command (from the project /docker directory):

docker compose port db 5432

Notes / Troubleshooting

Slow zsh prompt inside Docker

On Windows, the zsh prompt can become incredibly slow because of the git status check displayed as part of the prompt. To remove this delay, run the command:

git config oh-my-zsh.hide-info 1

Windows .ics files incorrectly linked

When checking out the project on Windows, the .ics files are not correctly linked and will cause many tests to fail. To fix this issue, run the Fix Windows Timezone File Linking task in VS Code or run manually the script docker/scripts/app-win32-timezone-fix.sh

The content of the source files will be copied into the target .ics files. Make sure not to add these modified files when committing code!

Missing assets in the data folder

Because including all assets in the image would significantly increase the file size, they are not included by default. You can however fetch them by running the Fetch assets via rsync task in VS Code or run manually the script docker/scripts/app-rsync-extras.sh

Linux file permissions leaking to the host system

If on the host filesystem you have permissions that look like this,

$ ls -la
total 4624
drwxrwxr-x   2  100999  100999    4096 May 25 07:56 bin
drwxrwxr-x   5  100999  100999    4096 May 25 07:56 client
(etc...)

Try uninstalling Docker Desktop and installing Docker Compose manually. The Docker Compose bundled with Docker Desktop is incompatible with our software. See also Rootless Docker: file ownership changes #3343, Docker context desktop-linux has container permission issues #75.