datatracker/docker/scripts/app-init.sh
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00

135 lines
4.3 KiB
Bash
Executable file

#!/bin/bash
WORKSPACEDIR="/workspace"
# Handle Linux host mounting the workspace dir as root
if [ ! -O "${WORKSPACEDIR}/ietf" ]; then
sudo chown -R dev:dev $WORKSPACEDIR
fi
# Start rsyslog service
sudo service rsyslog start &>/dev/null
# Add /workspace as a safe git directory
git config --global --add safe.directory /workspace
# Turn off git info in zsh prompt (causes slowdowns)
git config oh-my-zsh.hide-info 1
# Fix ownership of volumes
echo "Fixing volumes ownership..."
sudo chown -R dev:dev "$WORKSPACEDIR/.parcel-cache"
sudo chown -R dev:dev "$WORKSPACEDIR/__pycache__"
sudo chown -R dev:dev "$WORKSPACEDIR/.vite"
sudo chown -R dev:dev "$WORKSPACEDIR/.yarn/unplugged"
sudo chown dev:dev "/assets"
# Run nginx
echo "Starting nginx..."
sudo nginx
# Build node packages that requrie native compilation
echo "Compiling native node packages..."
yarn rebuild
# Silence Browserlist warnings
export BROWSERSLIST_IGNORE_OLD_DATA=1
# Generate static assets
echo "Building static assets... (this could take a minute or two)"
yarn build
yarn legacy:build
# Copy config files if needed
cp $WORKSPACEDIR/docker/configs/settings_postgresqldb.py $WORKSPACEDIR/ietf/settings_postgresqldb.py
if [ ! -f "$WORKSPACEDIR/ietf/settings_local.py" ]; then
echo "Setting up a default settings_local.py ..."
else
echo "Renaming existing ietf/settings_local.py to ietf/settings_local.py.bak"
mv -f $WORKSPACEDIR/ietf/settings_local.py $WORKSPACEDIR/ietf/settings_local.py.bak
fi
cp $WORKSPACEDIR/docker/configs/settings_local.py $WORKSPACEDIR/ietf/settings_local.py
if [ ! -f "$WORKSPACEDIR/ietf/settings_local_debug.py" ]; then
echo "Setting up a default settings_local_debug.py ..."
else
echo "Renaming existing ietf/settings_local_debug.py to ietf/settings_local_debug.py.bak"
mv -f $WORKSPACEDIR/ietf/settings_local_debug.py $WORKSPACEDIR/ietf/settings_local_debug.py.bak
fi
cp $WORKSPACEDIR/docker/configs/settings_local_debug.py $WORKSPACEDIR/ietf/settings_local_debug.py
if [ ! -f "$WORKSPACEDIR/ietf/settings_local_vite.py" ]; then
echo "Setting up a default settings_local_vite.py ..."
else
echo "Renaming existing ietf/settings_local_vite.py to ietf/settings_local_vite.py.bak"
mv -f $WORKSPACEDIR/ietf/settings_local_vite.py $WORKSPACEDIR/ietf/settings_local_vite.py.bak
fi
cp $WORKSPACEDIR/docker/configs/settings_local_vite.py $WORKSPACEDIR/ietf/settings_local_vite.py
# Create data directories
echo "Creating data directories..."
chmod +x ./docker/scripts/app-create-dirs.sh
./docker/scripts/app-create-dirs.sh
# Configure the development blobstore
echo "Configuring blobstore..."
PYTHONPATH=/workspace python ./docker/scripts/app-configure-blobstore.py
# Download latest coverage results file
echo "Downloading latest coverage results file..."
curl -fsSL https://github.com/ietf-tools/datatracker/releases/download/baseline/coverage.json -o release-coverage.json
# Wait for DB container
if [ -n "$EDITOR_VSCODE" ]; then
echo "Waiting for DB container to come online ..."
/usr/local/bin/wait-for db:5432 -- echo "PostgreSQL ready"
fi
# Run memcached
echo "Starting memcached..."
/usr/bin/memcached -u dev -d
# Initial checks
echo "Running initial checks..."
/usr/local/bin/python $WORKSPACEDIR/ietf/manage.py check --settings=settings_local
# Migrate, adjusting to what the current state of the underlying database might be:
/usr/local/bin/python $WORKSPACEDIR/ietf/manage.py migrate --fake-initial --settings=settings_local
if [ -z "$EDITOR_VSCODE" ]; then
CODE=0
python -m smtpd -n -c DebuggingServer localhost:2025 &
if [ -z "$*" ]; then
echo "-----------------------------------------------------------------"
echo "Ready!"
echo "-----------------------------------------------------------------"
echo
echo "You can execute arbitrary commands now, e.g.,"
echo
echo " ietf/manage.py runserver 8001"
echo
echo "to start a development instance of the Datatracker."
echo
echo " ietf/manage.py test --settings=settings_test"
echo
echo "to run all the python tests."
echo
zsh
else
echo "Executing \"$*\" and stopping container."
echo
zsh -c "$*"
CODE=$?
fi
sudo service rsyslog stop
exit $CODE
fi