* feat: basic blobstore infrastructure for dev * refactor: (broken) attempt to put minio console behind nginx * feat: initialize blobstore with boto3 * fix: abandon attempt to proxy minio. Use docker compose instead. * feat: beginning of blob writes * feat: storage utilities * feat: test buckets * chore: black * chore: remove unused import * chore: avoid f string when not needed * fix: inform all settings files about blobstores * fix: declare types for some settings * ci: point to new target base * ci: adjust test workflow * fix: give the tests debug environment a blobstore * fix: "better" name declarations * ci: use devblobstore container * chore: identify places to write to blobstorage * chore: remove unreachable code * feat: store materials * feat: store statements * feat: store status changes * feat: store liaison attachments * feat: store agendas provided with Interim session requests * chore: capture TODOs * feat: store polls and chatlogs * chore: remove unneeded TODO * feat: store drafts on submit and post * fix: handle storage during doc expiration and resurrection * fix: mirror an unlink * chore: add/refine TODOs * feat: store slide submissions * fix: structure slide test correctly * fix: correct sense of existence check * feat: store some indexes * feat: BlobShadowFileSystemStorage * feat: shadow floorplans / host logos to the blob * chore: remove unused import * feat: strip path from blob shadow names * feat: shadow photos / thumbs * refactor: combine photo and photothumb blob kinds The photos / thumbs were already dropped in the same directory, so let's not add a distinction at this point. * style: whitespace * refactor: use kwargs consistently * chore: migrations * refactor: better deconstruct(); rebuild migrations * fix: use new class in mack patch * chore: add TODO * feat: store group index documents * chore: identify more TODO * feat: store reviews * fix: repair merge * chore: remove unnecessary TODO * feat: StoredObject metadata * fix: deburr some debugging code * fix: only set the deleted timestamp once * chore: correct typo * fix: get_or_create vs get and test * fix: avoid the questionable is_seekable helper * chore: capture future design consideration * chore: blob store cfg for k8s * chore: black * chore: copyright * ci: bucket name prefix option + run Black Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes are just Black styling. * ci: fix typo in bucket name expression * chore: parameters in app-configure-blobstore Allows use with other blob stores. * ci: remove verify=False option * fix: don't return value from __init__ * feat: option to log timing of S3Storage calls * chore: units * fix: deleted->null when storing a file * style: Black * feat: log as JSON; refactor to share code; handle exceptions * ci: add ietf_log_blob_timing option for k8s * test: --no-manage-blobstore option for running tests * test: use blob store settings from env, if set * test: actually set a couple more storage opts * feat: offswitch (#8541) * feat: offswitch * fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior * chore: log timing of blob reads * chore: import Config from botocore.config * chore(deps): import boto3-stubs / botocore botocore is implicitly imported, but make it explicit since we refer to it directly * chore: drop type annotation that mypy loudly ignores * refactor: add storage methods via mixin Shares code between Document and DocHistory without putting it in the base DocumentInfo class, which lacks the name field. Also makes mypy happy. * feat: add timeout / retry limit to boto client * ci: let k8s config the timeouts via env * chore: repair merge resolution typo * chore: tweak settings imports * chore: simplify k8s/settings_local.py imports --------- Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
135 lines
4.3 KiB
Bash
Executable file
135 lines
4.3 KiB
Bash
Executable file
#!/bin/bash
|
|
|
|
WORKSPACEDIR="/workspace"
|
|
|
|
# Handle Linux host mounting the workspace dir as root
|
|
if [ ! -O "${WORKSPACEDIR}/ietf" ]; then
|
|
sudo chown -R dev:dev $WORKSPACEDIR
|
|
fi
|
|
|
|
# Start rsyslog service
|
|
sudo service rsyslog start &>/dev/null
|
|
|
|
# Add /workspace as a safe git directory
|
|
git config --global --add safe.directory /workspace
|
|
|
|
# Turn off git info in zsh prompt (causes slowdowns)
|
|
git config oh-my-zsh.hide-info 1
|
|
|
|
# Fix ownership of volumes
|
|
echo "Fixing volumes ownership..."
|
|
sudo chown -R dev:dev "$WORKSPACEDIR/.parcel-cache"
|
|
sudo chown -R dev:dev "$WORKSPACEDIR/__pycache__"
|
|
sudo chown -R dev:dev "$WORKSPACEDIR/.vite"
|
|
sudo chown -R dev:dev "$WORKSPACEDIR/.yarn/unplugged"
|
|
sudo chown dev:dev "/assets"
|
|
|
|
# Run nginx
|
|
echo "Starting nginx..."
|
|
sudo nginx
|
|
|
|
# Build node packages that requrie native compilation
|
|
echo "Compiling native node packages..."
|
|
yarn rebuild
|
|
|
|
# Silence Browserlist warnings
|
|
export BROWSERSLIST_IGNORE_OLD_DATA=1
|
|
|
|
# Generate static assets
|
|
echo "Building static assets... (this could take a minute or two)"
|
|
yarn build
|
|
yarn legacy:build
|
|
|
|
# Copy config files if needed
|
|
cp $WORKSPACEDIR/docker/configs/settings_postgresqldb.py $WORKSPACEDIR/ietf/settings_postgresqldb.py
|
|
|
|
if [ ! -f "$WORKSPACEDIR/ietf/settings_local.py" ]; then
|
|
echo "Setting up a default settings_local.py ..."
|
|
else
|
|
echo "Renaming existing ietf/settings_local.py to ietf/settings_local.py.bak"
|
|
mv -f $WORKSPACEDIR/ietf/settings_local.py $WORKSPACEDIR/ietf/settings_local.py.bak
|
|
fi
|
|
cp $WORKSPACEDIR/docker/configs/settings_local.py $WORKSPACEDIR/ietf/settings_local.py
|
|
|
|
if [ ! -f "$WORKSPACEDIR/ietf/settings_local_debug.py" ]; then
|
|
echo "Setting up a default settings_local_debug.py ..."
|
|
else
|
|
echo "Renaming existing ietf/settings_local_debug.py to ietf/settings_local_debug.py.bak"
|
|
mv -f $WORKSPACEDIR/ietf/settings_local_debug.py $WORKSPACEDIR/ietf/settings_local_debug.py.bak
|
|
fi
|
|
cp $WORKSPACEDIR/docker/configs/settings_local_debug.py $WORKSPACEDIR/ietf/settings_local_debug.py
|
|
|
|
if [ ! -f "$WORKSPACEDIR/ietf/settings_local_vite.py" ]; then
|
|
echo "Setting up a default settings_local_vite.py ..."
|
|
else
|
|
echo "Renaming existing ietf/settings_local_vite.py to ietf/settings_local_vite.py.bak"
|
|
mv -f $WORKSPACEDIR/ietf/settings_local_vite.py $WORKSPACEDIR/ietf/settings_local_vite.py.bak
|
|
fi
|
|
cp $WORKSPACEDIR/docker/configs/settings_local_vite.py $WORKSPACEDIR/ietf/settings_local_vite.py
|
|
|
|
# Create data directories
|
|
|
|
echo "Creating data directories..."
|
|
chmod +x ./docker/scripts/app-create-dirs.sh
|
|
./docker/scripts/app-create-dirs.sh
|
|
|
|
# Configure the development blobstore
|
|
|
|
echo "Configuring blobstore..."
|
|
PYTHONPATH=/workspace python ./docker/scripts/app-configure-blobstore.py
|
|
|
|
# Download latest coverage results file
|
|
|
|
echo "Downloading latest coverage results file..."
|
|
curl -fsSL https://github.com/ietf-tools/datatracker/releases/download/baseline/coverage.json -o release-coverage.json
|
|
|
|
# Wait for DB container
|
|
|
|
if [ -n "$EDITOR_VSCODE" ]; then
|
|
echo "Waiting for DB container to come online ..."
|
|
/usr/local/bin/wait-for db:5432 -- echo "PostgreSQL ready"
|
|
fi
|
|
|
|
# Run memcached
|
|
|
|
echo "Starting memcached..."
|
|
/usr/bin/memcached -u dev -d
|
|
|
|
# Initial checks
|
|
|
|
echo "Running initial checks..."
|
|
/usr/local/bin/python $WORKSPACEDIR/ietf/manage.py check --settings=settings_local
|
|
|
|
# Migrate, adjusting to what the current state of the underlying database might be:
|
|
|
|
/usr/local/bin/python $WORKSPACEDIR/ietf/manage.py migrate --fake-initial --settings=settings_local
|
|
|
|
if [ -z "$EDITOR_VSCODE" ]; then
|
|
CODE=0
|
|
python -m smtpd -n -c DebuggingServer localhost:2025 &
|
|
if [ -z "$*" ]; then
|
|
echo "-----------------------------------------------------------------"
|
|
echo "Ready!"
|
|
echo "-----------------------------------------------------------------"
|
|
echo
|
|
echo "You can execute arbitrary commands now, e.g.,"
|
|
echo
|
|
echo " ietf/manage.py runserver 8001"
|
|
echo
|
|
echo "to start a development instance of the Datatracker."
|
|
echo
|
|
echo " ietf/manage.py test --settings=settings_test"
|
|
echo
|
|
echo "to run all the python tests."
|
|
echo
|
|
zsh
|
|
else
|
|
echo "Executing \"$*\" and stopping container."
|
|
echo
|
|
zsh -c "$*"
|
|
CODE=$?
|
|
fi
|
|
sudo service rsyslog stop
|
|
exit $CODE
|
|
fi
|