Writeup of CDN-related changes.
- Legacy-Id: 9944
This commit is contained in:
Henrik Levkowetz
parent
71cfe9459f
commit
8aeca50394
222
README-CDN.rst
Normal file
222
README-CDN.rst
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
================================================================================
|
||||
Serving Static Datatracker Files via a CDN
|
||||
================================================================================
|
||||
|
||||
Intro
|
||||
=====
|
||||
|
||||
With release 6.4.0, the way that the static files used by the datatracker are
|
||||
handled changes substantially. Static files were previously versioned under a
|
||||
top-level ``static/`` directory, but this is not the case any more. External
|
||||
files (such as for instance ``jquery.min.js``) are now placed under
|
||||
``ietf/externals/static/`` and updated using a tool called bower_, while
|
||||
datatracker-specific files (images, css, js, etc.) are located under
|
||||
``ietf/static/ietf/`` and ``ietf/secr/static/secr/`` respectively.
|
||||
|
||||
The following sections provide more details about handling of internals,
|
||||
externals, and how deployment is done.
|
||||
|
||||
|
||||
Serving Static Files via CDN
|
||||
============================
|
||||
|
||||
If resources served over a CDN and/or with a high max-age don't have different
|
||||
URLs for different versions, then any component upgrade which is accompanied
|
||||
by a change in template functionality will have a long transition time
|
||||
during which the new pages are served with old components, with possible
|
||||
breakage. We want to avoid this.
|
||||
|
||||
The intention is that after a release has been checked out, but before it is
|
||||
deployed, the standard django 'collectstatic' management command will be
|
||||
run, resulting in all static files being collected from their working
|
||||
directory location and placed in an appropiate location for serving via CDN.
|
||||
This location will have the datatracker release version as part of its URL,
|
||||
so that after the deployment of a new release, the CDN will be forced to fetch
|
||||
the appropriate static files for that release.
|
||||
|
||||
An important part of this is to set up the ``STATIC_ROOT`` and ``STATIC_URL``
|
||||
settings appropriately. In 6.4.0, the setting is as follows in production
|
||||
mode::
|
||||
|
||||
STATIC_URL = "https://www.ietf.org/lib/dt/%s/"%__version__
|
||||
STATIC_ROOT = CDN_ROOT + "/a/www/www6s/lib/dt/%s/"%__version__
|
||||
|
||||
The result is that all static files collected via the ``collectstatic``
|
||||
command will be placed in a location served via CDN, with the release
|
||||
version being part of the URL.
|
||||
|
||||
In development mode, ``STATIC_URL`` is set to ``/static/``, and Django's
|
||||
``staticfiles`` infrastructure makes the static files available under that
|
||||
local URL root.
|
||||
|
||||
|
||||
Handling of External Javascript and CSS Components
|
||||
==================================================
|
||||
|
||||
In order to make it easy to keep track of and upgrade external components,
|
||||
these are now handled by a tool called ``bower``, via a new management
|
||||
command ``bower_install``. Each external component is listed in a file
|
||||
``ietf/bower.json``. In order to update the version of a component listed in
|
||||
``ietf/bower.json``, or add a new one, you should edit ``bower.json``, and
|
||||
then run the management command::
|
||||
|
||||
$ ietf/manage.py bower_install
|
||||
|
||||
(Not surprisingly, you need to have bower_ installed in order to use this
|
||||
management command.)
|
||||
|
||||
That command will fetch the required version of each external component listed
|
||||
in ``bower.json`` (actually, it will do this for *all* ``bower.json`` files
|
||||
found in the ``static/`` directories of all ``INSTALLED_APPS`` and the
|
||||
directories in ``settings.STATICFILES_DIRS``), saving them temporarily under
|
||||
``.tmp/bower_components/``; it will then extract the relevant production
|
||||
``js`` and ``css`` files and place them in an appropriately named directory
|
||||
under ``ietf/externals/static/``. The latter location is taken from
|
||||
``COMPONENT_ROOT`` in ``settings.py``.
|
||||
|
||||
Managing external components via bower has the additional benefit of
|
||||
managing dependencies -- components that have dependencies will pull in
|
||||
these, so that they also are placed under ``ietf/externals/static/``.
|
||||
You still have to manually add the necessary stylesheet and/or javascript
|
||||
references to your templates, though.
|
||||
|
||||
The ``bower_install`` command is not run automatically by ``bin/mkrelease``,
|
||||
since it needs an updated ``bower.json`` in order to do anything interesting.
|
||||
So when you're intending to update an external web asset to a newer version,
|
||||
you need to edit the ``bower.json`` file, run ``manage.py bower_install``,
|
||||
verify that the new version doesn't break things, and then commit the new
|
||||
files under ``ietf/externals/static/`` and the updated ``bower.json``.
|
||||
|
||||
.. _bower: http://bower.io/
|
||||
|
||||
The ``ietf/externals/static/`` Directory
|
||||
-----------------------------------------
|
||||
|
||||
The directory ``ietf/externals/static/`` holds a number of subdirectories
|
||||
which hold distribution files for external client-side components, collected
|
||||
by ``bower_install`` as described above. Currently
|
||||
(01 Aug 2015) this means ``js`` and ``css`` components and fonts.
|
||||
|
||||
These components each reside in their own subdirectory, which is named with
|
||||
the component name::
|
||||
|
||||
henrik@zinfandel $ ls -l ietf/externals/static/
|
||||
total 40
|
||||
drwxr-xr-x 6 henrik henrik 4096 Jul 25 15:25 bootstrap
|
||||
drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 bootstrap-datepicker
|
||||
drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 font-awesome
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery.cookie
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptmono
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptsans
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptserif
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2
|
||||
drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2-bootstrap-css
|
||||
|
||||
The ``pt*`` fonts are an exception, in that there is no bower component
|
||||
available for these fonts, so they have been put in place manually.
|
||||
|
||||
|
||||
Handling of Internal Static Files
|
||||
=================================
|
||||
|
||||
Previous to this release, internal static files were located under
|
||||
``static/``, mixed together with the external components. They are now
|
||||
located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr``, and will be
|
||||
collected for serving via CDN by the ``collectstatic`` command. Any static
|
||||
files associated with a particular app will be handled the same way (which
|
||||
means that all ``admin/`` static files automatically will be handled correctly, too).
|
||||
|
||||
Handling of Customised Bootstrap Files
|
||||
======================================
|
||||
|
||||
We are using a customised version of Bootstrap_, which is handled specially,
|
||||
by a SVN externals definition in ``ietf/static/ietf``. That pulls the content
|
||||
of the ``bootstrap/dist/`` directory (which is generated by running ``grunt``
|
||||
in the ``bootstrap/`` directory) into ``ietf/static/ietf/bootstrap``, from
|
||||
where it is collected by ``collectstatic``.
|
||||
|
||||
Changes to Template Files
|
||||
=========================
|
||||
|
||||
In order to make the template files refer to the correct versioned CDN URL
|
||||
(as given by the STATIC_URL root) all references to static files in the
|
||||
templates have been updated to use the ``static`` template tag when referring
|
||||
to static files. This will automatically result in both serving static files
|
||||
from the right place in development mode, and referring to the correct
|
||||
versioned URL in production mode and the simpler ``/static/`` urls in
|
||||
development mode.
|
||||
|
||||
.. _bootstrap: http://getbootstrap.com/
|
||||
|
||||
Deployment
|
||||
==========
|
||||
|
||||
During deployment, it is now necessary to run the management command::
|
||||
|
||||
$ ietf/manage.py collectstatic
|
||||
|
||||
before activating a new release. The deployment README file has been updated
|
||||
accordingly, and now reads as follows::
|
||||
|
||||
In order to fetch a new release of the django datatracker code, simply
|
||||
check out the appropriate tag from svn:
|
||||
|
||||
svn co http://svn.tools.ietf.org/svn/tools/ietfdb/tags/$releasenumber
|
||||
|
||||
Don't forget to copy $releasenumber/ietf/settings_local.py from the
|
||||
old release to the new one; otherwise it won't work!
|
||||
|
||||
cp $oldreleasenumber/ietf/settings_local.py $releasenumber/ietf/
|
||||
|
||||
Change into the new directory:
|
||||
|
||||
cd $releasenumber
|
||||
|
||||
Next, upgrade pip and install requirements:
|
||||
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
|
||||
Run migrations:
|
||||
|
||||
ietf/manage.py migrate
|
||||
|
||||
* NEW *
|
||||
Move static files to the appropriate directory for serving via CDN:
|
||||
|
||||
ietf/manage.py collectstatic
|
||||
|
||||
* NEW *
|
||||
Run some basic datatracker system checks:
|
||||
|
||||
ietf/manage.py check
|
||||
|
||||
Change back out to the parent directory:
|
||||
|
||||
cd ..
|
||||
|
||||
and then re-point the 'web' symlink:
|
||||
|
||||
rm ./web; ln -s $releasenumber web
|
||||
|
||||
and finally restart apache:
|
||||
|
||||
sudo /etc/init.d/apache2 restart
|
||||
|
||||
It's now also a good idea to go to the datatracker front page:
|
||||
|
||||
http://datatracker.ietf.org/
|
||||
|
||||
to check that it's alive and kicking, and displaying the new release
|
||||
number at the bottom of the left-side menubar :-) -- if not, revert the
|
||||
symlink step, re-pointing the symlink to the release that was running
|
||||
before the new release, and restart apache again to roll back to that.
|
||||
|
||||
Finally, make sure the datatracker rsyncs over to IETFB, and then run the
|
||||
PIP commands on ietfb as well:
|
||||
|
||||
ssh ietfb
|
||||
pip install --upgrade pip
|
||||
pip install -r requirements.txt
|
||||
|
||||
Loading…
Reference in a new issue