From 8aeca50394707c8ade19ce2c7e6f81f1549cefac Mon Sep 17 00:00:00 2001 From: Henrik Levkowetz Date: Sat, 1 Aug 2015 14:53:04 +0000 Subject: [PATCH] Writeup of CDN-related changes. - Legacy-Id: 9944 --- README-CDN.rst | 222 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 222 insertions(+) create mode 100644 README-CDN.rst diff --git a/README-CDN.rst b/README-CDN.rst new file mode 100644 index 000000000..33ecdb118 --- /dev/null +++ b/README-CDN.rst @@ -0,0 +1,222 @@ +================================================================================ + Serving Static Datatracker Files via a CDN +================================================================================ + +Intro +===== + +With release 6.4.0, the way that the static files used by the datatracker are +handled changes substantially. Static files were previously versioned under a +top-level ``static/`` directory, but this is not the case any more. External +files (such as for instance ``jquery.min.js``) are now placed under +``ietf/externals/static/`` and updated using a tool called bower_, while +datatracker-specific files (images, css, js, etc.) are located under +``ietf/static/ietf/`` and ``ietf/secr/static/secr/`` respectively. + +The following sections provide more details about handling of internals, +externals, and how deployment is done. + + +Serving Static Files via CDN +============================ + +If resources served over a CDN and/or with a high max-age don't have different +URLs for different versions, then any component upgrade which is accompanied +by a change in template functionality will have a long transition time +during which the new pages are served with old components, with possible +breakage. We want to avoid this. + +The intention is that after a release has been checked out, but before it is +deployed, the standard django 'collectstatic' management command will be +run, resulting in all static files being collected from their working +directory location and placed in an appropiate location for serving via CDN. +This location will have the datatracker release version as part of its URL, +so that after the deployment of a new release, the CDN will be forced to fetch +the appropriate static files for that release. + +An important part of this is to set up the ``STATIC_ROOT`` and ``STATIC_URL`` +settings appropriately. In 6.4.0, the setting is as follows in production +mode:: + + STATIC_URL = "https://www.ietf.org/lib/dt/%s/"%__version__ + STATIC_ROOT = CDN_ROOT + "/a/www/www6s/lib/dt/%s/"%__version__ + +The result is that all static files collected via the ``collectstatic`` +command will be placed in a location served via CDN, with the release +version being part of the URL. + +In development mode, ``STATIC_URL`` is set to ``/static/``, and Django's +``staticfiles`` infrastructure makes the static files available under that +local URL root. + + +Handling of External Javascript and CSS Components +================================================== + +In order to make it easy to keep track of and upgrade external components, +these are now handled by a tool called ``bower``, via a new management +command ``bower_install``. Each external component is listed in a file +``ietf/bower.json``. In order to update the version of a component listed in +``ietf/bower.json``, or add a new one, you should edit ``bower.json``, and +then run the management command:: + + $ ietf/manage.py bower_install + +(Not surprisingly, you need to have bower_ installed in order to use this +management command.) + +That command will fetch the required version of each external component listed +in ``bower.json`` (actually, it will do this for *all* ``bower.json`` files +found in the ``static/`` directories of all ``INSTALLED_APPS`` and the +directories in ``settings.STATICFILES_DIRS``), saving them temporarily under +``.tmp/bower_components/``; it will then extract the relevant production +``js`` and ``css`` files and place them in an appropriately named directory +under ``ietf/externals/static/``. The latter location is taken from +``COMPONENT_ROOT`` in ``settings.py``. + +Managing external components via bower has the additional benefit of +managing dependencies -- components that have dependencies will pull in +these, so that they also are placed under ``ietf/externals/static/``. +You still have to manually add the necessary stylesheet and/or javascript +references to your templates, though. + +The ``bower_install`` command is not run automatically by ``bin/mkrelease``, +since it needs an updated ``bower.json`` in order to do anything interesting. +So when you're intending to update an external web asset to a newer version, +you need to edit the ``bower.json`` file, run ``manage.py bower_install``, +verify that the new version doesn't break things, and then commit the new +files under ``ietf/externals/static/`` and the updated ``bower.json``. + +.. _bower: http://bower.io/ + +The ``ietf/externals/static/`` Directory +----------------------------------------- + +The directory ``ietf/externals/static/`` holds a number of subdirectories +which hold distribution files for external client-side components, collected +by ``bower_install`` as described above. Currently +(01 Aug 2015) this means ``js`` and ``css`` components and fonts. + +These components each reside in their own subdirectory, which is named with +the component name:: + + henrik@zinfandel $ ls -l ietf/externals/static/ + total 40 + drwxr-xr-x 6 henrik henrik 4096 Jul 25 15:25 bootstrap + drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 bootstrap-datepicker + drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 font-awesome + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery.cookie + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptmono + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptsans + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptserif + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2 + drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2-bootstrap-css + +The ``pt*`` fonts are an exception, in that there is no bower component +available for these fonts, so they have been put in place manually. + + +Handling of Internal Static Files +================================= + +Previous to this release, internal static files were located under +``static/``, mixed together with the external components. They are now +located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr``, and will be +collected for serving via CDN by the ``collectstatic`` command. Any static +files associated with a particular app will be handled the same way (which +means that all ``admin/`` static files automatically will be handled correctly, too). + +Handling of Customised Bootstrap Files +====================================== + +We are using a customised version of Bootstrap_, which is handled specially, +by a SVN externals definition in ``ietf/static/ietf``. That pulls the content +of the ``bootstrap/dist/`` directory (which is generated by running ``grunt`` +in the ``bootstrap/`` directory) into ``ietf/static/ietf/bootstrap``, from +where it is collected by ``collectstatic``. + +Changes to Template Files +========================= + +In order to make the template files refer to the correct versioned CDN URL +(as given by the STATIC_URL root) all references to static files in the +templates have been updated to use the ``static`` template tag when referring +to static files. This will automatically result in both serving static files +from the right place in development mode, and referring to the correct +versioned URL in production mode and the simpler ``/static/`` urls in +development mode. + +.. _bootstrap: http://getbootstrap.com/ + +Deployment +========== + +During deployment, it is now necessary to run the management command:: + + $ ietf/manage.py collectstatic + +before activating a new release. The deployment README file has been updated +accordingly, and now reads as follows:: + + In order to fetch a new release of the django datatracker code, simply + check out the appropriate tag from svn: + + svn co http://svn.tools.ietf.org/svn/tools/ietfdb/tags/$releasenumber + + Don't forget to copy $releasenumber/ietf/settings_local.py from the + old release to the new one; otherwise it won't work! + + cp $oldreleasenumber/ietf/settings_local.py $releasenumber/ietf/ + + Change into the new directory: + + cd $releasenumber + + Next, upgrade pip and install requirements: + + pip install --upgrade pip + pip install -r requirements.txt + + Run migrations: + + ietf/manage.py migrate + + * NEW * + Move static files to the appropriate directory for serving via CDN: + + ietf/manage.py collectstatic + + * NEW * + Run some basic datatracker system checks: + + ietf/manage.py check + + Change back out to the parent directory: + + cd .. + + and then re-point the 'web' symlink: + + rm ./web; ln -s $releasenumber web + + and finally restart apache: + + sudo /etc/init.d/apache2 restart + + It's now also a good idea to go to the datatracker front page: + + http://datatracker.ietf.org/ + + to check that it's alive and kicking, and displaying the new release + number at the bottom of the left-side menubar :-) -- if not, revert the + symlink step, re-pointing the symlink to the release that was running + before the new release, and restart apache again to roll back to that. + + Finally, make sure the datatracker rsyncs over to IETFB, and then run the + PIP commands on ietfb as well: + + ssh ietfb + pip install --upgrade pip + pip install -r requirements.txt +