================================================================================ Serving Static Datatracker Files via a CDN ================================================================================ Intro ===== With release 6.4.0, the way that the static files used by the datatracker are handled changes substantially. Static files were previously versioned under a top-level ``static/`` directory, but this is not the case any more. External files (such as for instance ``jquery.min.js``) are now placed under ``ietf/externals/static/`` and updated using a tool called bower_, while datatracker-specific files (images, css, js, etc.) are located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr/`` respectively. The following sections provide more details about handling of internals, externals, and how deployment is done. Serving Static Files via CDN ============================ If resources served over a CDN and/or with a high max-age don't have different URLs for different versions, then any component upgrade which is accompanied by a change in template functionality will have a long transition time during which the new pages are served with old components, with possible breakage. We want to avoid this. The intention is that after a release has been checked out, but before it is deployed, the standard django 'collectstatic' management command will be run, resulting in all static files being collected from their working directory location and placed in an appropiate location for serving via CDN. This location will have the datatracker release version as part of its URL, so that after the deployment of a new release, the CDN will be forced to fetch the appropriate static files for that release. An important part of this is to set up the ``STATIC_ROOT`` and ``STATIC_URL`` settings appropriately. In 6.4.0, the setting is as follows in production mode:: STATIC_URL = "https://www.ietf.org/lib/dt/%s/"%__version__ STATIC_ROOT = CDN_ROOT + "/a/www/www6s/lib/dt/%s/"%__version__ The result is that all static files collected via the ``collectstatic`` command will be placed in a location served via CDN, with the release version being part of the URL. In development mode, ``STATIC_URL`` is set to ``/static/``, and Django's ``staticfiles`` infrastructure makes the static files available under that local URL root. Handling of External Javascript and CSS Components ================================================== In order to make it easy to keep track of and upgrade external components, these are now handled by a tool called ``bower``, via a new management command ``bower_install``. Each external component is listed in a file ``ietf/bower.json``. In order to update the version of a component listed in ``ietf/bower.json``, or add a new one, you should edit ``bower.json``, and then run the management command:: $ ietf/manage.py bower_install (Not surprisingly, you need to have bower_ installed in order to use this management command.) That command will fetch the required version of each external component listed in ``bower.json`` (actually, it will do this for *all* ``bower.json`` files found in the ``static/`` directories of all ``INSTALLED_APPS`` and the directories in ``settings.STATICFILES_DIRS``), saving them temporarily under ``.tmp/bower_components/``; it will then extract the relevant production ``js`` and ``css`` files and place them in an appropriately named directory under ``ietf/externals/static/``. The latter location is taken from ``COMPONENT_ROOT`` in ``settings.py``. Managing external components via bower has the additional benefit of managing dependencies -- components that have dependencies will pull in these, so that they also are placed under ``ietf/externals/static/``. You still have to manually add the necessary stylesheet and/or javascript references to your templates, though. The ``bower_install`` command is not run automatically by ``bin/mkrelease``, since it needs an updated ``bower.json`` in order to do anything interesting. So when you're intending to update an external web asset to a newer version, you need to edit the ``bower.json`` file, run ``manage.py bower_install``, verify that the new version doesn't break things, and then commit the new files under ``ietf/externals/static/`` and the updated ``bower.json``. .. _bower: http://bower.io/ The ``ietf/externals/static/`` Directory ----------------------------------------- The directory ``ietf/externals/static/`` holds a number of subdirectories which hold distribution files for external client-side components, collected by ``bower_install`` as described above. Currently (01 Aug 2015) this means ``js`` and ``css`` components and fonts. These components each reside in their own subdirectory, which is named with the component name:: henrik@zinfandel $ ls -l ietf/externals/static/ total 40 drwxr-xr-x 6 henrik henrik 4096 Jul 25 15:25 bootstrap drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 bootstrap-datepicker drwxr-xr-x 4 henrik henrik 4096 Jul 25 15:25 font-awesome drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 jquery.cookie drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptmono drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptsans drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:24 ptserif drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2 drwxr-xr-x 2 henrik henrik 4096 Jul 25 15:25 select2-bootstrap-css The ``pt*`` fonts are an exception, in that there is no bower component available for these fonts, so they have been put in place manually. Handling of Internal Static Files ================================= Previous to this release, internal static files were located under ``static/``, mixed together with the external components. They are now located under ``ietf/static/ietf/`` and ``ietf/secr/static/secr``, and will be collected for serving via CDN by the ``collectstatic`` command. Any static files associated with a particular app will be handled the same way (which means that all ``admin/`` static files automatically will be handled correctly, too). Handling of Customised Bootstrap Files ====================================== We are using a customised version of Bootstrap_, which is handled specially, by a SVN externals definition in ``ietf/static/ietf``. That pulls the content of the ``bootstrap/dist/`` directory (which is generated by running ``grunt`` in the ``bootstrap/`` directory) into ``ietf/static/ietf/bootstrap``, from where it is collected by ``collectstatic``. Changes to Template Files ========================= In order to make the template files refer to the correct versioned CDN URL (as given by the STATIC_URL root) all references to static files in the templates have been updated to use the ``static`` template tag when referring to static files. This will automatically result in both serving static files from the right place in development mode, and referring to the correct versioned URL in production mode and the simpler ``/static/`` urls in development mode. .. _bootstrap: http://getbootstrap.com/ Deployment ========== During deployment, it is now necessary to run the management command:: $ ietf/manage.py collectstatic before activating a new release. The deployment README file has been updated accordingly, and now reads as follows:: In order to fetch a new release of the django datatracker code, simply check out the appropriate tag from svn: svn co http://svn.tools.ietf.org/svn/tools/ietfdb/tags/$releasenumber Don't forget to copy $releasenumber/ietf/settings_local.py from the old release to the new one; otherwise it won't work! cp $oldreleasenumber/ietf/settings_local.py $releasenumber/ietf/ Change into the new directory: cd $releasenumber Next, upgrade pip and install requirements: pip install --upgrade pip pip install -r requirements.txt Run migrations: ietf/manage.py migrate * NEW * Move static files to the appropriate directory for serving via CDN: ietf/manage.py collectstatic * NEW * Run some basic datatracker system checks: ietf/manage.py check Change back out to the parent directory: cd .. and then re-point the 'web' symlink: rm ./web; ln -s $releasenumber web and finally restart apache: sudo /etc/init.d/apache2 restart It's now also a good idea to go to the datatracker front page: http://datatracker.ietf.org/ to check that it's alive and kicking, and displaying the new release number at the bottom of the left-side menubar :-) -- if not, revert the symlink step, re-pointing the symlink to the release that was running before the new release, and restart apache again to roll back to that. Finally, make sure the datatracker rsyncs over to IETFB, and then run the PIP commands on ietfb as well: ssh ietfb pip install --upgrade pip pip install -r requirements.txt