This is a record of the activities issues dealt with and actions taken for the DLF website in early 2009. This list reflects the outcome of a priority-setting call between Barrie and Eric on 090507. We have decided to drop the preparation of the new DLF site for rollout due to the DLF's re-merger with CLIR. Instead we will spend May and June focussing on a simple transfer of the existing DLF site from New Target to DreamHost. I've left this page with all the old site cleanup items as they were so that Barrie, DLF, and CLIR have a record of the earlier activity.

Issues described below include: Portability   Polish   Backup   Policy   Search   Performance   Forums   Misc  

The only thing we are really working on now: Transfer  

Feel free to leave comments below.

Testing Portability <span class=todosurround onmouseover="tooltip.show('

There a number of good reasons to test the portability of the Drupal installation created by eCore.

• Moving the whole filesystem and db to an alternate host ensures that we know exactly what the critical parts are. This would expose, for examples, dependencies accidently left in the code on internal eCore resources.

• This gives us practice moving the site to a new host, just in case problems emerge with Dreamhost.

• This gives me a second "test" installation on which I can practice code and design fixes before transferring them to the live system.

• This helps us understand what a backup recovery process may be like.

I've decided to put a full copy of the DLF site up on my own FastDomain account. FastDomain is organized in ways that are significantly different from Dreamhost (for example, they use Control Panel to manage accounts, something many web hosts do and something that Dreamhost does not).

Getting test Drupal up at FastDomain <span class=todosurround onmouseover="tooltip.show('

(090103-05) Spent a few days getting FD drupal to work. Turns out that FD fantastico and simplescript installs of drupal both had flawed php.ini settings. Drupal needed more memory to run, and the "memory_max" line in the php.ini file had to be commented out to allow for enough headroom.

Getting site moved over <span class=todosurround onmouseover="tooltip.show('

(090105-06) After some testing with copying drupal installs from one domain to another within FD, finally started work on the transfer of DLF site. Steps...

1. created a subdomain (http://dlf.tenseg.net) on FD
2. created a standard drupal install on FD for that subdomain (this step could probably be replaced by the simple copying later, but then the db would have to be created via cp instead, not too big a deal)
3. put myself on the DH whitelist for remote mysql access
4. used Sequel Pro to log into DH db and do a full sql export with table drops
5. used Sequel Pro to log into FD db and import the sql (it crashed on first attempt, worked fine second time around)
6. used RBrowser (SFTP) to move drupal filesystem from DH to FD (this could probably be done with a tarball and curl instead)
7. edited files on FD to log into new FD db
   • /sites/default/settings.php
   • /sites/all/themes/DLF/node-tpage.php (not sure what this file does, but this move did reveal that eCore had left this file pointing at an internal db instead of at DH on the DH install, notified eCore of this error)
8. visited the new site to test

Moving to http://new.diglib.org

(090220) Used the procedure above to move the site to new digs.

1. created a subdomain (http://new.diglib.org) on DH
2. added the DH IP address to Network Solutions DNS for diglib.org
3. copied diglib.dreamhosters.com Drupal install to new.diglib.org directory
4. I was already on the DH whitelist for remote mysql access
5. used Sequel Pro to log into DH d.d.c db and do a full sql export
6. used Sequel Pro to log into DH n.d.o db and import the sql
7. edited file on n.d.o to log into new n.d.o db
   • /sites/default/settings.php db login
8. visited the new site to test

Design Polish <span class=todosurround onmouseover="tooltip.show('

There were a number of aspects of the site design that were acceptable but not quite as we'd like them when the site was finished by eCore.

Navigation text <span class=todosurround onmouseover="tooltip.show('

The navigation text was done as GIF images instead of as text. This results in a few issues: resizing text does not work well, screen readers may not respond well, the font used does not match the font of other key elements (see a particularly jarring example of this when you enter a particular workspace and look at the "section" label below the DLF logo), nav text cannot be changed without resorting to graphics work (particularly restrictive for the secondary nav).

(090105) Identified the font eCore had used as Myriad Pro. Since that font is not very common, decided to recreate elements in the nearest common font, Trebuchet MS.

(090107) Lots of CSS work (style.css) mixed with a bit of HTML/PHP (header.php) to get the top nav and primary nav to an acceptable point. One remaining issue is that we can't hold both the right and left margins to a defined point using just text and CSS, so we may have to settle for right justification (possibly a better solution for secondary nav anyway). Another issue is that at extremely large sizes some nav elements disappear below the header (too bad I can't specify a max size).

(090108) Got secondary nav working with plain text. See style.css and secondarynav.php. The big compromise here is that secondary nav is now right justified. I like it there more, but this is open for discussion.

(090108) Added identifying comments to top of each template file. Finding these files quite poorly written, I would not be proud of this work. It is scattershot, hard to follow, poorly thought out. Note the recurring inclusion of sect.php as an example. I've seen pages where this file is included three or four times!

(090108-09) Working on putting headline graphics into Trebuchet. This is proving to be quite a pain since the templates are all over the place in terms of how they present these headlines. Added "h1.bigred" to style.css and then changed files to use it.

• node-spage.tpl.php seems to present core content of many pages, including About and Membership. That's converted.

• node-flist.tpl.php handles the lists of forums (upcomming and past). This was simplified by implementing text headline. I think the current forum should be included in any "upcoming" list, though. I wonder what the levels are supposed to indicate (we've got 1, 2, and 3).

• node-fpage.tpl.php handles the presentation of a single forum. A big change here, I made the forum date/city the headline and got rid of upcoming and past designations here (seems redundant, given the dates!).

• left-fpage.php, left-flist.php, and node-flist.tpl.php all present the forums in lists. These did not include the current forum as an upcoming forum (an oversight I've fixed) and did not sort the forums in any way. I've now sorted the upcoming in date order and the past in reverse date order.

(090109) Section text (that stuff right under the DLF logo) was revised to be in Trebuchet. Also changed sectionimg.txt so that it would not use the word "Workspaces" inside each individual workspace. Note: growing the text very large will make portions of the section info (title and breadcrumbs) disappear.

(090109) Beginning to lose track of all the places that GIF text is being replaced... spent almost all day doing this and some associated code fixing. The code is in really poor shape, for something professional anyway. Found another bug in tagcloud-workspace.php that prevented the tagcloud box from being drawn. Uncovering this required rewriting some code so that it is readable (indents, spacing, why are these things not done?).

(090109) Revised admin theme to Garland with new colors to make it more like the rest of DLF site.

(090112) Worked on individual workspaces. There are significant functional issues now covered in a separate section of this document. In process of testing individual workspaces, I realized that there was no way to add members without their having to take action. Contacted Beth and arranged a fix for this (og/add_user).

(090112-13) Worked on the front page (page-front.tpl.php) and its assorted headlines.

(090113) Redesigned the way avatar images are shown (class="avatar") so that the border is done in CSS instead of image itself. Note! this will require re-uploading existing avatars for best effect.

(090113) Put $head_title into the title on all the page templates (except page-front.tpl.php). Makes for much nicer window titles.

To do:

• Note! these changes have been made at http://dlf.tenseg.net and have yet to be moved over to http://diglib.dreamhosters.com <span class=todosurround onmouseover="tooltip.show('

Things we won't be doing now:

• We should consider a complete rewrite of the basic structure of the left column and main text. It is very oddly mixed right now and different on home page (broken when you narrow the browser too much) and all other pages. But this would be a big task, one I won't bite off right now, if ever.

Bookmark/sidebar <span class=todosurround onmouseover="tooltip.show('

There were a few problems with the bookmark. The text began too far down the page, leaving a large gap that tended to destroy the sense of verticality of the bookmark. The admin menu at the top also contributed to this when an administrator used the system. The admin menu at the top also made it hard for an admin to visualize that a regular user would see on the site.

(090104) Tweeked responsible HTML/PHP pages (page-node.tpl.php, page-node-add.tpl.php, page-node-38.tpl.php) so that admin menu went to the bottom of the page. Added a style for the admin menu to the CSS (style.css).

(090114) Minor pixel nudges on the home page bookmark. This should be re-architected to match all the other pages.

(090114) DLF Announce rearranged and pointed to listserv, RSS logo still points to RSS feed. Note! that all RSS links are broken on the tenseg site, make sure they continue to work on DH after the move.

To do:

• the breadcrumbs really don't look right in the bookmark. Maybe they could go elsewhere? Maybe they need a background? I really don't have a good idea for this. Suggestions? Maybe abandon breadcrumbs except in the workspaces? <span class=todosurround onmouseover="tooltip.show('

Things we won't be doing now:

• We should re-architect the bookmark on the front page so that it matches the other pages. Not a current priority. (note user fix request below)

Workspaces <span class=todosurround onmouseover="tooltip.show('

Individual workspaces are still quite messy. A few fixes should be made to both navigation and functionality to make these a bit easier on users.

To do:

• create a workspace nav block in the sidebar (similar to the admin block) which includes subscription link, adding members, and unsubscribing <span class=todosurround onmouseover="tooltip.show('

Backups <span class=todosurround onmouseover="tooltip.show('

We need to figure out how the system will be backed up and what our restore procedure would be. I will contact Dreamhost to learn more about our options there.

(090113) Inquiry sent to DH via web panel support generated the following response:

We actually have two levels of backup Eric. The first level is our hardware-based RAID10 (or RAID1+0). In the event of a hardware failure, your data is stored on a fully redundant, with parity raid array for the greatest chance of recovery. However, on top of this, we offer restoration backups. These backups are NOT guaranteed and are considered a luxury, however a backup snapshot is made every 1 and 2 hours/days/weeks. As such, you can make a restoration request of your data by going to "domains->manage domains" and clicking "restore" for the domain. You can also restore your database by going to "goodies->manage mysql" and clicking "restore db".

Wow. This is actually pretty impressive! It certainly takes care of the self inflicted wound restores. Very nice.

Still, it looks like we should probably make at least a weekly offsite backup of both the db and the filesystem. DH makes a big deal of noting that their own backups are not guaranteed. I will have to consider how to accomplish this. Creating the backups should not be all that hard, but we will have to decide where to put them. Maybe we should use their Files Forever feature for this?

(090113) Began to consider https://www4.ibackup.com/online-backup-linux/.

(090311) Put /home/ecore4dfl/bin/backup.sh into place and added a daily cron job to run it. This generates a backup file around midnight each day that can then be found at http://diglib.dreamhosters.com/efc/get/. A one day older copy of the backup is kept at /home/ecore4dfl/backups. Note that this backup is technically available out in the open to facilitate easy pickup. That is a security risk, since it contains a full copy of the Drupal database and filesystem. A better solution for moving it offsite would be to push it there via scp or some other transfer. For that, though, we need a second server to which we could push it. I use FastDomain for this. Should DLF to the same or use ibackup above?

Offsite backup <span class=todosurround onmouseover="tooltip.show('

Brian would like a tarball of the old.diglib.org site that could be moved to the local CLIR server every now and then. I will write a script to create one each night and put it in a place CLIR can pick up form whenever they like.

(090613) Modified backup script created in March so that it makes a tarball of old.diglib.org instead of new.diglib.org. This backup is moved to http://diglib.dreamhosters.com/efc/get/diglib.tar.gz for pickup. CLIR staff will have to arrange to pick it up every now and then.

Privacy and copyright policy statements <span class=todosurround onmouseover="tooltip.show('

Add links to our Creative Commons license and to a privacy statement drafted by Peter and I to the footer of every page.

(090108) Barrie provided a link to the proper CC license and this was added to the Site configuration Δ Site information in drupal admin.

(090109) Why this is only affecting the home page is still a mystery. Solved. The footer.php file had hardcoded text instead of php calls to drupal for content. Fixed.

(090114) Added current policy statement to both FD and DH versions of the site. Display is still a bit odd on DH, but will resolve when we move over design changes.

• Note! some of these changes (the footer) have been made at http://dlf.tenseg.net and have yet to be moved over to http://diglib.dreamhosters.com <span class=todosurround onmouseover="tooltip.show('

Google search <span class=todosurround onmouseover="tooltip.show('

The drupal search invoked by the "Search DLF" box in the header is OK, but less than satisfactory in some ways. The major flaw is that it does not search PDF and other non-drupal-db content.

We should be able to implement a Google search of this site easily enough, but that may present some other problems. For example, how would we search inside limited-access workspaces with Google. (For that matter, how does that work with the existing drupal search?)

Performance <span class=todosurround onmouseover="tooltip.show('

I've noted some performance hiccups with the site, for example, my first load of the day when using the site is often very slow (on the order of 12 seconds to draw the page). Is this just a Drupal thing? Is this Dreamhost? How much of a noticeable problem is this for the run-of-the-mill user?

Forums (:todo 0903 abandoned 0905::)

We need to make sure that the call for Forum papers and Forum registration can be used. The call should be a relatively simple transition, but the registration is a bit more complicated.

Call for papers <span class=todosurround onmouseover="tooltip.show('

Right now the call for papers is done in Surveymonkey. A simple web form that generates email would probably be sufficient here. Of course, the real win and our goal is to make submitting a paper proposal the first step in creating a record (node) on drupal that eventually becomes the home of the forum presentation itself.

(090330) This would require a new node type, though the "presentation" type may well do the trick. I know Jaime was working on modifying this type and want to talk with her about it.

Forum registration <span class=todosurround onmouseover="tooltip.show('

Forum registration is currently done at reg.com (?) and it is not clear that the functionality of that service can be well done on the DLF site. One big benefit of moving registration to drupal, though, is that we can tie it into registration for the site itself, so that if you are already registered for the site, then your contact info is present in the Forum registration, and if you are registering for the forum, you also get an account on the site. Dreaming, I know, but if we can make that happen then a switch to drupal may be worth it.

(090320) Learned about the Signup module for Drupal and began to investigate. Determined that the module is not yet stable enough for our production system since the 6.x compatible release is still in release candidate stage with significant bugs still being addressed. Since the current model works for DLF and does not add significant costs, I recommend leaving forum registration as it is for the time being. If and when we want to bring this onto Drupal, I recommend implementing the Signup module, but I would wait at least another six months before doing so.

Clean up Forum URLs (:todo 0904 abandoned 0905::)

The various forums and forum presentations have a slew of shortcut URLs that are not at all uniform in their structure. I need to talk with Jaime about these and suggest cleaning them up. I could devise a consistent format for these and fix those that already exist. See http://new.diglib.org/admin/build/path for a glimpse of the problem.

Miscellaneous

Fixes suggested by users

(090114) (per Kathryn) Cleaned up partial OpenID icon below the OpenID link in the logon page. No idea why this was occurring, but fixed it with a CSS hack. See ".openid-link a" in CSS, which sets the background explicitly to white. <span class=todosurround onmouseover="tooltip.show('

(090309) (per Chuck) Installed Ubuntu to learn how to make fonts work. See http://ubuntu.wordpress.com/2005/09/09/installing-microsoft-fonts/. Default Linux fonts force a word wrap on nav bar. <span class=todosurround onmouseover="tooltip.show('

(090309) (per Chuck) Front page bookmark shuffle on narrow window size (noted above as well). Is this a priority? (:todo 0904 abandoned 0905::)

(090309) (per Kevin) Fix UMich link: "On http://new.diglib.org/node/18, the link to the University of Michigan is broken. (It was on the old site too.) It should go to http://www.lib.umich.edu/lit/." <span class=todosurround onmouseover="tooltip.show('

(090309) (per Kevin) Added javascript to handle dimming and removing of "Search DLF". In the process I unified the creation of the HTML

(090309) (per Kevin) (per David) Confusion of Members and Membership. We have to pick a course of action. (:todo 0904 abandoned 0905::)

(090311) (per Cilla) Have link color change or link in other way highlight on mouseover. We should consider this. (:todo 0904 abandoned 0905::)

(090317) (per Andrew) Could we link directly to documents instead of through the "attachment" links Drupal provides? (:todo 0904 abandoned 0905::)This may not be possible, but I will investigate.(:todoend:)

Other Fixes

Drupal can be made very convoluted and eCore seems to have done that. Maybe this is due to what we requested forcing their hand, but I just don't know enough about Drupal to know for sure. Some things would be nice to clean up.

• Why are many of our pages "spage" instead of "page" type? Is there any easy way to create a new spage? (Right now I add page and then switch it over to spage.) Was this done just to accommodate the red titles being images? Or was this needed to get left content accessible as well?

• Many of the drupal forms are way too burdensome visually. This is particularly true of simple page editing, which gives you only a tiny area to edit content an way too much space for everything else you don't need. Is there any way to fix this? Start with workspaces. (:todo 0903 abandoned 0905::)

  • (090319) Conferred with some Drupal gurus and learned that the problem with Drupal forms is pretty deep-seated. I don't think we will be able to fix this.

• The whole sect.php/secondarynav.php thing is a mess and requires way too much maintenance for real people. There must be some way to use the drupal menu scheme to handle this secondary nav automatically. What would it be?

  • (090228) Removed sect.php calls from secondarynav.php and sectionimg.php since it gets called from header.php anyway. Why call it more than once?

• We should use a second Drupal install to test new functionality and upgrades. Set up dev.diglib.org on 090220, maybe use that?

Coexistence with old site content <span class=todosurround onmouseover="tooltip.show('

We need to find a way to allow old URLs to work even though the old site content will be in a subdirectory on the new site. Likely strategy: add some PHP to 404 error so that it checks old content before delivering a 404.

• Investigate how Drupal handles 404.

  • (090220) Checked out http://tedserbinski.com/tags/drupal/preventing-drupal-handling-404s-performance which has some promising hints.

• Develop a script to redirect. Do we want/need some indication that this content is now considered "old" or "archival"?

• Move all old content from old host to new.

Transfer

In April and May we (Chuck, Barrie, Brian, Kathlin, and EFC) decided to abandon the new Drupal DLF site in order to give DLF and CLIR time for a smooth merger and the new program officer some control over how DLF presents itself in the future. Though abandoned for now, the Drupal site will be left on DreamHost and may be resurrected as part of a new DLF presence.

Meanwhile, we will instead focus on moving the old site from New Target to DreamHost.

Create DLF Search CSE at Google <span class=todosurround onmouseover="tooltip.show('

(090507) Created a new CSE at Google for diglib search since we will need this for the transferred old site as well. See http://www.google.com/coop/cse?cx=007056022632613166272:cpqkgjjfers

Move DLF content from New Target <span class=todosurround onmouseover="tooltip.show('

Barrie plans to freeze the DLF site on 5/15. We should have a snapshot of the site no later than 6/1. That snapshot must be transferred to DreamHost in June and come up as the live site no later than 6/30. New Target will stop serving the site on 7/1.

(090507) Contacted Wendy at UMN about using UMN to stage uploads to DreamHost.