maps

affinity bridge

more open data! how to make maps beautiful. (neat)

dude is reading his f’ng slides. boo. tuning out a bit. this is the part I know: what is open data, why is it awesome. localized, yay! ah, cross-mention w/ckan & datadotgc.ca project. disadvantages: privacy concerns, cost. (random thought: he looks like the kid from Arrested Development, esp with the askew tie. [updated: he was in the big dinner gang on Sunday, sat across from me. Really nice guy.])

map in 10 minutes: drinking fountains data, csv file. create content type: location, add fields: lat/long. choosing just a text field for simplicity. wouldn’t want to do with a large database, tho. set up gmap, api, bigger size, autozoom. using node import to bring in csv file. create view. interesting: using gmap view w/out using location module. 7 minutes flat. snazzy. (oh, this is the one mentioned in this morning’s presentation)

in 2 minutes: with install profile!

storage: cck, geo module, location (can be heavy), geotaxonomy – adds fields to a taxonomy term: tag content with a city and then pull lat/long from elsewhere, openlayers wkt. (“well-known text”?)

display: google maps, openlayers. (have wanted to play with openlayers. I wonder if it would be usable w/out security issue on branches page)

overlays/markers: heatmaps for non-point data?

tilesets: google, bing, mapbox, open street map. or generate your own! holy mackerel. big list. quantum gis – widely used open source mapping tool. mapnik. cascadenik a bit like css for ??? – missed something here.

ah, got distracted.

sleepy. dammit.

now I want data to play with! data.gov to try thing? (start w/http://www.data.gov/raw/1517?) github.com/tylor/quickmaps

got distracted looking at data. 🙂

oh, hey: http://data.wa.gov/

geocode module for taking db of address and translating to lat/long.

shape files are input to quantum gis. hard to interact w/shape file polygons? not 100% sure I understood that. polygons are a heavy server load?

server optimization

“because I have crashed servers that weren’t optimized” indeed.

suggestion from Adam (at lunch) to look at the actual theme(s) that were used with themekey, see if those have any squirrelly sql-type stuff in them. (also, I wonder about the recent memory upgrade. maybe I should just *try* it and see what happens.)

shared hosting (ie this site), virtual private server (no personal experience), dedicated server (ie, work)

apache: RAM hog. 20mb+ usage? list of items: keepalivetimeout – time waiting to kill process. 1-3 seconds for drupal. maxclients – simultaneous connections. maxrequestsperchild – don’t want to force unnec restarts. no caching for authenticated users! big issue for an intranet. advanced caching module? static & dynamic content on different apache servers. intriguing. image sprites may be more useful than zlib. ramdisk for most commonly used files. memcached daemon. loadmodule.conf file, remove unused (apache) modules.

mysql – pick good option file. my-huge.conf. log slow queries. (set for abt 5 secs) need to fix all to get improved performanced. query caching, in config file for mysql, 124mb (?). innodb tables, supposed to be good for writing a lot.

php – php cache option, accelerators (like Zend, APC) apc apparently to become standard w/php6. 2 confi options in php.ini: max_execution_time (30 secs, but test!), memory_limit, usually turns to 124/148mb.

load testing. ab = apache benchmark. ab -n 50 -c 2 http://example.com gives a command-line report.

performance monitoring – uptime, free -m. if all memory is in swap, you don’t have enough ram.

top -d: how to see what processes are using how much.

vmstat: input/output speed.

wtf is this app, for performance monitoring? nagios? yes: http://www.nagios.org/

based on a question: nginex – for the static stuff, then apache only does the dynamic. or I think I hear someone saying they use it for the whole thing instead of apache.

what’s causing overload, apachetop or top to see what process. apachetop sounds really useful. also a mysqltop. cacti.

better to use performance-related modules or do stuff on the server? getting a balance between overcaching & undercaching. don’t want to double-compress.

reduce css calls. I’ve got to dig back into that.

gtmetrics (gpmetrics?)

far-future expiration headers. yslow was really helpful, and almost all front-end.

pressflow – install profile? fork. http://pressflow.org/

d7 overview

came in late.

performance improvements.

not really taking great notes, but some interesting stuff nonetheless.

Xmas-time release? http://drupal7releasedate.com/

sprint tonite at century plaza hotel lobby.

download, install, test, report bugs. (could I do that w/my pet project? well, there’s a feeds alpha.)

I should make a spreadsheet like the one I did when I was looking towards moving work site to Drupal, waiting for D6 as an option, since it was SO much better.

contextual links module? [update: oh, a lot like the links in the zen theme. but nicer looking.]

documentation patches that need work. o hey. 🙂 “novice” tag.

shouldn’t be any big changes, esp between D7-beta & final.

naming standards for CSS/class names?

they need more themers. ah, that too is something I could look at. (again, the pet project.) theming guide is a critical issue.

sounds like multigroup is solid, altho no gui? would be curious to look into that more, because multigroup is a critical aspect of our site.

open gov data

ok, let’s see how long the battery lasts

not enough milk in my tea, and I steeped too long.

philosophy of open data.

okfn.org – open knowledge foundation.

comprehensive knowledge archive network (ckan)

microsoft open govt data initiative (ogdi), incl a tool

and…drupal!

cataloging system, open source, written in python. json-based api. ckan.net

(thinking dorothea would be all over this…or who knows: could give her own version?)

dead battery was dead. I have some paper notes, which hopefully I’ll be able to transcribe later.

paper notes:

showing ckan app, anyone can add data (oly police report, for example?)

URL munging gets JSON data.

ckan & ogdi wouldn’t play together -> drupal in the middle, using ckan api – used drupal because open source, modules, theming, also: tool they knew.

practicalweb.co.uk, made tool to connect ckan & drupal

ogdi: also repository (am kinda confused, could use a graphic)

fun w/small resolutions

I want more detail!

datadotgc.ca

ckan module, caching [I seem to remember some really interesting things in the use of caching]

datatables.net (jquery)

data isn’t in drupal db at all, except as cache. and except ministry list.

themes used instead of views. curious.

I’m still really confused.

oh, if the ckan item also exists as ogdi, then use ogdi for data.

interested data use example: stumble safely, vantrash, water (tyler sherman)

—-

follow-up thoughts: interesting stuff presented in a way that didn’t click very well for me. I think it would have benefited from fewer screenshots and more conceptual diagrams/charts.

keynote – Angela Byron

waking up, sorta.

saving just because.

wishing I’d spent some more time looking up what to do around here.

webchick! (ie, Angela Byron)

(would it be totally dorky to have her sign the book?)

community makes the project. community > technology. (which fits with one of my pet theories.) just shy of a million users on drupal.org?

OMG super qte cat figs!

Do-ocracy – illustrated with the Rosie the Riveter graphic. no one being paid to care about core, or most modules. wtf is that flow chart?

she was a serious f’ng geek – debian (?) linux on 7 floppy disks? “was that guy” re evangelizing free software, starting in 1995 – but didn’t contribute to anything until 2005.

doesn’t want that to be anybody else’s story.

1. see something & say “that’s dumb”

2. I want to see that fixed

3. can do something about it. – but NOT just code! even includes a well-written bug support. “rate of swear words per minutes” (as a usability measure?!)

don’t have to be einstein, lots of “everyone else” – wisdom of crowds.

gina the genius story. but NO. edwina the end user, filing a bug report. (well, in that case I guess I HAVE contributed) and paula the programmer, tatiana the tester, saying WTF? and iterative cycle. Wendy stuck on Windows XP. 🙂 “also, mind your spelling” and so on and so forth.

what is contribution? ginormous list! (issue queue farming?)

the myth of “they” – “there is no cabal” (ala MeFi!)

pie chart that’s basically just one thing: downloaded software. 0.05% registered AND did something with their account.

“you and your fancy ways”

answer 1 support request a day. 12000+ people who can’t figure out how to install drupal!

make progress on one issue per day. (views or another module you know well) mark duplicate, answer support request, etc.

when you learn something new, document it as you go. (which sometimes I do for myself!) ah, the curse of knowledge!

“I’m going to tell that you’re using WP” (guy next to me) 🙂

kitten!

secret to drupal success: being part of the 0.05%. gives others more incentive to help you, not “newb” but time management thing. learn faster, and save time/money. (more of the networking thing) get more business, attract better people. (WIIFM) keep yr finger on the pulse of drupal, and gives you stronger voice in project.

/contribute – places to jump in. but she prefers /community-initiatives. highlights things that are important.

seek out doers in the areas you’re interested in.

don’t wait for approval.

use the issue queue. (I think I’ve mentioned on twitter that I’m amused by having a bookmark labeled “my issues” – oh, hey, that feeds issue is fixed, isn’t it?)

irc. the people side of things. if it’s got to do w/actual issue, make sure it gets back in the queue!

oh, there’s another irc #drupal-pnw.

trade caring about other people’s patches. 🙂

awesome asshat graphic. a lot about giving people slack and providing guidance. tells the perfectionist pete vs sloppy sam story. which tho I’ve heard it before, but it’s worth repeating to myself, as I have a tendency to go off into a corner by myself.

(saving out of nervousness)

so nobody likes cvs. migrating to git. (I’ve been trying to figure out how the hell revision control works for years now.) “jangly things” multiple copies of the repository, easier to rename files. “break things w/out actually breaking things” “I keep saying I’m going to fix forums, and it never happens” exportable configurations. “butler” refactor something (some jerk w/phone talked over part of that)

I’m pretty excited now, actually.

5 things excited about in D7: image handling in core; usability improvements; automated testing framework; cck in core – fields can be added to anything, not just nodes; under-the-hood improvements for themers (actually, that sounds DELIGHTFUL). “it’s all great!”

Drupal upgrading tip

In my experience so far, the most annoying part of upgrading Core is disabling, then re-enabling, all the contrib modules. I’ve timed it more than once, and it’s accounted for more than half of the upgrade time. (Another huge chunk is waiting for files to upload/download/copy. Not much I can do about that. Update: what I can do about that is do all the backup and local unzipping (!) ahead of time.)

I know some people upgrade without disabling, but I’m WAY too paranoid. Apparently the Contrib Toggle module is dangerous/broken. And I have yet to figure out Drush.

On the other hand, I do know my way around an SQL statement and find & replace. So, here’s my new work-around:

SELECT name FROM system WHERE type = 'module' AND status = 1 AND filename LIKE 'sites/all/modules/%'

That gets the names of all the currently active contributed modules. Then I exported it to a CSV, and modified it so that each line read:

UPDATE system SET status='0' WHERE name='modulename';

Saved that as disable-modules.sql, then made a copy of the file, replacing 0 with 1, and named the new file enable-modules.sql.

Then all I had to do was run each file at the correct point in the upgrade sequence. I was pleasantly surprised to discover that it worked.

Things that made this MUCH nicer: not having to disable/enable modules in a specific order; not worrying about inducing the white screen of death; and being able to set up the files ahead of time. Since I have to come in at odd times to do updates, saving time is incredibly nice. 🙂

less painful than I expected

So I run two full versions of the site, and until I take the time to get Deploy etc working, I periodically have to reset the test version as it gets absurdly out of whack with the live version.

For reasons that are too boring to get into at the moment, I had to do that earlier this week. When I did, I completely spaced on the fact that a few weeks ago I was working on a major overhaul to some of the content types. (See my comments about rate tables in the conversion post from last September.) It took me quite a while to figure out the best way — or at least a better way — to get it set up. I hadn’t yet implemented it on the live site, because I was waiting on someone else’s schedule.

Today, the meeting chime came up saying that I was supposed to show the new setup to that someone.

Oh.

Crap.

Digression: some of you are aware of the ampersand disaster of 1996. (If not, see this post from 2001.) The sense of panic, sinking OMGWTFBBQ, punch-to-the-gut that I felt when I realized was almost exactly the same as when Sandra brought me that label addressed to “S&ra”.

I postponed the meeting and immediately (ok, after taking a little break for toast) set to work. At first, I thought the best call would be to find all the references in the database backup to the node type, nodes, and views and just copy in the SQL. Yes, making a backup is an established part of my reset routine!

But ugh. It wasn’t long at all before I determined that it was just too crazy-messy to even consider, which left me feeling pretty low. (And stupid, good grief the burning stupid.)

Then I realized I’d made custom theme files for the node types in question! That gave me the names of the content fields and some information about the node type settings and how I had used taxonomy. Then I discovered that my habit of letting Firefox remember things I type into fields had left a memory of the names I used, not just for the content fields, but for the multigroups they were bundled into. I did use the SQL backup to figure out what taxonomy vocabulary I’d used and what the terms were, plus the allowed values for a couple of fields. Dreamweaver is actually not too bad for searching ginormous SQL files.

Once I had the content types set, figuring out the Views I needed was fairly obvious. (To me, anyway.) I think I even improved them over my previous attempt. All that remained was copying in a bit of CSS from the backup version of the theme, and in less than 4 hours I had it all back together!

So yay for Drupal. (It’s been a week of “Drupal hates me” vs “Yay for Drupal” — at the moment I think “yay” is winning out.)

best of the year

There’s been a lot of stress and frustration over this last year, but in an effort to be positive & optimistic, here’s a few of the highlights of my 2009, the things that make me smile almost instantly when I think about them:

Creamsicle the cat…he actually came into our lives in the summer of 2008, as a crazy stray tom with matted dreadlocks of orange and white fur. But after keeping him safe through the snowpocalypse, he began to be part of the posse.

In March, we officially adopted him via a trip to the animal shelter — he was so ill-mannered in the cages that he was set to be killed — while he had a wound in his head from fighting, so bad that it had to have a tube put in it. We weren’t sure if it was going to work out, what with the fighting (not with ours, oddly enough), running away, spraying, etc., but wow! Neutering worked wonders. 🙂

His bad behavior gradually ended over the course of a few weeks, and with lots of love and attention he’s become a lovely friendly kitty, crazy about laps and treats. He’s the only cat I’ve ever had who sits on my lap in a way that allows me to keep typing, which I love. He still has a tendency to be “talkative” at night, but often if you just yell “hey” back at him that’s good enough to calm him down.

The Xtracycle, about which I have written several times: my awesome, awesome new bike this summer, the load-hauling machine.  Rides to the river, massive grocery trips, and plenty of commuting.

Working with Drupal — I converted my employer’s site this year, and so spent a lot of time playing with it, learning, and getting to know the delightful Drupal community. Even went to two (free!) events in Seattle this year, both of which were immensely helpful, even if the second one was cut short by C’s brush with the flu. I’m getting that feeling of mastery that’s so joyful (and a little addictive) — I feel like I could do wondrous things with Drupal.

And finally NaNoWriMo, which gave me back confidence in my creativity. As I’ve mentioned before, I’ve made a couple of unsuccessful attempts. I’ve also been stalled out on my writing for several years, stuck unable to finish the novel I’ve been puttering at since…um… 1994? NaNoWriMo gave me a chance to jump into something entirely new, and to commit to finishing it. I even tried some entirely new (to me) techniques, and discovered to my surprise that a little bit of planning ahead worked really, really well. (Snowflake method FTW!) Plus I found the fabulous Scrivener. I finished above the goal and before the deadline.

If you’re wondering, I’ve decided to continue with it, after some wonderful feedback from Paula, and after rereading it myself. I still like this story and its characters. I’ve already done some early editing, mostly continuity stuff. (There’s only one character whose name didn’t stay consistent, and he’s a pretty minor one at that!) I’m planning on getting through a complete second draft early in the new year. What happens after that I’ve no idea.

Which is how I feel about 2010 in general: I’ve no idea what happens now, really.

Drupal!

[this post is probably a draft. might be adding links and such later.]

As of yesterday, my work site is running on Drupal. It’s been a long process, starting probably about a year and a half ago. Yesterday I was a little insane getting it live, but I’m really happy now.

When I started this job, the site was all static pages, no templates even, with a number of small home-grown PHP apps to post the rates, jobs, and a couple of other little things, plus phpAds to show promotions. Since then, I added templates, upgraded phpAds, reworked the home-grown apps, added a WordPress installation for news, and Kayako’s Support Suite for email and chat. It’s been good, but unwieldy. And honestly? I’m a jack of all trades, not a PHP programmer, so those apps have always been a little…fragile.

I’ve used WordPress for a long time (heck, I’m using it right now), so I did look at it as an option. But the site in its cobbled-together form was just too complex to recreate without a lot of insane hacking, which gets back to me being the programmer.

I’d also used Drupal before, in the 4.x version back at Pierce. My snap judgement was that it was incredibly powerful, and a PITA to set up and manage. Taking a whirl with 5.x improved my opinion of setup and management…vastly, and at the time, in the early days of 6.x, it looked like that was going to be another big step up. So I started futzing, initially with 5.x. The point was, basically, to recreate what was on the site already, but in a unified interface via Drupal.

It was right at that cusp of deciding whether to use D5 or D6: for a while I had a spreadsheet with the modules I thought I’d need, noting where they were on D6 conversion. (I now have a similar one with notes about D7.) Then I had an opportunity to do a small subsite that didn’t need as much functionality, and it was my first go-round with D6. And it was good. 🙂 It went live about this time last year; the site setup — and its maintenance since then — have been quite good, which gave me confidence to go ahead with D6…once the modules were ready.

That’s the thing about Drupal, even more so than WordPress: there’s a module to do damn near anything, but not all of them have the same level of attention or quality. Many depend on the scratching-your-own-itch phenomenon, and if the itch is small enough, and once you’ve taken care of it, there’s not much motivation to keep the thing alive.

So it wasn’t really until maybe this spring that enough of the pieces were in place that I could definitely go ahead. From there on, I was gradually puttering along recreating functionality and figuring out how Drupal works.

I have to give huge kudos to Apress and the Pro Drupal Development book. I ended up with a free PDF copy after SXSW, and that thing really got me going. I spent several afternoons at the nearby Starbucks with printed out chapters, scribbling in the margins and slapping in post-it notes. Chapter 8 on theming was particularly helpful, as was Chapter 12 on search. Eventually I’ve worked my way (mostly) past the cargo-cult-coding stage, those pieces helped me get there.

Getting the theme right was a big deal, and though I still feel like I haven’t done it to my own personal standards, I’m thrilled that I couldn’t tell the difference, visually, between the old site and the new. I used Zen as the base theme, and went nuts from there. There are some particulars that I decided to handle differently, but those were (mostly) conscious choices. I was even able to clean up some of my design work from last summer’s redesign, which is always nice.

The other things that helped tons with theming were the Devel and Contemplate modules, along with Firebug. Devel and Firebug helped me see what was being created…Firebug in particular was great for converting class and ID names in the CSS. Trying out templates with Contemplate helped me understand how to create my own template files for all the different node types, even content fields.

Which brings up CCK, I suppose. Another one of those amazing, powerful, and often crazy-making tools. The thing that took the longest of the whole project was creating our rate tables. People love ’em, but wow are they complicated, especially since the folks who do the entry like to be able to enter rates ahead of time to appear when they become effective. I discovered the (experimental!) multigroup feature…and for that feature, my needs were actually pretty simple. It took me a long time to figure out how to theme a multigroup; the day that happened was the occasion of much rejoicing. I’ll be interested to see how it evolves in CCK 3, for sure. In any case, I had to try many different versions of content types for the rates before settling on my current scheme. Of course, that involves Views as well, which I’m also pretty thrilled with overall, even if there are bits and pieces that remain entirely mysterious to me. (Relationships?! And I never did get multiple arguments working quite how I wanted.)

I kept discovering modules over the project, occasionally that led to tearing up one way of doing something and trying something entirely different, running into dead ends, etc. (Don’t ask about nodepicker for tinymce.) I think I’ve gotten a feel for deciding whether a module is going to be worth my time…a look at the issues queue is often very instructive.

Some highlights: Panels is way cool for some of the multiple-resource type pages, and had I tried it out sooner, I probably would’ve used it for the homepage. Actually, the article I’m writing for our email newsletter, about site enhancements, is going to be entirely about new features made possible by Panels. Clone Module is simple and does exactly what I needed it to. Insert Block and Insert View are incredibly valuable for drawing little chunks of content into other pages. ThemeKey makes me a little crazy (sometimes I swear my selections don’t take) but very useful for handling some particular admin-type, but not actually admin, pages (also helpful: Simple Access), as well as a subsite that has a radically different theme. I still feel like I don’t totally understand Organic Groups, but that subsite couldn’t happen without it. I’m figuring I’m going to learn a whole lot more over the next few months! GMap rocks the branch map, although I still really want a way to make the links on the list of branches open the associated bubbles. I could do that with my hand-built code, but not with the module. 🙁 Getting advanced processing working in Webforms was one of those fist-in-the-air moments, and it took me a reeeeeally long time to get there. I should probably write some more detail about how I made that work.

I even wrote my own module. 🙂 It’s based on one from Pro Drupal Development, and searches the knowledgebase that comes with our contact center site. (Coordinating those two pieces has been a challenge, and I’m still not really done. I’ll be futzing with that for probably several more weeks.) It started out pretty cargo-cult, and then I had one of those epiphanies, and it just all came together.

The whole Drupal experience was a little like that: having a vague idea of what I wanted to happen, searching out a bunch of different ways to do it, being horribly frustrated, then usually pretty suddenly, figuring it out and going gangbusters. There’s a lot of power there, and with great power, etc., etc. Also, try not to cut your hand off with that chainsaw.

Importing data is another piece I should probably write about in more detail. I ended up doing half a dozen different things, but most useful were Node Export,  Taxonomy Import/Export via XML, Node Import, and Backup and Migrate. The first two I used primarily in transferring data from the Drupal subsite into the new site, and the first is really nice for moving individual nodes. Node Import was the happy bulk tool, although I did have to copy/paste a lot of HTML from individual files into a spreadsheet: clunky, but for the basic info pages, it worked just fine. Backup and Migrate helped most in the transition from test to live, and I’m assuming it will continue to serve that function. I even had to revert the live version once during the install because I did something incredibly dumb, and the reversion went wonderfully well. 🙂 There’s also a module specifically for importing WordPress content that got all of our news posts in nice and easy. The name escapes me at the moment. I am frustrated, though, at how long it took me to figure out how to do all that. I found a page, eventually, listing a bunch of options, and that helped…but at the time, it didn’t list Node Export! So there you are.

Which reminds me: I did get helpful information on a number of topics via the Drupal community in Seattle. In particular, Greg Dunlap’s slides on data import and deployment gave me lots of good ideas, and Jennifer Hodgdon’s DrupalCamp presentation about Views helped me get my brain around that stuff. I’ve been able to go to a few Drupal events in Seattle now, and while I haven’t always been able to stay the whole time, they’ve been great opportunities.

Twitter’s been a source of advice, assistance, encouragement and sympathy as well. In addition to Greg (heyrocker): gregoryheller, lauras, joeeastham, mahalie, xtfer, and shelleypowers. (In book acknowledgement tradition, I’m sure I’m missing someone.) I continue to be amazed by the power of the internet for learning and sharing.

I wanted to pour all of this out there while it was still fairly fresh in my mind. A lot of time has passed already, and it helps for me to write about the experience of getting started. I know I’m not done either. There’s still plenty to learn, features and modules I want to try, and new projects for the site.

Oh, and as you might imagine, I’ll be at the Pacific Northwest Drupal Summit next month.

(As for this site? I’m sticking with WordPress. It’s just about perfect for the basic bloggy thing that I’ve been doing here for the last 8+ years, and I still enjoy using it. I’m still into it for basic sites, too: the Eastside Neighborhood Association, Christmas Tree Lane, Friends of the Olympia Library. There’s a jump in complexity from WordPress to Drupal that isn’t always necessary, or even a good idea. In the same way that I’m essentially agnostic in the Windows/Mac religious wars, I’m good with both systems, for their own purposes.)