We’re a team of Ruby on Rails developers and UX practitioners at PatientsLikeMe, where patients share data about their treatments, symptoms, and disease outcomes. We’re classically trained ninjas, pirates, rockstars, and dinosaur hunters. Our keyboards are magic wands — with lots of buttons, so they’re even better than regular magic wands. Meet the Team ↓
PatientsLikeMe has built our own version of Google’s “20% Time” that we call “WOW Week”. WOW Week is a week of unstructured development time for engineers, where they can work on anything to improve our products as long as they demo their progress in front of the company at the end of the week.
The engineering team works in 2-week long development sprints. After three development sprints in a row, we have a “Technical Debt” week and a WOW Week.
Why a Week at a time versus 20% Time?
It’s easy to pay lip-service to the concept of 20% time for engineers while scheduling a full load of work. I’ve seen this happen many times at other companies. PatientsLikeMe avoids this pitfall by creating a public block of time for the entire company.
Scheduling a complete week allows a single context-switch into innovation mode for everyone. This maximizes the value of this time, instead of dividing it into smaller chunks that are diluted by context switching and deadlines.
Why do it at all? Benefits for Individuals and Product as a whole
WOW Week has profound impact on the individual behavior and development of engineers as well as launching innovative product ideas and proving risky concepts.
Engineers are often eager to find problems or inefficiency and tell you about it. As a leader, this can be a frustrating trend. Letting engineers set their own agenda and priorities helps them weigh the value of action versus simply advocating what should be done. Everyone is better off when people use their own time and energy to solve the problems they care about and become more action oriented.
As engineers develop in their career, they need to grow from executing plans to owning a solution to a problem from start to finish. Working on your own projects and solving problems you care about is often the most effective way to engender ownership. This kind of ownership is a habit that can carry over into regularly scheduled work.
Products: Innovation, Proof of Concepts, and Failure
It’s difficult to balance the trade-offs between executing a plan efficiently and the creative work of synthesizing new directions. Having an entire week of time allows engineers to engage deeply with a concept. That might mean building a working prototype to explore a concept and even having time to iterate on the design. Innovation and execution use two completely different parts of the brain, so it’s useful to stay in innovation mode.
Some ideas are popular but considered too difficult or risky to schedule in a normal release. An engineer can grapple with that problem and build a prototype that mitigates the risk and prepares that feature for normal scheduling.
Labeling a project a risky proof of concept builds a safe space for failure. If some of your projects aren’t failing then you’re playing it safe to avoid public failure. We embrace the courage to fail during WOW Week in ways we wouldn’t during a scheduled release when we’re committed to shipping.
Downsides: What if it never ships?
The passion of engineers isn’t always aligned with the product or the business, so it can be dangerous to stoke that passion. Sometimes an engineer’s pet project or feature doesn’t fit with the product or conflicts with a business goal. This can be a difficult outcome to accept, but the frustration is generally worth the experience of being able to realize your ideas.
Additionally, having something tangible to serve as the focus of debate about whether to release your feature elevates the conversation. Both sides can clearly see what’s at stake in a tangible way.
Overall: Trust Your Engineers and Invest in Their Growth
Building a program of unstructured development time is a long term investment, in both the engineers on your team and the projects they produce. It’s much like basic research in the sciences – it often pays dividends, but rarely in the ways you expect.
Yesterday, I was marking up a mockup (as you’ll often find me doing). It was time to convert this little navigation progress indicator:
So, when marking this up, it’s clearly a list. Since it is a list of steps you will take as you progress through this feature, it’s an ordered list. So, here’s the markup:
It’s a little span-heavy, I’ll admit. We tend to use an empty span.icon for dropping in icons from sprites. I would ideally just use background images, but this image needs to be reused in many different formats, so there’s no good way to avoid exposing the remainder of the image in all cases. So, if you want to use sprites and you want to re-use the image in multiple arrangements, this is sometimes the best way to go.
Besides, the span doesn’t actually appear in the unstyled markup. So, we’re cool.
Here’s how I’m styling this. Yes, it’s in Sass, but Sass is so easy to read that I’m sure you know what it means.
Just so you know, here are some mixins I use here:
+reset is just a simple mixin we drop in to kill the padding and margin on an element.
+alternate_header_font is what we use to drop in Lucida Grande. It also adjusts the letter-spacing when that typeface is used.
+sprite_icon is totally badass. Using syntax like +sprite_icon(-279px, -100px, 60px, 60px, "span.icon", $quick_start_sprite), I can (in one line) establish which sprite to use ($quick_start_sprite is a variable that directs to the path of the image), which element to apply the mixin to (span.icon), set the background positioning (-279px -100px) and the height and width (60px and 60px). It’s basically magic.
Those right angle quotes between the images bugged me a bit. They certainly shouldn’t be list items. In fact, they’re purely presentational, so they shouldn’t appear in the markup at all. I originally thought “background-image“. But a right angle quote is just text. That seemed like overkill. So, I opted for pseudo-elements.
Once I created the angle quotes using the :before pseudo-element (content: '»'), they stacked themselves on top of the images. That’s no good. So, I tried absolutely positioning them. It never occurred to me until yesterday that I could absolutely position generated content. But it was easy.
I used the :first-child pseudo-element to nix the margin on the first list item and then used two pseudo-elements at once to remove (yup, that’s a selector that looks like ol#steps li:first-child:before) the generated content from the first list item.
“But Adam, what about IE???” IE9? You’re covered. IE8? You’re also fine. IE7 and IE6? Guess what? They won’t appear. You’ll see three icons, but no angle braces. This works for me. It should work for you. This isn’t like your logo is missing or something. They are presentational angle quotes.
If not showing the angle quotes in IE7 and IE6 really bugs you, you could make a background image and serve it just to those two “browsers”. Seriously, have fun with that.
Summary
We have a network of production monitoring tools at patientslikeme.com, where monit, NewRelic, and Pingdom feed alerts through PagerDuty to produce e-mail, SMS, and Pager alerts for production issues. PagerDuty has a ticketing system to assign a given problem to a single person. It’s awesome.
Why PagerDuty? PagerDuty is the heart of the monitoring system at PatientsLikeMe. We’ve configured monit, NewRelic, and Pingdom to fire e-mail notifications to PagerDuty. PagerDuty collects all of the notifications, applies filtering rules, and opens Incident Tickets for anything needing attention.
Filter Out Signal from Noise (monit instance alerts, etc)
Single Person Assigned to Each Issue at a Time, w/ Escalation
Wired up E-Mail, SMS, and old-school Pagers to get your attention (even at 4am)
will_paginate is one of those gems that always makes me smile. Back in the dark ages of Spring MVC & Hibernate I specifically remember being part of an estimation that decided that adding pagination to an admin page would take 2 days. Today that seems crazy of course, but I don’t think it was a bad estimate at the time since there really wasn’t any built-in support for that. Fast forward to now and as we all know pagination is one of those things that take about 2 minutes to implement. Progress!
So pagination is great and a pretty much solved problem, but it’s always irked me that it requires 2 very similar queries per request. Is there anything to be done about that? Let’s see.
Pagination at the database level should be very simple. All it requires is adding a LIMIT and OFFSET to your query. Of course if you add a LIMIT to your query, you can no longer simply count the number of rows in order to get the total number of matching results and we’ve got to know how many total matches there are so that we can calculate the total number of pages. So it seems like we’re forced to query for that separately and that’s exactly what will_paginate does. Here’s a simple example from the PatientsLikeMe patient search.
SELECT count(*) FROM users WHERE users.role_id=1 and users.verified is true and users.deleted is false and users.birth_date < now() – interval ‘20 year’ and users.sex in (‘M’) AND users.last_update_time IS NOT NULL
SELECT users.id FROM users WHERE users.role_id=1 and users.verified is true and users.deleted is false and users.birth_date < now() – interval ‘20 year’ and users.sex in (‘M’) AND users.last_update_time IS NOT NULL ORDER BY users.last_update_time desc LIMIT 15 OFFSET 0
Two of those simple queries are no big deal, but as you start to add disease specific search criteria, you’ll find that the queries can get a lot more complex very quickly.
SELECT count(*) FROM users WHERE
users.role_id=1
and users.verified is true
and users.deleted is false
and users.birth_date < now() – interval ‘20 year’
and users.sex in (E’M')
and users.id in ( SELECT ci.user_id FROM condition_infos ci WHERE ci.condition_id = E’4′ and first_symptom_date between ‘1911-04-06′ and ‘2004-04-06′)
and users.id in ( SELECT ci.user_id FROM condition_infos ci WHERE ci.condition_id = E’4′ and first_definitive_diagnosis_date between ‘1992-04-06′ and ‘2111-04-06′)
and users.id in ( WITH latest_outcome as (select * from (select *, rank() over (PARTITION BY user_id order by date desc, id desc) from pdrs_surveys) ranked where rank = 1) select user_id from latest_outcome where updrs_score >= 10 )
and users.condition_ids @> ‘{4}’
and users.last_update_time IS NOT NULL
Footnote: note that I get that this query is not perfect, but do keep in mind that this comes out of a dynamic query generator and is not hand-written.
Having to run that query twice starts to affect response time. But what’s an engineer to do? Is it possible to easily make Postgres return both the limited results and the overall count in one go? It sure is! Common Table Expressions and Window Functions to the rescue.
Common Table Expressions are available in Postgres 8.4 and above and are just about my favorite SQL hammer. I find they allow for much much cleaner SQL when the going gets rough, because they allow you to separate elements of a query into small compassable parts.
Window Functions are brilliant as well and they’re going to do the heavy-ish lifting in our one-hit SQL pagination query. Generally they’re used in order to calculate something over a subset of results, but the trick that makes the following code work is they’re also happy to run over the entire result set by just not specifying a PARTITION.
Combining these two tricks together, we can create the following 1-shot query to replace to two queries we started with.
WITH sql_query as (SELECT users.id FROM users
WHERE users.role_id=1 and users.verified is true
and users.deleted is false and users.birth_date < now() – interval ‘20 year’
and users.sex in (E’M')
AND users.last_update_time IS NOT NULL ORDER BY users.last_update_time desc )
select count(*) OVER () as total_entries, * from sql_query LIMIT 15 OFFSET 0
This returns all of the user_ids, but adds a column to each row that contains the total_entries.
So what’s the performance improvement? Well for the hairy query above, on my local machine I get:
Count query: ~157ms
UserID query: ~165ms
vs
Combined Query: ~164ms
That’s free as in beer!
The last step is to integrate this with will_paginate. I’ve only taken a quick stab at that so far but it was simple to implement. All we need to do is wrap the pre-existing query within our CTE, run the query and then extract value of the total_entries column from any of the rows. A full will_paginate speedup would require detecting Postgres and implementing this for the other finders instead of just paginate_by_sql.
I hope you enjoyed this episode of Mav hits another SQL query with his Common Table Expression hammer. Stay tuned for “OMG You did that INSIDE! the JOIN”.
Earlier this week, I presented at Refresh Boston along with my friend Joel Oliveira of the 47th. Joel and I met at Build Guild a while back and have stayed in contact, mostly on the topic of Sass. While Joel and I both love using Sass, it seemed many of our fellow designers and developers were still hesitant to use it.
Rumor has it, Joel took some video and it may be surfacing soon. If so, I’ll update the post here so you can watch it and tell us we’re wonderful presenters. (Gosh, you’re nice.)
A lot of people from various stages of my professional life made the trip for the talk and I really appreciate it. It was great to see everyone again—and great to try to convince you all that Sass is the future. I hope it worked!
As a member of PatientsLikeMe's Data team, from time to time we're asked to compute how many unique users did action X on the site within a date range, say 28 days, or several date ranges (1,14,28 days for example). It's easy enough to do that for a given day, but to do that for every day over a span of time (in one query) took some thinking. Here's what I came up with.
One day at a time
First, a simplified example table:
create table events ( user_id integer, event varchar, date date)
Getting unique user counts by event on any given day is easy. Below, we'll get the counts of unique users by events for the 7 days leading up to Valentine's day:
select count(distinct user_id), event from events
where date between '2011-02-07' and '2011-02-14'
group by 2
Now Do That For Every Day
The simplest thing that could possibly work is to just issue that query to compute the stats for the time span desired. We're looking for something faster, and a bit more elegant.
Stepping back a bit, for a seven day time window, we're asking that an event on 2/7/2011 count for that day, and also count for the 6 following days - effectively we're mapping the events of each day onto itself and 6 other days. If this was a map/reduce tool, we could emit the event 7 times, one under each date key. In SQL, that sounds like a join waiting to happen. Once joined, its easy to group by the mapped date, and do a distinct count.
With a table like the one below:
from_date
to_date
2011-01-01
2011-01-01
2011-01-01
2011-01-02
2011-01-01
2011-01-03
2011-01-01
2011-01-04
2011-01-01
2011-01-05
2011-01-01
2011-01-06
2011-01-01
2011-01-07
2011-01-02
2011-01-02
...
This SQL becomes easy.
select to_date, event, count(distinct user_id) from events
join dates_plus_7 on events.date = dates_plus_7.from_date
group by 1,2
The output looks like this:
to_date
event
count
...
2011-01-05
bar
20
2011-01-05
baz
27
2011-01-05
foo
24
2011-01-06
bar
31
...
You'll then need to trim the ends of your data to adjust for where the windows ran off the edge of the data.
That works for me on Postgresql 8.4. Your mileage may vary with other brands.
How Do I Get One of Those?
A dates table like that is a one-liner using the generate_series method:
select date::date as from_date, date::date+plus_day as to_date from
generate_series('2011-01-01'::date, '2011-02-28'::date, '1 day') as date,
generate_series(0,6,1) as plus_day ;
There we get the cartesian product of the set of dates in the desired range, and the set of numbers from 0 to 6. Sum the two, treating the numbers as offsets and Bob's your uncle.
It’s all about the forward slash, people. When you put a forward slash in front of the controller name, url_for will treat you right. Does the Rails documentation for url_for tell you this? No, it does not. This guy mentions it, though he’s not my original source for the fix, which I can’t find again (sorry, original source). With luck, one more blog post on the subject will make it easier for others to google the answer fast.
That’s all, folks. Little problem, little solution. Yay.
When I start building a WordPress theme, one of the first things I do is “Save As” from a theme I built in the past. WordPress is great and has a lot of neat features—I just don’t need all of them. I like making simple themes that let the content speak for itself. That was one of the goals when building the theme for the PatientsLikeMe Tech Blog.
This past August, Cris and I converted the Tech Blog theme to HTML5. I think it’s useful enough that we decided to take the theme, strip out the PatientsLikeMe-related stuff, and share it for anyone who wants it. Let’s call it Save As.
Again, I’ll stress that it’s a very basic theme—quite intentionally. Here’s what you’ll get from it:
Markup that is completely free from the CSS. In fact, the theme includes only a few lines of CSS—so you can do it yourself.
All the HTML5 tags you’d come to expect: header (with hgroup!) for the blog title and tagline, footer for the… footer, aside for the sidebar (though I used an aside with the ID “sidebar” so that you can use asides in your individual posts, too), time for proper post time-stamping, section for the main parts of the page (like the main posts area or the comment form), article for individual posts, and even mark for highlighting search matches (in titles only, for now) on your search results page.
A fresh start from which to create some awesome.
The theme is available on Github. I’d love it if you’d download it, try it out, and build upon it for your next HTML5 WordPress theme. I know I will!
In between creating awesome for PatientsLikeMe, Jeffrey and I built an app based on the Dribbble API called Liiikes. If you want to know more about Liiikes, you can check out the About page, my blog post, or Jeffrey’s announcing it. Today, we’re gonna talk about just one little design element.
Specifically, I want to talk about the arrow on this popup:
What I wanted to solve:
Attach an arrow to an absolutely positioned box to anchor it to the draftee count.
Don’t use images (since the entire app doesn’t use images).
Needs to accept rGBA background color values (to get some subtle transparency—for this reason, rotating a square wouldn’t work because you’d see the overlap).
If you suppress the content area, you suddenly get this:
You seeing what I’m seeing? Now make 3 of those borders transparent!
Blizzam! Now all you have to do is absolutely position it to align nicely with your popup, or whatever you’re attaching the arrow to.
For the Liiikes example, here’s the HTML. In this case, I used an empty div. If that makes you uncomfortable, I’m sure you can apply it to another element somewhere. You’re a smart kid.
<div class="arrow"></div>
And here’s the good stuff:
div.arrow {
position: absolute;
top: 20px;
left: -40px;
border-top: 20px solid transparent;
border-bottom: 20px solid transparent;
border-right: 20px solid rgba(0,0,0,.85);
border-left: 20px solid transparent;
}
Basically, if you’ve merged a feature in by mistake, you can simply revert the merge to get back to a happy state. The mind bending comes when you decide that the feature is ready to be merged for real. While intuition might tell you to just git merge feature again, what you really want to do is “revert the revert”.
It makes sense, and it’s awesome, and it’s righteously scary in that wonderful way that only git can be.
At PatientsLikeMe we were familiar with this, but we hadn’t had a chance to do it in practice until last week. A branch was merged in prematurely and we successfully reverted.
The new wrinkle that I faced today was that the branch that was merged in and since reverted, is actually a long running branch and development continues on it. I wanted to merge master into this long running branch to stay up to date, but when I did that, I noticed pretty quickly that something was awry. 27 conflicts and a bucket of fail. Worse, my mergetool was consistently picking the wrong side of the merge.
Let’s go to the simulated instant replay, sponsored by GitX.
This is the state of the repo after we’re reverted our accidental merge.Here is the repo with more work on both the feature and the master branch.
So, I’d like to merge master into the feature branch, but that leads to a sorts of erroneous conflicts. Why? Well, because git merge master asks git to merge the revert commit into my feature branch, that’s why. The revert commit essentially contains diffs that say to remove all the early work on feature branch. It makes sense to apply this to master, but it is a terrible commit to apply to the feature branch. It’s a bit like time travelling and accidentally killing your parents.
So what do we do? We know that we need to ‘revert the revert’ at some point. It turns out that there’s no reason not to do that immediately; we just need a new ‘shim’ branch. We’ll call it master_w_revert_reverted.
Voila! we’ve successfully merged our master branch into the long running feature and we’ve taken care of the ‘revert the revert’ going forward.
The next time we want to merge from master to the long running feature branch, or indeed from when we’re ready to merge the feature into master, we just merge as we would normally. Our days of reversion are over.
