http.Handler and Error Handling in Go

 ∗ go, web, go

I wrote an article a while back on implementing custom handler types to avoid a few common problems with the existing http.HandlerFunc—the func MyHandler(w http.ResponseWriter, r *http.Request) signature you often see. It's a useful "general purpose" handler type that covers the basics, but—as with anything generic—there are a few shortcomings:

My previous approach used the func(http.ResponseWriter, *http.Request) (int, error) signature. This has proven to be pretty neat, but a quirk is that returning "non error" status codes like 200, 302, 303 was often superfluous—you're either setting it elsewhere or it's effectively unused - e.g.

func SomeHandler(w http.ResponseWriter, r *http.Request) (int, error) {
    db, err := someDBcall()
    if err != nil {
        // This makes sense.
        return 500, err
    }

    if user.LoggedIn {
        http.Redirect(w, r, "/dashboard", 302)
        // Superfluous! Our http.Redirect function handles the 302, not 
        // our return value (which is effectively ignored).
        return 302, nil
    }

}

It's not terrible, but we can do better.

A Little Different

So how can we improve on this? Let's lay out some code:

package handler

// Error represents a handler error. It provides methods for a HTTP status 
// code and embeds the built-in error interface.
type Error interface {
    error
    Status() int
}

// StatusError represents an error with an associated HTTP status code.
type StatusError struct {
    Code int
    Err  error
}

// Allows StatusError to satisfy the error interface.
func (se StatusError) Error() string {
    return se.Err.Error()
}

// Returns our HTTP status code.
func (se StatusError) Status() int {
    return se.Code
}

// A (simple) example of our application-wide configuration.
type Env struct {
    DB   *sql.DB
    Port string
    Host string
}

// The Handler struct that takes a configured Env and a function matching
// our useful signature.
type Handler struct {
    *Env
    h func(e *Env, w http.ResponseWriter, r *http.Request) error
}

// ServeHTTP allows our Handler type to satisfy http.Handler.
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    err := h.h(h.Env, w, r)
    if err != nil {
        switch e := err.(type) {
        case Error:
            // We can retrieve the status here and write out a specific
            // HTTP status code.
            log.Printf("HTTP %d - %s", e.Status(), e)
            http.Error(w, e.Error(), e.Status())
        default:
            // Any error types we don't specifically look out for default
            // to serving a HTTP 500
            http.Error(w, http.StatusText(http.StatusInternalServerError),
                http.StatusInternalServerError)
        }
    }
}

The code above should be self-explanatory, but to clarify any outstanding points:

If we don't want to inspect them, our default case catches them. Remember that the ServeHTTP method allows our Handler type to satisfy the http.Handler interface and be used anywhere http.Handler is accepted: Go's net/http package and all good third party frameworks. This is what makes custom handler types so useful: they're flexible about where they can be used.

Note that the net package does something very similar. It has a net.Error interface that embeds the built-in error interface and then a handful of concrete types that implement it. Functions return the concrete type that suits the type of error they're returning (a DNS error, a parsing error, etc). A good example would be defining a DBError type with a Query() string method in a 'datastore' package that we can use to log failed queries.

Full Example

What does the end result look like? And how would we split it up into packages (sensibly)?

package handler

import (
    "net/http"
)

// Error represents a handler error. It provides methods for a HTTP status 
// code and embeds the built-in error interface.
type Error interface {
    error
    Status() int
}

// StatusError represents an error with an associated HTTP status code.
type StatusError struct {
    Code int
    Err  error
}

// Allows StatusError to satisfy the error interface.
func (se StatusError) Error() string {
    return se.Err.Error()
}

// Returns our HTTP status code.
func (se StatusError) Status() int {
    return se.Code
}

// A (simple) example of our application-wide configuration.
type Env struct {
    DB   *sql.DB
    Port string
    Host string
}

// The Handler struct that takes a configured Env and a function matching
// our useful signature.
type Handler struct {
    *Env
    h func(e *Env, w http.ResponseWriter, r *http.Request) error
}

// ServeHTTP allows our Handler type to satisfy http.Handler.
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    err := h.h(h.Env, w, r)
    if err != nil {
        switch e := err.(type) {
        case Error:
            // We can retrieve the status here and write out a specific
            // HTTP status code.
            log.Printf("HTTP %d - %s", e.Status(), e)
            http.Error(w, e.Error(), e.Status())
        default:
            // Any error types we don't specifically look out for default
            // to serving a HTTP 500
            http.Error(w, http.StatusText(http.StatusInternalServerError),
                http.StatusInternalServerError)
        }
    }
}

func GetIndex(env Env, w http.ResponseWriter, r *http.Request) error {
    users, err := env.DB.GetAllUsers()
    if err != nil {
        // We return a status error here, which conveniently wraps the error
        // returned from our DB queries. We can clearly define which errors 
        // are worth raising a HTTP 500 over vs. which might just be a HTTP 
        // 404, 403 or 401 (as appropriate). It's also clear where our 
        // handler should stop processing by returning early.
        return StatusError{500, err}
    }

    fmt.Fprintf(w, "%+v", users)
    return nil
}

... and in our main package:

package main

import (
    "net/http"
    "github.com/you/somepkg/handler"
)

func main() {
    db, err := sql.Open("connectionstringhere")
    if err != nil {
          log.Fatal(err)
    }

    // Initialise our app-wide environment with the services/info we need.
    env := &handler.Env{
             DB: db,
             Port: os.Getenv("PORT"),
             Host: os.Getenv("HOST"),
             // We might also have a custom log.Logger, our 
             // template instance, and a config struct as fields 
             // in our Env struct.
    }

    // Note that we're using http.Handle, not http.HandleFunc. The 
    // latter only accepts the http.HandlerFunc type, which is not 
    // what we have here.
    http.Handle("/", handler.Handler{env, handler.GetIndex})

    // Logs the error if ListenAndServe fails.
    log.Fatal(http.ListenAndServe(":8000", nil))
}

In the real world, you're likely to define your Handler and Env types in a separate file (of the same package) from your handler functions, but I've keep it simple here for the sake of brevity. So what did we end up getting from this?

If you have questions about the post, drop me a line via @elithrar on Twitter, or the Gopher community on Slack.

Running Go Applications in the Background

 ∗ go, web, go

A regular question on the go-nuts mailing list, in the #go-nuts IRC channel and on StackOverflow seems to be: how do I run my Go application in the background? Developers eventually reach the stage where they need to deploy something, keep it running, log it and manage crashes. So where to start?

There's a huge number of options here, but we'll look at a stable, popular and cross-distro approach called Supervisor. Supervisor is a process management tool that handles restarting, recovering and managing logs, without requiring anything from your application (i.e. no PID files!).

Pre-Requisites

We're going to assume a basic understanding of the Linux command line, which in this case is understanding how to use a text-editor like vim, emacs or even nano, and the importance of not running your application as root—which I will re-emphasise throughout this article! We're also going to assume you're on an Ubuntu 14.04/Debian 7 system (or newer), but I've included a section for those on RHEL-based systems.

I should also head off any questions about daemonizing (i.e. the Unix meaning of daemonize) Go applications due to interactions with threaded applications and most systems (aka Issue #227).

Note: I'm well aware of the "built in" options like Upstart (Debian/Ubuntu) and systemd (CentOS/RHEL/Fedora/Arch). I'd even originally wrote this article so that it provided examples for all three options, but it wasn't opinionated enough and was therefore confusing for newcomers (at whom this article is aimed at).

For what it's worth, Upstart leans on start-stop-daemon too much for my liking (if you want it to work across versions), and although I really like systemd's configuration language, my primary systems are running Debian/Ubuntu LTS so it's not a viable option (until next year!). Supervisor's cross-platform nature, well documented configuration options and extra features (log rotation, email notification) make it well suited to running production applications (or even just simple side-projects).

Installing Supervisor

I've been using Supervisor for a long while now, and I'm a big fan of it's centralised approach: it will monitor your process, restart it when it crashes, redirect stout to a log file and rotate that all within a single configuration.

There's no need to write a separate logrotated config, and there's even a decent web-interface (that you should only expose over authenticated HTTPS!) included. The project itself has been around 2004 and is well maintained.

Anyway, let's install it. The below will assume Ubuntu 14.04, which has a recent (>= 3.0) version of Supervisor. If you're running an older version of Ubuntu, or an OS that doesn't package a recent version of Supervisor, it may be worth installing it via pip and writing your own Upstart/systemd service file.

$ sudo apt-get install supervisor

Now, we also want our application user to be able to invoke supervisorctl (the management interface) as necessary, so we'll need to create a supervisor group, make our user a member of that group and modify Supervisor's configuration file to give the supervisor group the correct permissions on the socket.

$ sudo addgroup --system supervisor
# i.e. 'sudo adduser deploy supervisor'
$ sudo adduser <yourappuser> supervisor
$ logout
# Log back in and confirm which should now list 'supervisor':
$ groups

That's the group taken care of. Let's modify the Supervisor configuration file to take this into account:

[unix_http_server]
file=/var/run/supervisor.sock   
chmod=0770                       # ensure our group has read/write privs
chown=root:supervisor            # add our group

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = /etc/supervisor/conf.d/*.conf # default location on Ubuntu

And now we'll restart Supervisor:

$ sudo service supervisor restart

If it doesn't restart, check the log with the below:

$ sudo tail /var/log/supervisor/supervisord.log

Typos are the usual culprit here. Otherwise, with the core configuration out of the way, let's create a configuration for our Go app.

Configuring It

Supervisor is infinitely configurable, but we'll aim to keep things simple. Note that you will need to modify the configuration below to suit your application: I've commented the lines you'll need to change.

Create a configuration file at the default (Ubuntu) includes directory:

# where 'mygoapp' is the name of your application
$ sudo vim /etc/supervisor/conf.d/mygoapp.conf 

... and pull in the below:

[program:yourapp]
command=/home/yourappuser/bin/yourapp # the location of your app
autostart=true
autorestart=true
startretries=10
user=yourappuser # the user your app should run as (i.e. *not* root!)
directory=/srv/www/yourapp.com/ # where your application runs from
environment=APP_SETTINGS="/srv/www/yourapp.com/prod.toml" # environmental variables
redirect_stderr=true
stdout_logfile=/var/log/supervisor/yourapp.log # the name of the log file.
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

Let's step through it:

Now, let's reload Supervisor so it picks up our app's config file, and check that it's running as expected:

$ supervisorctl reload
$ supervisorctl status yourapp

We should see a "running/started" message and our application should be ready to go. If not, check the logs in /var/log/supervisor/supervisord.log or run supervisorctl tail yourapp to show our application logs. A quick Google for the error message will go a long way if you get stuck.

Fedora/CentOS/RHEL

If you're running CentOS 7 or Fedora 20, the directory layout is a little different than Ubuntu's (rather, Ubuntu has a non-standard location), so keep that in mind. Specifically:

Otherwise, Supervisor is much the same: you'll need to install it, create a system group, add your user to the group, and then update the config file and restart the service using sudo systemctl restart supervisord.

Summary

Pretty easy, huh? If you're using a configuration management tool (i.e. Ansible, Salt, et. al) for your production machines, then it's easy to automate this completely, and I definitely recommend doing so. Being able to recreate your production environment like-for-like after a failure (or moving hosts, or just for testing) is a Big Deal and worth the time investment.

It's also easy to see from this guide how easy it is to add more Go applications to Supervisor's stable: add a new configuration file, reload Supervisor, and off you go. You can choose how aggressive restarts need to be, log rotations and environmental variables on a per-application basis, which is always useful.

HTTP Request Contexts & Go

 ∗ go, web, go

Alternatively titled map[string]interface.

Request contexts, for those new to the terminology, are typically a way to pass data alongside a HTTP request as it is processed by composable handlers (or middleware). This data could be a user ID, a CSRF token, whether a user is logged in or not—something typically derived from logic that you don't want to repeat over-and-over again in every handler. If you've ever used Django, the request context is synonymous with the request.META dictionary.

As an example:

func CSRFMiddleware(http.Handler) http.Handler {
    return func(w http.ResponseWriter, r *http.Request) {
        maskedToken, err := csrf.GenerateNewToken(r)
        if err != nil {
            http.Error(w, "No good!", http.StatusInternalServerError)
            return
        }

        // How do we pass the maskedToken from here...
    }
}

func MyHandler(w http.ResponseWriter, r *http.Request) {
    // ... to here, without relying on the overhead of a session store,
    // and without either handler being explicitly tied to the other?
    // What about a CSRF token? Or an auth-key from a request header?
    // We certainly don't want to re-write that logic in every handler!
}

There's three ways that Go's web libraries/frameworks have attacked the problem of request contexts:

  1. A global map, with *http.Request as the key, mutexes to synchronise writes, and middleware to cleanup old requests (gorilla/context)

  2. A strictly per-request map by creating custom handler types (goji)

  3. Structs, and creating middleware as methods with pointer receivers or passing the struct to your handlers (gocraft/web).

So how do these approaches differ, what are the benefits, and what are the downsides?

Global Context Map

gorilla/context's approach is the simplest, and the easiest to plug into an existing architecture.

Gorilla actually uses a map[interface{}]interface{}, which means you need to (and should) create types for your keys. The benefit is that you can use any types that support equality as a key; the downside is that you need to implement your keys in advance if you want to avoid any run-time issues with key types.

You also often want to create setters for the types you store in the context map, to avoid littering your handlers with the same type assertions.

import (
    "net/http"
    "github.com/gorilla/context"
)

type contextKey int

// Define keys that support equality.
const csrfKey contextKey = 0
const userKey contextKey = 1

var ErrCSRFTokenNotPresent = errors.New("CSRF token not present in the request context.")

// We'll need a helper function like this for every key:type
// combination we store in our context map else we repeat this
// in every middleware/handler that needs to access the value.
func GetCSRFToken(r *http.Request) (string, error) {
    val, ok := context.GetOk(r, csrfKey)
    if !ok {
        return "", ErrCSRFTokenNotPresent
    }

    token, ok := val.(string)
    if !ok {
        return "", ErrCSRFTokenNotPresent
    }

    return token, nil
}

// A bare-bones example
func CSRFMiddleware(h http.Handler) http.Handler {
    return func(w http.ResponseWriter, r *http.Request) {
        token, err := GetCSRFToken(r)
        if err != nil {
            http.Error(w, "No good!", http.StatusInternalServerError)
            return
        }

        // The map is global, so we just call the Set function
        context.Set(r, csrfKey, token)

        h.ServeHTTP(w, r)
    }
}

func ShowSignupForm(w http.ResponseWriter, r *http.Request) {
    // We'll use our helper function so we don't have to type assert
    // the result in every handler that triggers/handles a POST request.
    csrfToken, err := GetCSRFToken(r)
    if err != nil {
        http.Error(w, "No good!", http.StatusInternalServerError)
        return
    }

    // We can access this token in every handler we wrap with our
    // middleware. No need to set/get from a session multiple times per
    // request (which is slow!)
    fmt.Fprintf(w, "Our token is %v", csrfToken)
}


func main() {
    r := http.NewServeMux()
    r.Handle("/signup", CSRFMiddleware(http.HandlerFunc(ShowSignupForm)))
    // Critical that we call context.ClearHandler here, else
    // we leave old requests in the map.
    http.ListenAndServe("localhost:8000", context.ClearHandler(r))
}

Full Example

The plusses? It's flexible, loosely coupled, and easy for third party packages to use. You can tie it into almost any net/http application since all you need is access to http.Request—the rest relies on the global map.

The downsides? The global map and its mutexes may result in contention at high loads, and you need to call context.Clear() at the end of every request (i.e. on each handler). Forget to do that (or wrap your top-level server handler) and you'll open yourself up to a memory leak where old requests remain in the map. If you're writing middleware that uses gorilla/context, then you need to make sure your package user imports context calls context.ClearHandler on their handlers/router.

Per Request map[string]interface

As another take, Goji provides a request context as part of an (optional) handler type that embeds Go's usual http.Handler. Because it's tied to Goji's (fast) router implementation, it no longer needs to be a global map and avoids the need for mutexes.

Goji provides a web.HandlerFunc type that extends the default http.HandlerFunc with a request context: func(c web.C, w http.ResponseWriter, r *http.Request).

var ErrTypeNotPresent = errors.New("Expected type not present in the request context.")

// A little simpler: we just need this for every *type* we store.
func GetContextString(c web.C, key string) (string, error) {
    val, ok := c.Env[key].(string)
    if !ok {
        return "", ErrTypeNotPresent
    }

    return val, nil
}

// A bare-bones example
func CSRFMiddleware(c *web.C, h http.Handler) http.Handler {
    fn := func(w http.ResponseWriter, r *http.Request) {
        maskedToken, err := GenerateToken(r)
        if err != nil {
            http.Error(w, "No good!", http.StatusInternalServerError)
            return
        }

        // Goji only allocates a map when you ask for it.
        if c.Env == nil {
            c.Env = make(map[string]interface{})
        }

        // Not a global - a reference to the context map
        // is passed to our handlers explicitly.
        c.Env["csrf_token"] = maskedToken

        h.ServeHTTP(w, r)
    }

    return http.HandlerFunc(fn)
}

// Goji's web.HandlerFunc type is an extension of net/http's
// http.HandlerFunc, except it also passes in a request
// context (aka web.C.Env)
func ShowSignupForm(c web.C, w http.ResponseWriter, r *http.Request) {
    // We'll use our helper function so we don't have to type assert
    // the result in every handler.
    csrfToken, err := GetContextString(c, "csrf_token")
    if err != nil {
        http.Error(w, "No good!", http.StatusInternalServerError)
        return
    }

    // We can access this token in every handler we wrap with our
    // middleware. No need to set/get from a session multiple times per
    // request (which is slow!)
    fmt.Fprintf(w, "Our token is %v", csrfToken)
}

Full Example

The biggest immediate gain is the performance improvement, since Goji only allocates a map when you ask it to: there's no global map with locks. Note that for many applications, your database or template rendering will be the bottleneck (by far), so the "real" impact is likely pretty small, but it's a sensible touch.

Most useful is that you retain the ability to write modular middleware that doesn't need further information about your application: if you want to use the request context, you can do so, but for anything else it's just http.Handler. The downside is that you still need to type assert anything you retrieve from the context, although like gorilla/context we can simplify this by writing helper functions. A map[string]interface{} also restricts us to string keys: simpler for most (myself included), but potentially less flexible for some.

Context Structs

A third approach is to initialise a struct per-request and define our middleware/handler as methods on the struct. The big plus here is type-safety: we explicitly define the fields of our request context, and so we know the type (unless we do something naive like setting a field to interface{}).

Of course, what you gain in type safety you lose in flexibility. You can't create "modular" middleware that uses the popular func(http.Handler) http.Handler pattern, because that middleware can't know what your request context struct looks like. It could provide it's own struct that you embed into yours, but that still doesn't solve re-use: not ideal. Still, it's a good approach: no need to type assert things out of interface{}.

import (
    "fmt"
    "log"
    "net/http"

    "github.com/gocraft/web"
)

type Context struct {
    CSRFToken string
    User      string
}

// Our middleware *and* handlers must be defined as methods on our context struct,
// or accept the type as their first argument. This ties handlers/middlewares to our
// particular application structure/design.
func (c *Context) CSRFMiddleware(w web.ResponseWriter, r *web.Request, next web.NextMiddlewareFunc) {
    token, err := GenerateToken(r)
    if err != nil {
        http.Error(w, "No good!", http.StatusInternalServerError)
        return
    }

    c.CSRFToken = token
    next(w, r)
}

func (c *Context) ShowSignupForm(w web.ResponseWriter, r *web.Request) {
    // No need to type assert it: we know the type.
    // We can just use the value directly.
    fmt.Fprintf(w, "Our token is %v", c.CSRFToken)
}

func main() {
    router := web.New(Context{}).Middleware((*Context).CSRFMiddleware)
    router.Get("/signup", (*Context).ShowSignupForm)

    err := http.ListenAndServe(":8000", router)
    if err != nil {
        log.Fatal(err)
    }
}

Full Example

The plus here is obvious: no type assertions! We have a struct with concrete types that we initialise on every request and pass to our middleware/handlers. But the downside is that we can no longer "plug and play" middleware from the community, because it's not defined on our own context struct.

We could anonymously embed their type into ours, but that starts to become pretty messy and doesn't help if their fields share the same names as our own. The real solution is to fork and modify the code to accept your struct, at the cost of time/effort. gocraft/web also wraps the ResponseWriter interface/Request struct with its own types, which ties things a little more closely to the framework itself.

How Else?

One suggestion would be to provide a Context field on Go's http.Request struct, but actually implementing it in a "sane" way that suits the common use case is easier said than done.

The field would likely end up being a map[string]interface{} (or with interface{} as the key). This means that we either need to initialise the map for users—which won't be useful on all of those requests where you don't need to use the request context. Or require package users to check that the map is initialised before using it, which can be a big "gotcha" for newbies who will wonder (at first) why their application panics on some requests but not others.

I don't think these are huge barriers unto themselves, but Go's strong preference for being clear and understandable—at the cost of a little verbosity now and then—is potentially at odds with this approach. I also don't believe that having options in the form of third-party packages/frameworks is a Bad Thing either: you choose the approach that best fits your idioms or requirements.

Wrap

So which approach should you choose for your own projects? It's going to depend on your use-case (as always). Writing a standalone package and want to provide a request context that the package user can easily access? gorilla/context is probably going to be a good fit (just document the need to call ClearHandler!). Writing something from scratch, or have a net/http app you want to extend easily? Goji is easy to drop in. Starting from nothing? gocraft/web's "inclusive" approach might fit.

Personally, I like Goji's approach: I don't mind writing a couple of helpers to type-assert things I commonly store in a request context (CSRF tokens, usernames, etc), and I get to avoid the global map. It's also easy for me to write middleware that others can plug into their applications (and for me to use theirs). But those are my use cases, so do your research first!

Custom Handlers and Avoiding Globals in Go Web Applications

 ∗ go, web, go

Go's net/http package is extremely flexible, thanks to the fact that it centres on the http.Handler interface. Building around an interface gives you the option of both extending the included implementation and keeping it compatible with other packages out in the wild. Given that the default implementation is pretty simple, we'll look at how we can build our own handler type (to remove error handling repetition), and how to extend it so we can explicitly pass a "context" containing our database pool, template map, a custom logger and so on, letting us remove any reliance on global variables.

Creating Our Custom Handler Type

net/http provides a basic HandlerFunc type that is just func(w http.ResponseWriter, r *http.Request). It's easy to understand, pervasive, and covers most simple use-cases. But for anything more than that, there's two immediate "issues": a) we can't pass any additional parameters to http.HandlerFunc, and b) we have to repeat a lot of error handling code in each handler. If you're new to Go it may not seem immediately obvious how to resolve this but still retain compatibility with other HTTP packages, but thankfully it's an easy problem to solve.

We create our own handler type that satisfies http.Handler (read: it has a ServeHTTP(http.ResponseWriter, *http.Request) method), which allows it to remain compatible with net/http, generic HTTP middleware packages like nosurf, and routers/frameworks like gorilla/mux or Goji.

First, let's highlight the problem:

func myHandler(w http.ResponseWriter, r *http.Request) {
    session, err := store.Get(r, "myapp")
    if err != nil {
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return // Forget to return, and the handler will continue on
    }

    id := // get id from URL param; strconv.AtoI it; making sure to return on those errors too...
    post := Post{ID: id}
    exists, err := db.GetPost(&post)
    if err != nil {
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return // Repeating ourselves again 
    }

    if !exists {
        http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
        return // ... and again.
    }

    err = renderTemplate(w, "post.tmpl", post)
    if err != nil {
        // Yep, here too...
    }
}

Things are not only verbose (we have to do this in every handler), but we're at the risk of a subtle and hard-to-catch bug. If we don't explicitly return when we encounter an error—such as a serious database error or when a password comparison fails—our handler will continue. At best this might mean we render an empty struct to our template and confuse the user. At worst, this might mean we write a HTTP 401 (Not Authorised) response and then continue to do things that (potentially) only a logged in user should see or be able to do.

Thankfully, we can fix this pretty easily by creating a handler type that returns an explicit error:

type appHandler func(http.ResponseWriter, *http.Request) (int, error)

// Our appHandler type will now satisify http.Handler 
func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if status, err := fn(w, r); err != nil {
        // We could also log our errors centrally:
        // i.e. log.Printf("HTTP %d: %v", err)
        switch status {
        // We can have cases as granular as we like, if we wanted to
        // return custom errors for specific status codes.
        case http.StatusNotFound:
            notFound(w, r)
        case http.StatusInternalServerError:
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        default:
            // Catch any other errors we haven't explicitly handled
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        }
}

func myHandler(w http.ResponseWriter, r *http.Request) (int, error) {
    session, err := store.Get(r, "myapp")
    if err != nil {
        // Much better!
        return http.StatusInternalServerError, err
    }

    post := Post{ID: id}
    exists, err := db.GetPost(&post)
    if err != nil {
        return http.StatusInternalServerError, err
    }

    // We can shortcut this: since renderTemplate returns `error`,
    // our ServeHTTP method will return a HTTP 500 instead and won't 
    // attempt to write a broken template out with a HTTP 200 status.
    // (see the postscript for how renderTemplate is implemented)
    // If it doesn't return an error, things will go as planned.
    return http.StatusOK, renderTemplate(w, "post.tmpl", data)
}

func main() {
    // Cast myHandler to an appHandler
    http.Handle("/", appHandler(myHandler))
    http.ListenAndServe(":8000", nil)
}

This is, of course, nothing new: Andrew Gerrand highlighted a similar approach on the Go blog back in 2011. Our implementation is just an adaptation with a little extra error handling. I prefer to return (int, error) as I find it more idiomatic than returning a concrete type, but you could certainly create your own error type if you wished (but let's just keep it simple for now).

Extending Our Custom Handler Further

A quick aside: global variables get a lot of hate: you don't control what can modify them, it can be tricky to track their state, and they may not be suitable for concurrent access. Still, used correctly they can be convenient, and plenty of Go docs & projects lean on them (e.g. here & here). database/sql's *sql.DB type can be safely used as a global as it represents a pool and is protected by mutexes, maps (i.e. template maps) can be read from (but not written to, of course) concurrently, and session stores take a similar approach to database/sql.

After being inspired by @benbjohnson's article from last week on structuring Go applications and a debate with a fellow Gopher on Reddit (who takes a similar approach), I figured I'd take a look at my codebase (which has a few globals of the above types) and refactor it to explicitly pass a context struct to my handlers. Most of it was smooth sailing, but there's a couple of potential pitfalls you can run into if you want your context instance to be available in more than just the handlers themselves.

Here's the actual global variables I had before:

var (
    decoder   *schema.Decoder
    bufpool   *bpool.Bufferpool
    templates map[string]*template.Template
    db        *sqlx.DB
    store     *redistore.RediStore
    mandrill  *gochimp.MandrillAPI
    twitter   *anaconda.TwitterApi
    log       *log.Logger
    conf      *config // app-wide configuration: hostname, ports, etc.
)

So, given the custom handler type we created above, how can we turn this list of global variables into a context we can pass to our handlers and our ServeHTTP method—which may want to access our template map to render "pretty" errors or our custom logger—and still keep everything compatible with http.Handler?

package main

import (
    "fmt"
    "log"
    "net/http"

    "html/template"

    "github.com/gorilla/sessions"
    "github.com/jmoiron/sqlx"
    "github.com/zenazn/goji/graceful"
    "github.com/zenazn/goji/web"
)

// appContext contains our local context; our database pool, session store, template
// registry and anything else our handlers need to access. We'll create an instance of it
// in our main() function and then explicitly pass a reference to it for our handlers to access.
type appContext struct {
    db        *sqlx.DB
    store     *sessions.CookieStore
    templates map[string]*template.Template
    decoder *schema.Decoder
    // ... and the rest of our globals.
}

// We've turned our original appHandler into a struct with two fields:
// - A function type similar to our original handler type (but that now takes an *appContext)
// - An embedded field of type *appContext
type appHandler struct {
    *appContext
    h func(*appContext, http.ResponseWriter, *http.Request) (int, error)
}

// Our ServeHTTP method is mostly the same, and also has the ability to
// access our *appContext's fields (templates, loggers, etc.) as well.
func (ah appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // Updated to pass ah.appContext as a parameter to our handler type.
    status, err := ah.h(ah.appContext, w, r)
    if err != nil {
        log.Printf("HTTP %d: %q", status, err)
        switch status {
        case http.StatusNotFound:
            http.NotFound(w, r)
            // And if we wanted a friendlier error page, we can
            // now leverage our context instance - e.g.
            // err := ah.renderTemplate(w, "http_404.tmpl", nil)
        case http.StatusInternalServerError:
            http.Error(w, http.StatusText(status), status)
        default:
            http.Error(w, http.StatusText(status), status)
        }
    }
}

func main() {
    // These are 'nil' for our example, but we'd either assign
    // the values as below or use a constructor function like
    // (NewAppContext(conf config) *appContext) that initialises
    // it for us based on our application's configuration file.
    context := &appContext{db: nil, store: nil} // Simplified for this example

    r := web.New()
    // We pass an instance to our context pointer, and our handler.
    r.Get("/", appHandler{context, IndexHandler})

    graceful.ListenAndServe(":8000", r)
}

func IndexHandler(a *appContext, w http.ResponseWriter, r *http.Request) (int, error) {
    // Our handlers now have access to the members of our context struct.
    // e.g. we can call methods on our DB type via err := a.db.GetPosts()
    fmt.Fprintf(w, "IndexHandler: db is %q and store is %q", a.db, a.store)
    return 200, nil
}

Everything still remains very readable: we lean on the type system and existing interfaces, and if we just want to use a regular http.HandlerFunc, we can do that too. Our handlers are still wrappable by anything that takes (and spits out) a http.Handler, and if we wanted to ditch Goji and use gorilla/mux or even just net/http, we don't have to change our handler at all. Just make sure that your context's fields are safe for concurrent access. Putting a map in there that requests write to would not be, for example: you'll need a mutex from the sync package for that.

Other than that, it just works. We've reduced repetition around our error handling, we've removed our reliance on globals and our code is still readable.

Addendum

Approximating html/template Inheritance

 ∗ go, web, go

Go's html/template package is fairly minimal compared to templating packages associated with other languages (Jinja, Mustache, even Django's templates), although it makes up for this with security and great docs.

There are however a few "tricks" to using it: specifically when it comes to approximating template inheritance. Being able to specify a base layout (or layouts), stake out your blocks and then fill those blocks with template snippets isn't immediately clear. So how do we do this?

First, we define base.tmpl:

{{ define "base" }}
<html>
<head>
    {{ template "title" . }}
</head>
<body>
    {{ template "scripts" . }}
    {{ template "sidebar" . }}
    {{ template "content" . }}
<footer>
    ...
</footer>
</body>
</html>
{{ end }}
// We define empty blocks for optional content so we don't have to define a block in child templates that don't need them
{{ define "scripts" }}{{ end }}
{{ define "sidebar" }}{{ end }}

And index.tmpl, which effectively extends our base template.

{{ define "title"}}<title>Index Page</title>{{ end }}
// Notice the lack of the script block - we don't need it here.
{{ define "sidebar" }}
    // We have a two part sidebar that changes depending on the page
    {{ template "sidebar_index" }} 
    {{ template "sidebar_base" }}
{{ end }}
{{ define "content" }}
    {{ template "listings_table" . }}
{{ end }}

Note that we don't need to define all blocks in the base layout: we've "cheated" a little by defining them alongside our base template. The trick is ensure that the {{ define }} blocks in the base template are empty. If you define two blocks and both have content, the application will panic when it attempts to parse the template files (on startup, most likely). There's no "default" content we can fall back on. It's not a a deal-breaker, but it's worth remembering when writing these out.

In our Go application, we create a map of templates by parsing the base template, any necessary snippets, and the template that extends our base template. This is best done at appication start-up (and panics are okay here) so we can fail early. A web application with broken templates is probably not much of a web application.

It's also critical that we ensure any look-ups on map keys (template names) that don't exist are caught (using the comma-ok idiom): otherwise it's a run-time panic.

import (
    "fmt"
    "html/template"
    "net/http"
    "path/filepath"
)

var templates map[string]*template.Template

// Load templates on program initialisation
func init() {
    if templates == nil {
        templates = make(map[string]*template.Template)
    }

    templatesDir := config.Templates.Path

    layouts, err := filepath.Glob(templatesDir + "layouts/*.tmpl")
    if err != nil {
        log.Fatal(err)
    }

    includes, err := filepath.Glob(templatesDir + "includes/*.tmpl")
    if err != nil {
        log.Fatal(err)
    }

    // Generate our templates map from our layouts/ and includes/ directories
    for _, layout := range layouts {
        files := append(includes, layout)
        templates[filepath.Base(layout)] = template.Must(template.ParseFiles(files...))
    }

}

// renderTemplate is a wrapper around template.ExecuteTemplate.
func renderTemplate(w http.ResponseWriter, name string, data map[string]interface{}) error {
    // Ensure the template exists in the map.
    tmpl, ok := templates[name]
    if !ok {
        return fmt.Errorf("The template %s does not exist.", name)
    }

    w.Header().Set("Content-Type", "text/html; charset=utf-8")
    tmpl.ExecuteTemplate(w, "base", data)

    return nil
}

We create our templates from a set of template snippets and the base layout (just the one, in our case). We can fill in our {{ template "script" }} block as needed, and we can mix and match our sidebar content as well. If your pages are alike, you can generate this map with a range clause by using a slice of the template names as the keys.

Slightly tangential to this, there's the common problem of dealing with the error returned from template.ExecuteTemplate. If we pass the writer to an error handler, it's too late: we've already written (partially) to the response and we'll end up with a mess in the user's browser. It'll be part of the page before it hit the error, and then the error page's content. The solution here is to write to a bytes.Buffer to catch any errors during the template rendering, and then write out the contents of the buffer to the http.ResponseWriter.

Although you can create your own buffer per-request, using a pool (https://github.com/oxtoacart/bpool) reduces allocations and garbage. I benchmarked and profiled a bare approach (as above; write out directly), a 10K fixed buffer per-request (big enough for most of my responses), and a pool of buffers. The pooled approach was the fastest, at 32k req/s vs. 26k req/s (fixed buffer) and 29k req/s (no buffer). Latency was no worse than the bare approach either, which is a huge plus.

import (
    "fmt"
    "html/template"
    "net/http"

    "github.com/oxtoacart/bpool"
)

var bufpool *bpool.BufferPool

// renderTemplate is a wrapper around template.ExecuteTemplate.
// It writes into a bytes.Buffer before writing to the http.ResponseWriter to catch
// any errors resulting from populating the template.
func renderTemplate(w http.ResponseWriter, name string, data map[string]interface{}) error {
    // Ensure the template exists in the map.
    tmpl, ok := templates[name]
    if !ok {
        return fmt.Errorf("The template %s does not exist.", name)
    }

    // Create a buffer to temporarily write to and check if any errors were encounted.
    buf := bufpool.Get()
    defer bufpool.Put()

    err := tmpl.ExecuteTemplate(buf, "base", data)
    if err != nil {
        return err
    }

    // Set the header and write the buffer to the http.ResponseWriter
    w.Header().Set("Content-Type", "text/html; charset=utf-8")
    buf.WriteTo(w)
    return nil
}

func init() {
    ...
    bufpool = bpool.NewBufferPool(64)
    ...
}

We can catch that returned error in our handler and return a HTTP 500 instead. The best part is that it also makes testing our handlers easier. If you try to take over the http.ResponseWriter with your error handler, you've already sent a HTTP 200 status header, making it much harder to test where things are broken. By writing to a temporary buffer first, we ensure that don't set headers until we're sure the template will render correctly; making testing much simpler.

And that's about it. We have composable templates, we deal with our errors before writing out, and it's still fast.

Postscript

Tamagotchi Hive

 ∗ xkcd.com

The Singularity happened, but not to us.

Infocon: green

 ∗ SANS Internet Storm Center, InfoCON: green

Analyzing Quarantine Files

Analyzing Quarantine Files, (Fri, Jul 3rd)

 ∗ SANS Internet Storm Center, InfoCON: green

Quarantine files are produced by anti-virus programs. When an anti-virus detects a file (a positi ...(more)...

Rachel Andrew on the Business of Web Dev: Software Audits for the Tiny Business

 ∗ A List Apart: The Full Feed

In the dot com boom, I headed up a technical team. One of my responsibilities was the hardware and software needed by our team and other teams in the company. Back then, software was typically boxed and came with packs of licenses. I had an internal software audit routine to check that all of our computers were correctly licensed.

In my own business, I’m part of a two-person team. Many of our software licenses are purchased far less formally and are for one user only. We also rely on a huge amount of Software as a Service, where there is no box or permanent license to use the service. As owner-managers, we sometimes buy something using a personal email address or bank card, despite it being for company use. Also, we contract third parties to do work for us, and they purchase software or create accounts on our behalf. With no physical boxes, or volume licenses to keep track of, small companies now develop websites and software in a far more informal way.

It’s a common pattern we see at Perch: the company contacting us has had a website developed, and sometime later the person who developed the site and bought the software license leaves the company. The company has no idea how to access the license information, and password reset attempts go to a now-dead email address. Sometimes the account is linked to a personal email address, or was even bought by a third party who had been contracted to build the site.

With accounts spread around between owners, employees, and contractors, businesses can easily leave themselves exposed. Services get canceled because no one saw the reminder that a credit card had expired. Domains fail to auto-renew, leaving the company in danger of losing them. Software purchased via an employee or contractor becomes inaccessible due to that person leaving. Therefore, I propose that every business needs a software and services audit. Collate all of this information, then make sure it is accessible to you and other trusted team members.

What should we audit?

What should you take a look at? To get you started, I’ve listed a few key areas that most web agencies and software businesses will recognize. Once you start this process, you will no doubt think of more. (If you think of something other business owners might forget, add it to the comments.)

Domains and DNS

Does the company own all of the domains you use, or are any linked to personal accounts?

Can you move all of the domains and DNS management to one location? Services such as DNSimple offer domain hosting and DNS management, and can be a better choice than having domains registered alongside your webhosting.

Do you have a solid way to track expiration dates? Are domains set to auto-renew with up-to-date payment details? Do all of the domains have correct contact details?

Email

How many email accounts are being used in your business? Are you, or your employees, using personal addresses for company business?

Is important and valuable information stored in your email, or in the email of employees? For example, if you work with clients by email, who has the history of the conversations?

Is email securely backed up?

Software used on your websites

Many websites rely on paid software-whether that is a full CMS like Perch, or paid add-ons to WordPress. Who owns those licenses?

Which email address is linked to these licenses? The developer might use it to let you know of important security updates to the software.

Do you have access details for any account on the third-party website?

Do you know where and how to get support for any of the components of your website that haven’t been developed in-house?

If you develop websites for clients, you may be able to help them to manage this better by providing a handover pack on payment that includes all of the key details and clarifies their significance.

Recurring payments

Go through your bank and card statements. What does your business pay for, monthly or yearly? Cancel anything you don’t use.

For services you do use, check if you can save money and bookkeeping by switching to an annual plan.

Check that the plan you are on is up to date and still the right one for you. SaaS companies often introduce new levels of service—you might save money or get new functionality by switching.

Keep a list or spreadsheet of services and the date on which you last checked them. You can repeat this process every six months, to make sure you don’t pay for things that you are not using.

Start with one small thing

Auditing everything you use in your business is likely to save you time and stress in the long run, but with our limited time it’s an area we can find easy to avoid. You don’t need to do this all at once. Next time you have a couple of spare hours, pick off one area and start there.

In a couple of hours you could list all of your domains, check when they expire and make sure that you can access the registration for each. You may need to add another to do to your list to consolidate them at one provider, or to update contact details. However, you have moved forward just by knowing what the status of your domains is. You are already less likely to have the nasty surprise of waking up to find a website offline due to the domain expiring.

Let me know in the comments if this is an area you struggle with; or if you have found good ways to store, maintain, and share this kind of information.

GopherChina Trip Report

 ∗ The Go Programming Language Blog

We have known for some time that Go is more popular in China than in any other country. According to Google Trends, most searches for the term “golang” come from The People’s Republic than anywhere else. Others have speculated on the same observation, yet so far we have had sparse concrete information about the phenomenon.

The first Go conference in China, GopherChina, seemed like an excellent opportunity to explore the situation by putting some Western Gopher feet on Chinese ground. An actual invitation made it real and I decided to accept and give a presentation about gofmt’s impact on software development.

Hello, Shanghai!

The conference took place over an April weekend in Shanghai, in the Puruan Building of the Shanghai Pudong Software Park, easily reachable by subway within an hour or less from Shanghai’s more central parts. Modelled after GopherCon, the conference was single-track, with all talks presented in a conference room that fit about 400 attendees. It was organized by volunteers, lead by Asta Xie, and with robust sponsorship from major industry names. According to the organizers, many more people were hoping to attend than could be accommodated due to space constraints.

The welcoming committee with Asta Xie (2nd from left), the primary organizer.

Each attendee received a bag filled with the obligatory GopherChina t-shirt, various sponsor-related informational brochures, stickers, and the occasional stuffed “something” (no fluffy Gophers, though). At least one 3rd party vendor was advertising technical books, including several original (not translated from English) Go books.

Go books!

On first impression, the average attendee seemed pretty young, which made for an enthusiastic crowd, and the event appeared well run.

With the exception of my talk, all presentations were given in Mandarin and thus were incomprehensible to me. Asta Xie, the primary organizer, assisted with a few simultaneous translations whispered into my ear, and the occasional English slide provided additional clues: “69GB” stands out even without any Mandarin knowledge (more on that below). Consequently, I ended up listening to a handful of presentations only, and instead spent much of my time talking with attendees outside the main conference room. Yet judging from the slides, the quality of most presentations seemed high, comparable with our experience at GopherCon in Denver last year. Each talk got a one hour time slot which allowed for plenty of technical detail, and many (dozens) of questions from an enthusiastic audience.

As expected, many of the presentations were about web services, backends for mobile applications, and so on. Some of the systems appear to be huge by any measure. For instance, a talk by Yang Zhou described a large-scale internal messaging system, used by Qihoo 360, a major Chinese software firm, all written in Go. The presentation discussed how his team managed to reduce an original heap size of 69GB (!) and the resulting long GC pauses of 3-6s to more manageable numbers, and how they run millions of goroutines per machine, on a fleet of thousands of machines. A future guest blog post is planned describing this system in more detail.

Packed conference room on Saturday.

In another presentation, Feng Guo from DaoCloud talked about how they use Go in their company for what they call the “continuous delivery” of applications. DaoCloud takes care of automatically moving software hosted on GitHub (and Chinese equivalents) to the cloud. A software developer simply pushes a new version on GitHub and DaoCloud takes care of the rest: running tests, Dockerizing it, and shipping it using your preferred cloud service provider.

Several speakers were from well-recognized major software firms (I showed the conference program to non-technical people and they easily recognized several of the firm’s names). Much more so than in the US, it seems Go is not just hugely popular with newcomers and startups, but has very much found its way into larger organizations and is employed at a scale that we are only starting to see elsewhere.

Not being an expert in web services myself, in my presentation I veered off the general conference theme a bit by talking about gofmt and how its widespread use has started to shape expectations not just for Go but other languages as well. I presented in English but had my slides translated to Mandarin beforehand. Due to the significant language barrier I wasn’t expecting too many questions on my talk itself. Instead I decided the keep it short and leave plenty of time for general questions on Go, which the audience appreciated.

No social event in China is complete without fantastic food.

A couple of days after the conference I visited the 4-year-old startup company Qiniu (“Seven Bulls”), at the invitation of its CEO Wei Hsu, facilitated and translated with the help of Asta Xie. Qiniu is a cloud-based storage provider for mobile applications; Wei Hsu presented at the conference and also happens to be the author of one of the first Chinese books on Go (the leftmost one in the picture above).

Qiniu lobby, engineering.

Qiniu is an extremely successful all-Go shop, with about 160 employees, serving over 150,000 companies and developers, storing over 50 Billion files, and growing by over 500 Million files per day. When asked about the reasons for Go’s success in China, Wei Hsu is quick to answer: PHP is extremely popular in China, but relatively slow and not well-suited for large systems. Like in the US, universities teach C++ and Java as primary languages, but for many applications C++ is too complex a tool and Java too bulky. In his opinion, Go now plays the role that traditionally belonged to PHP, but Go runs much faster, is type safe, and scales more easily. He loves the fact that Go is simple and applications are easy to deploy. He thought the language to be “perfect” for them and his primary request was for a recommended or even standardized package to easily access database systems. He did mention that they had GC problems in the past but were able to work around them. Hopefully our upcoming 1.5 release will address this. For Qiniu, Go appeared just at the right time and the right (open source) place.

According to Asta Xie, Qiniu is just one of many Go shops in the PRC. Large companies such as Alibaba, Baidu, Tencent, and Weibo, are now all using Go in one form or another. He pointed out that while Shanghai and neighboring cities like Suzhou are high-tech centres, even more software developers are found in the Beijing area. For 2016,  Asta hopes to organize a larger (1000, perhaps 1500 people) successor conference in Beijing.

It appears that we have found the Go users in China: They are everywhere!

Some of the GopherChina materials, including videos, are now available alongside Go coursework on a 3rd party site.

ISC StormCast for Thursday, July 2nd 2015 http://isc.sans.edu/podcastdetail.html?id=4553, (Thu, Jul 2nd)

 ∗ SANS Internet Storm Center, InfoCON: green

...(more)...

Another example of Angler exploit kit pushing CryptoWall 3.0, (Thu, Jul 2nd)

 ∗ SANS Internet Storm Center, InfoCON: green

Introduction

Angler exploit kit (EK) has been evolving quite ...(more)...

Patch for Default Account Credentials in Cisco Unified Communications Domain Manager http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20150701-cucdm, (Wed, Jul 1st)

 ∗ SANS Internet Storm Center, InfoCON: green

---
Johannes B. Ullrich, Ph ...(more)...

Strengths and Weaknesses

 ∗ xkcd.com

Do you need me to do a quicksort on the whiteboard or produce a generation of offspring or something? It might take me a bit, but I can do it.

This week's sponsor: TeamGantt

 ∗ A List Apart: The Full Feed

Our sponsor TeamGantt wants to help you organize your projects better. Check out their 30-day free trial to get started.

Container Queries: Once More Unto the Breach

 ∗ A List Apart: The Full Feed

We’re a perseverant crew, the RICG. Responsive images were dead and buried forever in particularly dramatic fashion—what—two, three times there? “Dead and buried forever” refused to stick, though, thanks to all of you. Every time responsive images were knocked down, they got up a little stronger than before.

It isn’t hard to see why the going was so tough. Changing up a feature as old—and in software terms, as stable—as the img element was no small request. But designers and developers had spoken: we wanted a feature that stood to save our users a tremendous amount of bandwidth, and we carved out a seat at the same table as browser representatives and long-established standards bodies to make that feature happen. It was a long and failure-riddled process, but it’s what got us to the solutions we have today: not just one new element, but a whole suite of enhancements to the img element as well. Now we have options for smarter asset delivery based on any combination of viewport size, pixel density, and file format support.

After all that hard-won progress, there was no sense in packing up our GitHub organization and going home. We’ve changed the “I” in “RICG” from “Images” to “Issues,” and set our sights high right out of the gate: we aim to change the way we all write CSS.

The trouble with media queries

Styling a module meant to occupy multiple containers with viewport-based media queries means scoping that module’s styles to all the containers that it might occupy. That sentence isn’t the easiest thing in the world to parse, but it’s something I’m betting we’re all familiar with in practice.

The ideal responsive website is a system of flexible, modular components that can be repurposed to serve in multiple contexts. Using pattern libraries and tools like Pattern Lab by Brad Frost and Dave Olsen, we should be able to style the most fundamental parts of a website in a vacuum, assemble them into cohesive modules independent of the page, and then include them in the page layout in whatever contexts they were meant to occupy. But the reality isn’t as clear-cut as it might sound.

For the sake of discussion, let’s say we’ve been tasked with building a landing page for a store that sells Whitworth-based tools, a measurement system known only to those brave enough—or foolhardy enough—to own a very old vehicle of British make. I count myself among the latter group.

It’s a pretty simple design. The whole page is made up of product modules that occupy a large central container, and one identical “featured item” module that occupies a secondary container. The overall page layout only changes at a single breakpoint, where the “featured item” container shifts to a sidebar. Here’s a very sparse demo illustrating that layout change. The CSS for this is an easy read:

.col-a,
.col-b {
	clear: both;
	float: left;
}
@media( min-width: 960px ) {
  .col-a,
	  .col-b {
		clear: none;
	}
	.col-a {
		width: 70%;
	}
	.col-b {
		width: 27%;
		float: right;
	}
}

We really only have to deal with the modules in two different contexts, and only at our highest breakpoint: the primary container and featured containers are the same width until the featured container turns into a sidebar. We’ll put a class of .featured on that container so we can scope our styles to it later on.

Now that we have the page layout squared away, we can focus on the layout of the individual modules. They’ll switch between a vertical and a horizontal layout, whichever is best for the available space:

The vertical layout really doesn’t need much finessing in CSS. The natural flow of our markup does most of the work for us; we’ll just make a few small tweaks to constrain the size of our product images and center them beyond that size:

.mod img {
	display: block;
	margin: 0 auto;
	width: 100%;
	max-width: 250px;
}

The styles for the horizontal layout aren’t too much harder. For now, we’ll focus on the primary container—so we won’t scope these styles to our .featured class. Since we want the modules to go back to the vertical layout above 800px for the three-across layout, we’ll only apply the horizontal layout styles between 400px and 799px:

@media( min-width: 400px ) and ( max-width: 799px ) {
	.mod img {
		float: left;
		width: 30%;
		max-width: 999px;
	}
	.mod .mod-header,
	  .mod .mod-desc {
		float: right;
		width: 68%;
		padding-left: 2%;
	}
}

Here’s how those styles play out in a real page. They work just fine, up to a point—in our “featured item” container, where there will only ever be one module at a time, things fall apart a little at medium breakpoints:

Not ideal, but fixable. We’ll just write a new media query to overlap with the one we wrote for the modules in the primary container, then scope all our styles to the .featured class we put on that secondary container element:

@media( min-width: 40em ) and ( max-width: 60em ) {
	.featured .mod img {
		float: left;
		width: 30%;
	}
	.featured .mod .mod-header,
	  .featured .mod .mod-desc {
		float: right;
		width: 68%;
	}
}

Well, okay. This works just fine—as you can see here—but it’s a little gross, code-wise. Not a DRY approach to writing CSS, by a long shot.

This is still bearable when we’re working with such a simple design, but it can get a lot worse when we start fine-tuning details. For example, the vertical layout calls for the “add to cart” button to appear on the right side of the module, but there isn’t enough room when we switch to the three-across layout at medium breakpoints:

If we finesse our styles so the “add to cart” and “quantity remaining” modules deliberately break below the product prices at these sizes—aligned to the left, instead of floating in space on the right—our stylesheet tangles a little more.

Switching between left and right alignment for these elements is simple in terms of the styles themselves: we’re really only dealing with text-align and float properties on the “quantity remaining” and the “add to cart” button. For the media queries that control those simple styles, though, we have a whole list of considerations to work around. We’ll need to add two breakpoints to both of the existing breakpoints that handle vertical alignment. Those styles will impact our “featured” module as well, so we’ll need to override those new styles using our scoping class, with media queries to match. Since the featured module has a little more space to work with than the modules lined up three-across, the “add to cart” button can move back to the right sooner in that context, at the highest breakpoint—yet another breakpoint. But wait, there’s more: these styles aren’t future-proof. If we should ever want to add a module to a new context, we’ll have to copy and paste all our styles over again, with a new scoping class and a fistful of new media queries. If the size of the page layout changes in any way—more or less padding, margins, any size-based styling change to the layout elements—we’ll have to adjust all of those many, many media queries. We now have to find a new viewport size where our breakpoints make sense. Our decisions are disconnected from our layout.

This page comes nowhere near the level of complexity we might encounter in a real project, but we already have a tough stylesheet to maintain.

Enter element queries

All this complication comes from our reliance on the viewport to make our styling decisions. Media queries work just fine for our high-level page layout, but they’re nowhere near as useful for modular components. What we really need is the ability to style page components based on the space they occupy rather than the size of the viewport.

“Element queries” assume a syntax that does exactly that. We’re accustomed to writing a media query for styles that apply only when the viewport is greater than 40em, like this:

@media( min-width: 40em ) {
}

Instead, imagine a syntax where the query itself is scoped to an element and populated with styles that only apply when that element is larger than 40em:

.mod:media( min-width: 40em ) {
}

Well, this would change almost everything we’ve written for our Whitworth page so far. We might still use media queries for our page’s one major layout change, where we go from a linear layout to a larger primary element and a sidebar; basing that on the viewport size makes sense. Once we started building our modules themselves, though, this new syntax would allow us to write all the styles they could ever need just once:

.mod img {
	display: block;
	margin: 0 auto;
	width: 100%;
	max-width: 250px;
}

.mod:media( min-width: 425px ) img {
	float: left;
	width: 30%;
	max-width: 9999px;
}
.mod:media( min-width: 425px ) .mod-header,
.mod:media( min-width: 425px ) .mod-desc {
	float: right;
	width: 68%;
	padding-left: 2%;
}

That’s it, those are all the styles our module layouts will need—see for yourself in this demo, which uses a JavaScript shim to make this theoretical syntax work for real. When the module is larger than 425px, the browser uses the horizontal layout. If the module is smaller than 425px, the browser uses the vertical layout. The styling is now 100% dependent on the container that the module occupies—if it has enough space for the horizontal layout, those are the styles that get used. If we changed the three-across layout to two-across with each module occupying half the container, or we introduced a completely new and unpredictable context for one of these modules, we wouldn’t need to change a thing about their styling. Imagine how much more useful a pattern library would be if we removed the contextual dependency from these kinds of standalone modules: you’d be able to grab any component from the page and move it to any container in the page, without needing to rework hundreds of lines of CSS—without reworking any CSS at all.

For fine-tuning details like the position of the “add to cart” button and quantity remaining, things wouldn’t really get any more complex. Since we wouldn’t be dealing with scoping or finessing module breakpoints to match the viewport-based layout breakpoints, we’d only have one concern: “Is the module too small to have the price and ‘add to cart’ button side by side?” A little experimentation tells us that there’s enough room for both elements as long as the module is larger than 320px, and that’s where we’ll put our module-based breakpoint:

.mod:media( min-width: 320px ) .mod-cost {
	float: left;
}
.mod:media( min-width: 320px ) .mod-buy {
	float: right;
}
.mod:media( min-width: 320px ) .buy-remaining {
	text-align: right;
}

From dozens of lines of redundant CSS, media queries, and overrides to nine lines containing the styles we need. More important, nothing would break if any other styles on the page should change. You can try this out for yourself by fiddling with styles in your browser’s dev tools in this demo, which is using a script that mimics element query behavior—it isn’t something you’d want to use on a production website, but it’s a great way to get your head around the concept.

If this syntax existed, we’d be finished with this Whitworth project in just a few lines of CSS.

But element queries can’t exist—at least, not the way we’d been thinking about them. Just like the first version of the picture specification, element queries are dead and buried forever. But like the search for responsive images, we’re not about to let that stick—not when we have such a clear problem to solve.

Exit element queries

This concept isn’t a hard sell—not for us developers, anyway—but neither were responsive images. Once we get into the gritty implementation details, though, things change.

The first step for the RICG was framing out a Use Cases and Requirements document for element queries, describing and diagramming the problem—not proposing a solution, just providing a clear problem to be solved. So far it’s shaping up to be a lot more focused than the Use Cases and Requirements document for responsive images because it really only sets out to solve one overarching issue.

As word of this document got around, people started thinking through the problem a little further, just like we’d hoped. That led many of us to the same conclusion, before we’d even started tinkering with a proposed specification: element queries can never work.

This has been documented in more detail elsewhere, but the gist is this: we’re using CSS to control CSS, and that means we can cause infinitely looping style changes. If we can tell an element to restyle itself using an element query, what happens if we use that element query to change the element’s width to one where the element query no longer applies?

.our-element:media(min-width: 500px) {
	width: 499px;
}

Well, since the query no longer matches, the new width is no longer applied. Since that new width is never applied, the element query would match again, so the new width would be applied, so the query would no longer match, so the new width wouldn’t be applied—and so on unto infinity. We’ve achieved a TARDIS-caliber paradox with just a few lines of CSS, and there’s no predictable way for the browser to handle it. Should the browser, upon encountering one of these situations, throw the element query styles away entirely? Only throw away the new width? Uninstall itself from the user’s machine? Start a small-to-medium-sized fire? Nothing sounds particularly appealing about any of this, least of all from a browser maker’s standpoint. They’ll never let something with this kind of error potential see the light of day. Game over.

Exciting news, right? We figured out that element queries were impossible in a fraction of the time it took to figure out that responsive images were impossible. The first time, I mean.

Okay, hear me out on this. “Element queries” are over, sure, but the problem hasn’t changed—the Use Cases and Requirements document isn’t going anywhere. This still needs to be solved, and now we know how not to solve it: elements can’t be restyled based on their own properties.

Enter container queries

To solve the problem outlined in the Use Cases and Requirements document, we need to reframe the way we talk about a potential solution. Since a solution can’t allow an element to restyle itself, we can build that constraint into the specification: queries attached to an element can only influence the styling of that element’s child elements.

Armed with our new knowledge of the impossible, the search for element queries has been reframed as container queries.

What would this change about our Whitworth project? Well, nothing—we were only styling the child elements of .mod anyway, not .mod itself. It might just mean a few behavioral changes in terms of how the browser treats these elements. The browser couldn’t allow an element with an attached query to be influenced by the size of its child elements, for example. We might end up with an implicit browser-level overflow style on any element with a query attached, to prevent the infinite looping behavior—much more predictable than the browser picking and choosing bits of our stylesheets to avoid melting down.

So, how do we keep things moving forward from here? By finding the next impossible-to-solve issue. So far, “container queries” have held up to more scrutiny than “element queries” did—but talk is cheap. This recursion issue is a big one, but it still only scratches the surface of the potential issues we could run into as a solution starts to materialize.

We’ll make a lot more progress by experimenting with this pattern, which isn’t the easiest thing to do when we’re dealing with syntaxes that don’t exist yet. That’s how we ended up with the very first version of Scott Jehl’s Picturefill, lo those many years ago. Without tinkering—without actually using something resembling the picture markup pattern—we couldn’t make much sense of the spec we were just starting to write. So in the same vein, the RICG has started up a repository for container query demos, using the same shim as our Whitworth demos. Now we can all try this syntax out for ourselves, and help find the issues we’ll run into when using this kind of CSS pattern for real—the sorts of problems you can only find by building something. Clone the repo, copy the demo-template directory into /demos, give your demo a name, and build something using container queries. Be sure to let us know how it goes, whether good or bad.

Exit container queries..?

As the first version of the container query spec comes together, well, its days may very well be numbered, too. The Use Cases and Requirements document isn’t going anywhere, though; the problem we need to solve is out there, looming, demanding that it be solved somehow.

The answer may be this very first proposal, for all we know. It might be something completely different—something we haven’t even begun to consider. But the RICG isn’t a decision-maker in the world of web standards, and we don’t aim to be one. We want to be a rallying point—to provide ways for designers and developers who might not want to wade through arcane listservs and IRC channels to get involved in the standards that affect their daily work.

There’ll be plenty more work to do after this, and we’ll need all of you to get it done. With your help—with thousands of designer and developer voices contributing to the search—it’s only a matter of time until we find the answer together. We’ve done it before, and we’ll do it again.

Stiffen the sinews; summon up the blood. Banish the phrase “element queries” from your vocabulary forever. Let’s get to work on container queries, regardless of what name or form they might take in the future.

Rian van der Merwe on A View from a Different Valley: Unsuck the Enterprise

 ∗ A List Apart: The Full Feed

There’s something strangely appealing about trying to make enterprise software not universally despised. I guess I believe in a utopian vision where enterprise software is useful, usable, and (gasp!) enjoyable.

But until we get there, I think we can all agree that enterprise software mostly still sucks. And I think it sucks mainly for two reasons:

  1. A lack of empathy for end users.
  2. Too much legacy.

The lack of empathy issue is an understandable outcome of the process. See, we have this piece of software that we sell to company leaders, who care about things like control, configurability, compliance, and how many features the thing has. But the piece of software is mostly used by people who have very different needs.

The people who use the software every day only care about one thing: getting stuff done effectively. And if they can’t do that, a really ugly death spiral happens. As more people realize they can’t get anything done with the software, fewer people want to use it, until eventually no one uses it anymore.

In short, a buyer-focused product strategy builds for features and timelines. A user-focused product strategy builds for the Job-to-be-Done. Those are very different things.

Second, there’s too much legacy that drags large corporations down. There are waterfall processes masquerading as “agile,” well-established and well-defended functional silos, and many layers of bureaucracy. The result is something Jon Kolko sums up well in “Dysfunctional Products Come from Dysfunctional Organizations”:

The dysfunction in the organization becomes the dysfunction in the product, and that gets passed on to the customers.

How can we unsuck this predicament? Four things have helped me, and continue to help me as we make this transition toward designing in leaner, more empathetic ways.

Show the business value of design

The 2014 Design Value Index report just came out, and it showed something quite compelling about design-led companies:

The Design Value Index (DVI), a market capitalization-weighted index comprised of design-driven companies, shows … 10-year returns of 219% over that of the … S&P 500 from 2004 - 2014.


Good Design Drives Shareholder Value

From “Good Design Drives Shareholder Value” by Jeneanne Rae

So we know design-led companies make more money. But is it also possible for design to save on development costs?

This chart is based on data from the book Software Engineering: A Practitioner’s Approach, and it shows how much cheaper it is to make changes to software before or during development. This quote from Marty Cagan also makes the point really well:

Instead of using one prototyper for a few weeks, [many companies] use the full engineering team for full release cycles to build the software that is then QA’ed and deployed into production systems. This is why it typically takes so many companies three or more releases over one to two years to get something usable and useful. They are using the engineering organization to build a very, very expensive prototype, and they use their live customers as unwitting test subjects.

This is a way to tie user-centered design to both revenue increase and cost reduction. That’s an essential (and compelling) combination to get people to take design and user empathy seriously.

Shrink user-centered design to fit

The biggest reservation that teams at enterprise software companies usually have about research and user experience work is that it takes too long and costs too much. Our job is to show teams that we can still get enormous value from UX methods, even if the budget is relatively small.

That’s the beauty of user-centered design: it shrinks to fit. Can’t do an ethnographic study? Do some phone interviews. Can’t build out a full HTML prototype? Make a clickable prototype in Axure, or heck, make a paper sketch. Can’t do a full usability study? Go to Starbucks Stumptown¹ and ask someone if you can buy them a coffee in exchange for some feedback:
-->

                                                                                                                                             
Big budgetMedium budgetSmall budget
ExploreEthnographic studyPhone interviewsAsk a friend
PrototypeHTMLClickable prototypePaper sketch
User testingFormal usability testingRITEShow someone at a coffee shop

Turn sales into a product design function

This is a big one. I’ve seen enough animosity between sales teams and product teams to last a lifetime. And both sides usually have legitimate points.

Product teams usually complain that the sales team sells stuff that doesn’t exist in the product—and even worse, promises release dates—which means they have no flexibility to base their roadmaps on user feedback and strategic direction.

Sales teams usually complain that product teams don’t see them as partners, and ignore their feedback constantly. This is a huge mistake, because sales teams often know the product’s capabilities and shortcomings the best of anyone in the organization. They should absolutely be part of the development process.

How do you do this? What’s worked for me is to provide a framework that allows both teams to speak a common language. For me, that framework, is Jobs-to-be-Done, and more specifically, the Product Forces Framework.

Product forces diagram, based on jobstobedone.org

For someone to move from their existing behavior (a product they’re currently using) to new behavior (switching to a new product), there are two types of forces at work: progress-making forces, and progress-hindering forces.

Progress-making forces move people from their existing behavior to the new behavior, and consist of the push of the current situation (things they’re not happy with in the current product) and the pull of the new idea (things that sound appealing about the new product).

Progress-hindering forces hold people back from switching to new behavior. They consist of allegiance to the current behavior (things they really like about the current product) and the anxiety of the new solution (worries about learning curves and not being able to accomplish their goals with the new solution).

For someone to switch from an existing product to a new product, the progress-making forces have to be stronger than the progress-hindering forces. This might seem obvious, but applying this model to your product planning can inject an extremely healthy dose of reality. Is the product really that much better than a current solution? What does the new product have to do to overcome people’s allegiance to what they’re currently using?

This is not only a very good sales strategy, it’s also a good way to gather product feedback and come to agreement on what will make the product better (and help it sell better!).

As for putting a stop to selling features that don’t exist yet… I’ll be honest, that can take some time. But the more the sales team and the product team collaborate (keep reading for more on that) and have a common language, the better this will get as well. There is hope.

Break down silos through collaboration

“Collaboration” has become a pretty overused word, and it’s now difficult to know exactly what we mean by it. So let’s be clear: just because you sat in a meeting with a different team doesn’t mean you collaborated. Collaboration, to me, means that you made something together. There is always some output during collaboration—from a solid idea that a designer can go work on, to personas that everyone worked on together, to piles of sticky notes that eventually become a customer journey map.

This kind of collaboration is especially important in the enterprise, where a designer’s role is often mostly about facilitation.

There are several techniques to encourage real product collaboration:

If I can further summarize a way to bring empathy-driven design to an enterprise, here are the methods that I try to communicate to teams I work with:

  1. Show them why it’s important.
  2. Show them it’s not going to make their lives difficult.
  3. Give them a framework that covers the whole product.
  4. Make them part of the process.

So if you’re someone who works in enterprise software, come a bit closer—I need to tell you something…

I know the work can be difficult. I know there are an infinite number of factors involved in getting product live, and sometimes what gets launched isn’t what you had in mind. I know there are sleepless nights about this sometimes. But don’t give up. Don’t think that enterprise design has to be boring or tasteless. With a little bit of effort and a lot of tenacity, it’s possible to create great enterprise product. We need you to make it happen, though. Who else is going to do it?

Footnotes

July OUCH Newsletter - Social Media: http://www.securingthehuman.org/ouch, (Wed, Jul 1st)

 ∗ SANS Internet Storm Center, InfoCON: green

...(more)...

Apple "Patch Tuesday", (Wed, Jul 1st)

 ∗ SANS Internet Storm Center, InfoCON: green

Yesterday, Apple released patches for OS X, iOS, Safari, Mac EFI, iTunesand Quicktime (Windows) [ ...(more)...

Thoughts on Time-series Databases

 ∗ jmoiron.net blog

Preetam "recently" blogged about catena, a time-series metric store. There was another blog post about benchmarking boltdb by a Fog Creek engineer, also looking to write a time series database. This is something of a pattern in the Go community, which already boasts seriesly, InfluxDB, and prometheus; there are almost certainly others.

Time series data has been de rigueur at least since the Etsy's seminal blog post on StatsD, though in reality that was just an inflection point. Time series modeling and graphing predates computer systems, but they have been a popular way of tracking and visualizing systems and networking data since at least the early 90s with MRTG. A few factors are converging now to make these kinds of systems more important: "Big Data" is getting much, much bigger; virtualization and containerization has increased the number of independent "nodes" for a typical distributed application; and the economies of the cloud have put the brakes on the types of performance increases typically attributed to "Moore's Law."

This topic is relevant to my work at Datadog, and I've been thinking about it for a long time now. I've wanted to collect my thoughts somewhere for a while, because some of them are identical to those expressed in other recent projects, and others are antithetical to them. I figured this would make my input at worst interesting.

For a primer on this subject, please read Baron's Time-Series Database Requirements. There's a reason that most other recent articles cite it; it contains a brief but complete description of the problem, the requirements for many large-scale time-series users, and some surprises for the uninitiated. Some of the implications that are implicit in these requirements but perhaps not obvious to onlookers:

Time-series Storage Layer

People often ask themselves:

If answers to the first two questions is yes; the storage layer must optimise for both writes and reads. You can't have a useful system if reads are not optimised, and you can't have a scalable system if writes are not optimised.

If you try to write multiple series to one indexed collection; think something like an sql table with (key, timestamp, value) rows, then you're ranging across too much data when servicing reads. If you separate every series out to its own collection, you have possibly the worlds worst write pattern for persisting to disk: high frequency writes across all of your data.

This is why you can't measure your system's performance for 100m points by just batching 10000 points up for 10000 metrics: even if you have 1s resolution reporting, it takes almost 3 hours to get 10000 points; at the statsd popularized 10sec resolution, it takes over a day. It's not a realistic write pattern.

Despite this, it's the desired write pattern. So people spool points in memory and try to flush them later in larger chunks. This is the idea behind LSM Tree stores, and it's why people are increasingly using them to solve this problem. I've seen a few basic approaches:

These approaches have advantages and disadvantages. To explore how they work in practice, lets describe the problem a little differently:

A time-series is a vector of points.

There are only a few primary operations we have to deal with:

Using Files

Obviously all databases "use files" somehow, but the specific approach I am talking about here is the file-as-a-series approach. With this, you get highly efficient buffered appends and buffered linear reads for free. On reasonably sophisticated filesystems (zfs, ext4, xfs, likely others), directory entry lookups can be hash table based, which means they are O(1).

Filesystems seem to be getting increasingly complex and difficult to understand, and tuning them is getting to be something of a black art. Although some of them were designed to handle the case of millions of files, none of them are optimised specifically for it. If your data is very sparse, the overhead for a small amount of data is significant (1 block, typically 4k). If your query ranges are small (and typically they are), you may be paging in mostly empty space when ranging across many timeseries when they could be more efficiently packed together. Finally, ranges across your entire dataset, which are vital to your operations in terms of backups, batch jobs, etc, are slow, and it takes a long time to get all of your handles in memory.

Using a tree-based store

With tree-based k/v stores, we have a few possible approaches. Keys can either identify a single series, a single point, or perhaps a time slice of a series. Regardless of your decision on your key structure or even your tree type, key lookups are going to be O(log n), which is potentially leaving a lot on the table.

At scales I'm interested in, the number of points is too high for a range over the keys to be anything but a CPU cache disaster, so k/v pairs that look anything like {series, timestamp} -> {point} simply won't work. Open source time-series stores in use at smaller scales may attempt to go this route, but I can say with a high degree of confidence that files are simpler and faster unless you regularly read a large percentage of your dataset. With O(log n) lookups, we don't want to add extra time to our reads as well.

That leaves us with writes to look at, which is where image and reality can start to become blurred. The COW approach used by both B-tree databases seems like it'd be a very poor fit for a huge volume of random writes; BoltDB creator Ben B. Johnson says as much. In practice it's a little less clear, but I think it's likely a wash. These all use mmap, and frequent tiny writes to most of the pages in your database will obviously dirty most of the pages in your database, causing some mighty I/O on f/msync.

Appends on LSM's might seem better, but the tree merging compaction they use is a very intense operation and I just can't see a way where it seems like an irreducible complexity for storing and concatenating related vectors.

In general, I feel the tree approach puts efficient writes and efficient reads at odds with eachother. Optimised local linear reads clash with the optimal write pattern of appending keys. Preferably, these would be complementary or worst orthogonal.

Query Semantics

Regarding the semantics that must be supported in the time-series store in service of the query language, I suspect that the only required semantics are that you can batch and parallelise reads; everything else is better served as a operation on top of a different store that narrows these reads, and better implemented outside the specialized, demanding constraints of time-series storage.

Dimensionality

Having thorough experience with systems that have and lack dimensionality, I have to say that it really is absolutely vital; at the "product" level, lacking support for it is a guarantee that something that does support it will eventually replace you. This is true of both companies attempting to enter the market and open source solutions.

Stuffing all your metadata into the metric key and then treating things as a key/value lookup isn't a particularly great model for querying; you want to be able to have at least or/and logic across dimensions; eg: az=us-east-1 AND role=db-master, version=2.1 OR version=2.3, and preferably a richer language. Graphite's metric name globbing system (system.cpu.az-us-east-1.role-db-master.host-*.avg) definitely isn't it.

But does the storage layer need to support any extra semantics to enable this?

Currently, dimensions are already implemented successfully on top of stores unaware of them; Prometheus does this with a separate index, for example. The general pattern is that queries across dimensions first lookup the appropriate keys in an index (LevelDB for Prometheus), identify the series' involved, and then joins are done across those series by a query engine that supports all sorts of other aggregations.

It is perhaps at this level that some kind of cooperation would really bear fruit; some of the real world cardinalities I've seen makes me think it's unlikely related series could always be stored near each other on disk, but making them happen at least on one node is a distinct possibility.

In a dimension-aware store, the indexes involved in these queries will either point to tuples stored across potentially vast sectors of unrelated data, or contain the data themselves and be both enormous and slow to write to. While SSDs might smooth this, they still perform best when reading whole pages; locality still matters.

I suspect it will remain faster to do 1000 highly optimised reads than to do these kinds of ranges at scale.

I think there may be some clever maths to be discovered or combined here to save us from some of this madness, and do for time-series what HyperLogLog does for counting or what Bloom-Filters do for set membership, but I'm not that confident it will be generally useful. Google can get away with these tricks for Google trends, but people generally test TSDBs with known data and will see inaccuracies and query weirdness quickly.

Features like InfluxDB's continuous queries can alleviate this for known high-cardinality joins, but this particular implementation is expensive, and in practice these requirements can generally be aggregated into a single time-series manually without much effort.

Aggregation, Expiry, Etc.

Time-aggregation and data expiry are first-class citizens in round-robin databases, which are still really the open-source standard. There's rrdtool, an old mainstay in the industry, and there's graphite; specifically, its old "whisper" engine.

These have a few significant problems that make them unusable for some. RRDs are constant space, and lots of legitimate time series usage involves a large number of sparse series. Another problem is that it does compaction/aggregation on writes, which causes lots of write amplification.

If you get the basic storage layer right, aggregation and expiry start to look a lot like dimensionality: they can be implemented asynchronously in a separate policy layer. It doesn't seem important that the actual storage engine is mindful of these; in fact, it's probably better ignored for efficiency's sake.

If you are treating a Time-series database as a holistic concept, you'll definitely want some kind of approach to this; there's simply no good way of combining thousands of 1s-resolution series across 6 months that doesn't involve a lot of processing power or waiting quite some time.

ISC StormCast for Wednesday, July 1st 2015 http://isc.sans.edu/podcastdetail.html?id=4551, (Wed, Jul 1st)

 ∗ SANS Internet Storm Center, InfoCON: green

...(more)...

Create a Content Compass

 ∗ A List Apart: The Full Feed

Project types

Chapter 2 mentioned that breaking off a small project might make sense, rather than tackling all the content across your organization at once. Make sure you remember where your project falls in your organization or client’s entire content universe.

Your project’s place in the organization’s content universe affects what you’re able to control or influence. It also is affected by the formal and informal strategies guiding content efforts throughout the organization.

Content strategy for an entire organization and across all channels is super complex and very steeped in business management and operations. For the purposes of this book, I focus on the three main project types I’ve encountered most: function, property, and subset.

Function

Function refers to a functional unit within an organization, such as sales, marketing, communications, customer service, or human resources. Of course, every organization is set up a bit differently, so yours could have different names for similar functions.

Very often, my client is a senior leader within one of these functions who wants help with content strategy across all the content types and delivery channels. Marketing is probably the most common.

Diagram showing that customer experience is an example of a functional unit within an organization.

Property

Property refers to a single digital destination, such as a website or an application, or even a social media channel, such as Facebook. Most of my digital property projects are for new website launches or existing website refreshes or redesigns. They could be a client’s primary web presence or a secondary web property for a product, service, event, or other singular purpose.

Other times, I’ll be asked to help specifically with a mobile site or mobile application. (Don’t get me started on mobile websites, but do read Karen McGrane’s book, Content Strategy for Mobile.)

Diagram showing how a website is an example of a single digital property targeted for a project.

Subset

Subset refers to any defined portion of content on a digital property. You can define that subset by where it lives in the site, such as the Help section or the About Us section. Or, you may define it by whom the content is for (current customers, prospective customers, and so on) or what the content is about (products, company information, area of expertise, and so on).

Diagram showing how the help section of a website is an example of a subset of digital content.

Core strategy statement

The core strategy statement is the central component of your content compass. It answers the fundamental questions for providing the right content, to the right people, at the right time, for the right reasons. Table 11.1 summarizes the four questions a core strategy statement must answer.

Content product What content should we produce, procure, curate, and share?
Audience Who, specifically, is that content for?
User needs Why do those audiences need or expect that content from us?
Business goals What outcomes does providing this content help us achieve?

Table 11.1 Core content strategy statement questions

What does a core strategy statement look like?

Core strategy statements come in many formats, and there’s no one right way to document one. A few key characteristics and concepts will help you craft an effective statement.

First, it must resonate with and be memorable for the people who need to know and use it so that they can apply the key concepts to make content decisions.

One way I’ve tested my statements is to present them to my colleagues with details about the four questions they answer and then ask them to recap the strategy. If people who don’t have much context for the project can get the gist and explain it back to me, it’s usually in pretty good shape.

Second, it needs to be prescriptive enough to allow you to say no. One way to check it is to look at your past 10 content projects. If you would have decided not to do some of those projects if you had had the statement at the time, it’s probably strong enough.

Let’s look at an example that my boss and CEO of Brain Traffic, Kristina Halvorson, has used in presentations (I changed it up a little).

To reduce customer service center costs, we will provide user-facing, task-based support content that makes our professional customers feel confident when configuring products for their clients.

Now let’s break that down a bit into the four key components. The business goal for this project is to decrease service center costs (by allowing people to self-serve). To do that, they’ll produce user-facing content that’s focused on tasks users want to complete. The content’s audience is the professional customers, meaning people who are quite experienced and who install the organization’s products for clients. The user need provides help with those installations.

The strategy statement with portions mapped to the business goal, the content product, the audience, and the user needs.

Next up, use the statement to make some content decisions. Use the list of content projects and ideas, such as in Table 11.2, and decide whether your organization should do them.

Yes No Idea
    Repurpose word-for-word content from the customer service call center knowledge base.
    Reorganize help content based on analytics data about frequency of use and search terms.
    Display an interactive timeline showing how product X was developed on the help landing page.
    Conduct user interviews of a survey with professional customers to find out if they are getting what they need from help content.
    Feature professional customer profiles throughout the help section.
    provide a video from our CEO about how important it is for customers to be able to self-serve on our site.

Table 11.2 Ideas for consideration

Here’s what I decided and why:

How do you craft a core strategy statement?

The best part about putting together your core strategy statement is that you already have all the information you need from the discovery phase. Even better, your stakeholders have already agreed on the important aspects: who the content is for, why they need it, and what the organization is trying to achieve.

Now you just need to put all that together and add some specificity about the kind of content to produce, through either a collaborative or individual approach.

Collaborative approach

With the collaborative method to crafting a core strategy statement, you hand out a Mad Lib worksheet to the stakeholders. They then work individually to fill in the blanks. This is one I’ve adapted from a version Sara Wachter-Boettcher created:

<Organization, Department, Property, Section> helps our company accomplish <this goal> and <this goal> by providing <descriptive phrase> and <descriptive phrase> content that makes <this audience> feel <this emotion or adjective> and <this emotion or adjective> so they can <complete this task> or <complete this task>.

If I’m doing this exercise in-person, I have each person read their statement out loud with no discussion. Then, I’ll ask them to talk about the similarities and differences, the words or phrases that resonated most or didn’t seem quite right, and anything else that stood out to the group.

From there, I take a stab at combining everything into a single statement and getting agreement from stakeholders on the aggregated version. You don’t need to wordsmith it at this point; just get the concepts down. You can make it an editorial masterpiece later (or not).

If you have time, brainstorm a list of recent content projects, and ask participants to say whether they would still do the projects given the statement they just created together. This is a great way to demonstrate how it works and set expectations about how it might affect future work.

If you can’t get the group together, you can send the Mad Lib out via email to stakeholders. The first time I did this, I was skeptical, but it actually worked fairly well.

You don’t get the benefit of the resulting discussion with the email approach. But you can go back to stakeholders to work through disagreements before you.

Individual approach

Some projects don’t have enough days in the timeline or dollars in the budget to craft the statement collaboratively with stakeholders. That’s OK. You can do it yourself.

I’m more of an internal processor of information, so my method is typically to grab my notebook or insights spreadsheet and start reading through my notes. I jot down the things that stick out related to content, audience, business goals, and user needs. I’ll start writing multiple versions of the core strategy statement until I feel I nailed it. That usually involves several renditions and pieces from each getting pulled together, edited, discarded, and massaged.

That approach works for me, but sometimes I get stuck and need to do something a bit more methodological. In those cases, I still grab my notebook or insights spreadsheet.

But I transfer the key insights to sticky notes, note cards, or slips of paper. Then, I categorize those insights into the four categories: business goals, content product, audience, and user needs. Next, I review each category to see what key themes or ideas emerge. And finally, I take those themes and work out the core strategy statement, which typically goes through several revisions.

Diagram of the process of crafting a strategy statement, showing how stakeholder insights are categorized, enabling themes to emerge, which lead to the statement draft.

Messaging framework

A messaging framework clarifies what you want your audiences to know and believe about you, and tries to prove that this message is true.

As a component of your content compass, the messaging framework helps ensure that every piece of content you create supports the framework. If it doesn’t, it doesn’t belong or needs to be revised.

What does a messaging framework look like?

As with a core strategy statement, you can document your messaging framework multiple ways. I tend to create something somewhat visual to show a hierarchy or progression. Other practitioners I know use bulleted lists.

I write the framework from the user’s perspective. Other practitioners write it in the organization’s voice. It really doesn’t matter, as long as you adhere to the following:

One of the reasons I like to write the messaging framework from the users’ perspective is because it’s a nice foil to the core strategy statement. The core strategy statement is primarily written for the business. By writing the messaging framework from the users’ mindset, you have a well-rounded compass by which to set the direction for content and stay on track over time.

This example builds upon the core strategy statement and details the first impression you want professional customers to have when they visit the support section, why your support section is valuable to them, and how you’re demonstrating that value with your content. Notice that the proof statements are related to what the content communicates and how it’s organized (substance and structure).

Diagram showing an example of a messaging framework, progressing from first impression to value statement to proof.

How do you develop a messaging framework?

You probably won’t be surprised to read that you can do pretty much the same things you do for the core strategy statement. Like right down to the collaborative and individual approaches.

Mad Libs, again, work well in a collaborative setting. Some ideas for Mad Libs to have stakeholders fill out for each component of the framework include:

First impression
When a user first lands on the site, we want them to feel <blank> and <blank>.

Value statement
After spending a few minutes on our site, users should feel <blank> because they understand we provide <blank> and <blank>.

Proof
Our content demonstrates that we provide just what our users need because <blank>, <blank>, <blank>, and <blank>.

Once you’ve collected the Mad Libs, you can use the data to determine the patterns and themes and form those into the message framework. I go through that process in much the same way I create the core strategy statement—either processing the information in my head or using sticky notes to organize the data.

If you’re not able to get input from your stakeholders in the form of the Mad Libs, don’t worry. Chances are, you have all the information you need from your stakeholder interviews.

Grab your notebook or discovery insights spreadsheet. Better yet, do the Mad Lib and use your discovery insights to develop your messaging framework.

Show the way

Now that you have the two key components of your content strategy in place—core content strategy and messaging framework—you can tell your content people about them. Schedule some time with the key teams who work on content—from creating it to reviewing it to publishing it—to go over your content compass and how to use it.

The next chapter discusses developing a plan for measuring the effectiveness of your content strategy and resulting content efforts. Your content compass will help you decide what to measure.

Copyright © Brain Traffic, Inc. and Meghan Casey. All rights reserved by Pearson Education, Inc. and New Riders.

A List Apart № 423: container queries, responsive content

 ∗ Jeffrey Zeldman Presents The Daily Report: Web Design News & Insights Since 1995

A List Apart 423

WHETHER the topic is responsive CSS or content that responds to the right user at the right time, Issue № 423 of A List Apart is all about finding the path forward:

Container Queries: Once More Unto the Breach

by Mat Marquis

Mat MarquisMedia queries have been the go-to tool in building responsive sites, allowing us to resize and recombine modules to suit multiple contexts, layouts, and viewports. But relying on fixed viewport sizes can quickly twist stylesheets into Gordian knots. We still need a future-friendly way to manage responsive CSS. Mat Marquis explores the problem and the progress toward the solution—and issues a rallying call.


Create a Content Compass

by Meghan Casey

Meghan CaseyContent projects need a sense of direction: something to help you and your team provide the right content to the right people at the right time. Enter the content compass—centered on your strategy and supported by your messaging—to keep your content efforts on track. In this excerpt from Chapter 11 of The Content Strategy Toolkit, Meghan Casey explains her methodology for developing a core strategy statement and messaging framework.

How Malware Campaigns Employ Google Redirects and Analytics, (Tue, Jun 30th)

 ∗ SANS Internet Storm Center, InfoCON: green

The email message sent to the bank employee claimed that the sender received a wire transfer from ...(more)...

ISC StormCast for Tuesday, June 30th 2015 http://isc.sans.edu/podcastdetail.html?id=4549, (Tue, Jun 30th)

 ∗ SANS Internet Storm Center, InfoCON: green

...(more)...

The Powershell Diaries 2 - Software Inventory, (Mon, Jun 29th)

 ∗ SANS Internet Storm Center, InfoCON: green

After last weeks story, hopefully youve got your problem users accounts identified. With that wor ...(more)...

Margaret

 ∗ xkcd.com

Otherwise known as Margaret the Destroyer, I will bring pain to the the Great One. Then again, maybe I won't.

ISC StormCast for Monday, June 29th 2015 http://isc.sans.edu/podcastdetail.html?id=4547, (Mon, Jun 29th)

 ∗ SANS Internet Storm Center, InfoCON: green

...(more)...

The EICAR Test File, (Sun, Jun 28th)

 ∗ SANS Internet Storm Center, InfoCON: green

Im sure most of you are familiar with the

Is Windows XP still around in your Network a year after Support Ended?, (Sat, Jun 27th)

 ∗ SANS Internet Storm Center, InfoCON: green

This week Computerworld [

On Our Radar: Precious Humans

 ∗ A List Apart: The Full Feed

Molly Holzschlag is a great educator, tough fighter, and vital friend of what folks are now calling “the open web,” and she needs our help.

Molly Holzchlag and Jeffrey Zeldman
Molly Holzchlag and Jeffrey Zeldman.

She took over as project leader when I left The Web Standards Project in 2002. In that role, Molly did hugely important (and often thanklessly unheralded) work bringing improved compliance, plus a renewed respect for web standards, to some famously intractable browser companies. She also directed The Web Standards Project’s important and multi-dimensional educational outreach effort.

Between those efforts, her two decades of public speaking, and the dozens of web development books she has written, Molly has contributed meaningfully to the foundational thinking and understanding of tens of thousands of web developers, designers, and engineers. If you don’t know her name and her work, you don’t fully understand this industry.

Molly has been diagnosed with chronic pancytopenia, a life-threatening blood disorder, and her insurance is about to disappear. She’ll need the support of friends and strangers to afford lifesaving blood treatments and medications. Pray for her, chant for her, help her if you can. —Jeffrey Zeldman, founder and publisher

Your weekend reading

  1. I’m a big fan of the Library of Congress Flickr feed (mostly for the old-timey photographs of the type that Shorpy cleans up and posts) and recently they’ve been posting old graphic posters from the collection. Like this poster for a 1938 photography exhibition in New York City. The description for each piece links back to the Library of Congress, where they often have very high-resolution images available to download. —Mike Pick, creative director and designer
  2. This week, Wired talked up new startup Doxa, which aims to use an “OKCupid-like” algorithm to match women with tech companies where they’re likely to thrive. It’s an interesting approach to getting beyond “diversity” percentages, and it’s nice to see Wired lead with the truth: “Tech has a diversity problem.” But I was more convinced by Danilo Campos’s take: that “shoving underrepresented people into the existing startup order”—even using smart algorithms—won’t make those organizations actually ready to support diverse employees. “If you’re serving at a place where no one in leadership understands your needs, getting accommodation for those needs can become challenging—or even alienating, when you’re misunderstood,” he writes. In other words, crunching survey data to help women find a better fit might be nice, but real change happens when leadership teams are as diverse as the people they’re trying to hire. —Sara Wachter-Boettcher, editor-in-chief
  3. Richard Rutter has Kickstarted a book he’s writing on web typography. I remember first reading an article Richard wrote for us on the topic back in 2007, and this book has been in the works since before then. Richard lives and breathes this stuff: he helped set up Fontdeck, and has also run several installations of Ampersand Conference. I have a feeling this book is going to be the canonical reference on web typography for years to come. —Anna Debenham, technical editor
  4. I was doing some reading on SVG this week for a project we’re working on. I came across Jake Giltsoff‘s SVG on the Web, a single-purpose site outlining the use of the SVG format. Giltsoff went into quite a bit of depth, addressing the use of SVG as an image type from a development perspective, and responsive approaches to using the format. It’s a bookmark-worthy resource for in-depth or at-a-glance referencing. —Erin Lynch, production manager
  5. I was privileged to hear Leslie Jensen-Inman speak last week, and she explained how adding a single question to the daily stand-ups at Center Centre has helped the team create an environment that is enthusiastically supportive of ongoing learning. “What is the most important thing I learned since the last time we met and how will what I learned change the way I approach things in the future?” In Leslie’s Meet for Learning, she goes into more detail about the ways this open-ended question has changed their work culture for the better. —Aaron Parkening, project manager

Overheard in ALA Slack

“I watched a chipmunk suck water out of my friend’s Camelback at the Grand Canyon.”

Your Friday gif

A glitchy image of a screen that says “Learn to Use the Internet, Level 2”

Big Web Show № 132: Modern Layouts with Jen Simmons

 ∗ Jeffrey Zeldman Presents The Daily Report: Web Design News & Insights Since 1995

Jen Simmons

THE BIG WEB SHOW is back from its break. My guest this week is Jen Simmons (@jensimmons) of The Web Ahead. We discuss moving beyond cookie-cutter layouts on the web; the ins and outs of podcasting; tradeoffs when designing a website; learning from your users; Jen’s journey from theater to technology; and more. Sponsored by Dreamhost. Enjoy The Big Web Show № 132.

URLS

http://thewebahead.net/81 (great links in the show notes!)
https://twitter.com/jensimmons
http://labs.thewebahead.net/thelayoutsahead/
https://github.com/jensimmons/thelayoutsahead
http://alistapart.com/article/css-shapes-101
https://css-tricks.com/examples/ShapesOfCSS/
http://sarasoueidan.com/blog/css-shapes/index.html
http://thewebahead.net/80
http://www.w3.org/TR/css3-grid-layout/
http://thewebahead.net/49
http://next.zengrids.com
http://www.jensimmons.com/about/
http://www.pagetutor.com/common/bgcolors216.html

Cisco default credentials - again!, (Fri, Jun 26th)

 ∗ SANS Internet Storm Center, InfoCON: green

From NYPL to DC Comics: the lettering of Ira Schnapp

 ∗ Jeffrey Zeldman Presents The Daily Report: Web Design News & Insights Since 1995

Action Comics logo, 1938

Superman, 1940

HE DESIGNED the lettering on The New York Public Library and the James Farley Post Office (“neither snow nor rain…”), created titles for silent movies, movie posters, and pulp magazines in the 1920s, and started working for DC Comics in 1938, where he designed the masthead for Action Comics, refined the Superman logo, and brought dozens of DC Comics texts and titles to life. A new exhibit at The Type Directors Club honors Ira Schnapp and sheds light on his decades of influential work.

Daybreak in Myanmar: Photos by Geoffrey Hiller

 ∗ Jeffrey Zeldman Presents The Daily Report: Web Design News & Insights Since 1995

Book cover, Daybreak in MyanmarMyanmar in Southeast Asia is one of the least known places in the world, due to the military dictatorship that has isolated the country for the past sixty years. Now that the government is making the transition to democracy, the veil is slowly lifting, as are travel and economic sanctions. In Daybreak in Myanmar these images of a place once frozen in time are unique and timely.

Photographer Geoffrey Hiller has been documenting the people of Burma since 1987 and has returned several times since the historic opening in 2011 to capture evidence of change, not only images of rallies for Aung San Suu Kyi, but the anticipation, hope and concerns of a nation forgotten by the world. Following his award-winning web documentary from 2000, Burma: Grace Under Pressure, Hiller is publishing this selection of 170 color photographs.

Source: DAYBREAK IN MYANMAR : Geoffrey Hiller :: Documentary Photographer in Portland, Oregon

Reliably hosted by WebFaction.com