Mindchunk

Documenting Code and Pictures

2011-10-15T18:25:00.001-04:00

For the SVGo workshop I had to document many examples of code+pictures, so I created a workflow to create consistently formatted illustrations of SVGo code and the pictures they produce.Below is the script that automates the process. The heart of the script is a small Go program, codepic which creates a SVG file of code and picture:

Here's the script:


#!/bin/sh
for i in $*
do
    base=`basename $i .go`
    gofmt -w -spaces -tabindent=false -tabwidth=4 $i &&
    goc $base && 
    ./${base} > ${base}.svg && 
    codepic -codeframe=f -font Inconsolata -fs 14 -ls 16 $i > ${base}-slide.svg && 
    svg2pdf ${base}-slide.svg
done

for each file the script:

formats the source
builds the source
runs the program and captures its output
make a slide showing the code and output
convert the slide to PDF for inclusion in Keynote

Stock/Product Comparisons with SVGo

2011-08-07T20:54:00.032-04:00

The latest visualization tool written with SVGo is stockproduct, a tool to compare stock prices with the release of products over time. This is a Go version of the tool described earlier. Like other SVGo tools, stockproduct reads data from XML (see example below) and produces SVG.


<stockproduct title="Apple Purchases and Stock Price">

   <sdata price="12.40" date="2002-04" product="Ti PowerBook" image="tipb.jpg"/>

   <sdata price="7.38" date="2002-08" product="Jaguar" image="jaguar.png"/>

   <sdata price="261.09" date="2010-04" product="iPad 3G" image="ipad3g.jpg"/>

   <sdata price="399.68" date="2011-07" product="Lion" image="lion.png"/>

</stockproduct>

The example above plots my purchases of Apple products vs. the stock price.

Stockproduct scales the graph by the specified size, and the graph can be placed anywhere on the SVG canvas. Note that every chart component: bars, images, product names. prices, scales, and images may be turned on and off using command line options.

This example:

compares the adjusted closing price vs. operating system releases from Microsoft and Apple, ordered by the date of release. Between 1991-2001, both organizations were able to keep pace, with Microsoft releasing Windows 3.1, 95, 98, and Apple releasing System 7, 8, and 9.

However, between 2001 and 2005 Apple steadily released five versions of Mac OS X (Cheetah, Puma, Jaguar, Panther, and Tiger), but in the same time period, Microsoft released only XP. After 2005, Apple released Leopard, Snow Leopard and Lion, where Microsoft released VIsta and Windows 7.

From 1991-2007, there were no large differences in stock prices, but in 2007, the there was large gain for Apple, growing from $184.70 to $386.90, with Microsoft's price staying flat, at $23.75 at the release of Windows 7 in 2007.

In fact the range of prices and releases during past 20 years is Microsoft: $2-23.75, and 6 operating system releases, with Apple's stock price ranging from $4.11 - $386.90, with 11 operating system releases (not including iOS).

Here's the same data with Apple products on the left, peaking at Lion, and Microsoft products on the right:

Bullet Graphs with SVGo

2011-07-11T19:54:00.024-04:00

Here is the output of bulletgraph -- an implemetation of Few's Bullet Graphs using SVGo. Like pmaps, bulletgraph reads a XML representation of the data, unmarshals the input into data structures, and produces the bullet graph from the structures. The input looks like this:

<bulletgraph>
 <bdata title="Revenue 2005" subtitle="USD (1,000)"
      scale="0,300,50"  qmeasure="150,225" cmeasure="250" measure="275"/>
</bulletgraph>

Each bdata element produces a graph according to its attributes (titles, scales, and measures)

The program has various options to control colors of the various components (data color, comparative color, target color), and the indicator showing the target measure can either be the conventional vertical line or a circle. The -help option shows all of the option flags, with an example XML definition.

bulletgraph -help

Usage: bulletgraph [options] file...

 -bc="rgb(200,200,200)": bar color
 -bg="white": background color
 -bh=48: bar height
 -cc="black": comparative color
 -circle=false: circle mark
 -dc="darkgray": data color
 -f=18: fontsize (px)
 -g=30: gutter
 -h=800: height
 -help=false: usage and example
 -showtitle=false: show title
 -t="Bullet Graphs": title
 -w=1024: width

Example Defintion:

<bulletgraph top="50" left="250" right="50">
  <bdata title="Revenue 2005"      subtitle="USD (1,000)"     scale="0,300,50"   qmeasure="150,225"    cmeasure="250"  measure="275"/>
  <bdata title="Profit"            subtitle="%"               scale="0,30,5"     qmeasure="20,25"      cmeasure="27"   measure="22.5"/>
  <bdata title="Avg Order Size"    subtitle="USD"             scale="0,600,100"  qmeasure="350,500"    cmeasure="550"  measure="320"/>
  <bdata title="New Customers"     subtitle="Count"           scale="0,2500,500" qmeasure="1700,2000"  cmeasure="2100" measure="1750"/>
  <bdata title="Cust Satisfaction" subtitle="Top rating of 5" scale="0,5,1"      qmeasure="3.5,4.5"    cmeasure="4.7"  measure="4.85"/>
</bulletgraph>

Thanks to Richard Masci for the example usage suggestion.

Google Web Fonts, SVGo and code sketching with goplay

2011-05-16T07:06:00.039-04:00

This program, webfonts, demonstrates Google Web Fonts and its API with SVGo. The key to its operation is the URI:

http://fonts.googleapis.com/css?family={fonts}

where {fonts} is a pipe-delimited list of font names. The result of performing a HTTP GET on this URI is CSS code that specifies the fonts. For example,

http://fonts.googleapis.com/css?family=Pacifico

produces:

@font-face {
font-family: 'Pacifico';
font-style: normal;
font-weight: normal;
src: local('Pacifico'), url('http://themes.googleusercontent.com/font?kit=fKnfV28XkldRW297cFLeqfesZW2xOQ-xsNqO47m55DA') format('truetype');
}

The program works by using a Go function to perform the GET (googlefont(fontname string)), and place the resulting CSS in a SVG defs element. The program is then able to use the web fonts by name just like any other local font. In this case looping over the names in the list, and displaying "Hello, World" in the corresponding font.

The screenshot shows the code and output in Google Chrome (although any modern browser that can handle inline SVG will work), using a web app that comes with the Go distribution, goplay. Goplay allows you to "play" with snippets of Go code and quickly compile and see the results in a web browser. In the case of programs like the one shown, the generated SVG is placed in-line, interpreted directly, showing the picture.

Here's how to run it: go to a directory where your code lives, and then run goplay -html, which sends the results unaltered to the browser, instead of in a plain text block. A word of caution from the goplay documentation: anyone with access to the goplay web interface can run arbitrary code on your computer. Goplay is not a sandbox, and has no other security mechanisms. Do not deploy it in untrusted environments.

cd [dir]
goplay -html

Next, point your browser to the content served by goplay (which listens on localhost, port 3999 by default):

http://localhost:3999/webfonts.go

Your code will be shown in the textarea, ready to be edited and built. Hit Shift-ENTER and code is compiled and run with the output next to the code. If you want to tweak the program, by changing say, a font name or the string to be displayed, just move to the textarea, make the change, and hit Shift-ENTER again. This method allows you to sketch in code, with immediate feedback.

With the WebKit Inspector found in Chrome and Safari, you can examine timing, code, and font information.

Mobile Market Share: 2010

2010-11-01T19:30:00.001-04:00

Mobile Market Share: 2010
Originally uploaded by ajstarks

Here is a collection of pmaps that outline the changing mobile market share. The data is from Canalys, IDC, CNet, and Asymco.com

Proportional Maps: an alternative to pie charts

2010-10-07T00:16:00.013-04:00

Proportional Maps: Browser Market share

Proportional maps (pmaps) are an alternative to the venerable pie chart for showing the how a set of data adds up to a whole.

Pmaps are designed to show the data more clearly than pie charts by using color (the darker the color the greater the proportion), size (the width of the rectangle corresponds to the data's proportion) and position (the data is ordered from greater to least proportion).

The compact nature of pmaps allows you to compare related datasets by scanning vertically. For example the illustration shows how different sources report browser market share.

The pmap program is implemented in the Go programming language, and produces SVG via the svgo library

The program has various strategies to clearly show all the data: you may show any combination of labels, percentages, or raw data. The labels and data are automatically arranged for maximum visibility: if the data rectangle is too narrow, its labels is displayed as a call out. (the thresholds for label length and data size are adjustable). The style of call out is also configurable: staggered or alternating between above and below the data)

The command line:

pmap -stagger -p -g 100 -bg lightsteelblue -t "Browser Market Share" -showtitle bs.xml > bs.svg

created the illustration above. It means: read data from bs.xml, create the pmap as a SVG file called bs.svg, stagger the overflow labels, show percentages, separate each map with 100 pixels, create a contrasting background, and display a title.

The input XML format is simple: a pmap consists of one or more data sets (pdata elements), which in turn contain items with a corresponding value:

<pmap>
<pdata legend="W3C Counter">
   <item value="43.2">Internet Explorer</item>
   <item value="31.2">Firefox</item>
   <item value="10.7">Chrome</item>
   <item value="4.2">Safari</item>
   <item value="1.4">Opera</item>
</pdata>
</pmap>

A quick tour of Go's dict package

2010-08-13T20:33:00.003-04:00

The 2010-08-11 release of the Go programming language includes the dict package to talk to dictionary servers as defined by RFC 2229. Let's check it out.

First, here's a mimimal program that uses the Dial() and Define() functions to display a definition.

package main

import "net/dict"

func main() {
 c, _ := dict.Dial("tcp", "dict.org:2628")
 defn, _ := c.Define("wn", "go")
 for _, result := range defn {
    println(string(result.Text))
 }
}

This program does no error checking, and has three hardcoded bits: the server (dict.org:2628), the database ("wn") and word to lookup ("go").

For the next version, let's add some flexibility -- lookup items specified on the command line, and let's add some error checking and cleanup:

package main

import (
 "net/dict"
 "os"
)

func main() {
 c, neterr := dict.Dial("tcp", "dict.org:2628")
 if neterr == nil {
    defer c.Close()
    for _, word := range os.Args {
       defn, deferr := c.Define("wn", word)
       if deferr == nil {
          for _, result := range defn {
             println(string(result.Text))
          }
       }
    }
 }
}

Ok, great, what if you want to switch databases? You can ask the server. If you run the command with no arguments, it will list the available databases using the Dicts() function:

package main

import (
 "net/dict"
 "os"
)

func main() {
 c, neterr := dict.Dial("tcp", "dict.org:2628")
 if neterr == nil {
    defer c.Close()
    if len(os.Args) == 0 {
       dicts, dicterr := c.Dicts()
       if dicterr == nil {
          for _, dl := range dicts {
             println(dl.Name, dl.Desc)
          }
       }
    } else {
       for _, word := range os.Args {
          defn, deferr := c.Define("wn", word)
          if deferr == nil {
             for _, result := range defn {
                println(string(result.Text))
             }
          }
       }
    }
 }
}

One more update: let's add flags to specify the dict server and database. We'll also update the error processing to show the errors instead of failing silently:

package main

import (
 "net/dict"
 "flag"
 "fmt"
)
var (
 db      = flag.String("d", "wn", "Dictionary database")
 dserver = flag.String("s", "dict.org:2628", "Dictionary Server")
)

func main() {
 flag.Parse()
 c, neterr := dict.Dial("tcp", *dserver)
 if neterr == nil {
    defer c.Close()
    if len(flag.Args()) == 0 {
       dicts, dicterr := c.Dicts()
       if dicterr == nil {
          for _, dl := range dicts {
             fmt.Println(dl.Name, dl.Desc)
          }
       } else {
          fmt.Println(dicterr)
       }
    } else {
       for _, word := range flag.Args() {
          defn, dferr := c.Define(*db, word)
          if dferr == nil {
             for _, result := range defn {
                fmt.Println(string(result.Text))
             }
          } else {
             fmt.Println(dferr)
          }
       }
    }
 } else {
    fmt.Println(neterr)
 }
}

There you go--a robust, flexible dictionary client in a little over 40 lines of code.

Font Specimens with SVG

2010-04-16T23:12:00.010-04:00

These font specimens were created with SVGo, patterned after the specimens found in wikipedia

Here are the current and past defaults for Microsoft Office, Calibri and Times Roman

The original vs. the knock-off: Helvetica and Arial.

And finally, the canonical mono-spaced font Courier compared to Inconsolata, my current go-to monofont

Leaning Flowers

2010-03-24T21:11:00.001-04:00

Here is another variation of grain2. It's like the "flower" program, but it includes stems, and a porportion of "lean" (0 - full lean to the left, 100 - full lean to the right) is programmed in.

The variations were produced with this script:


for i in 0 25 50 75 100
do
  ./grain -n 150 -w 900 -h 200 -l 175 -nl $i > grain$i.svg
done

Grain 0

Grain 25

Grain 50

Grain 75

Grain 100

Here's the Go program



// grain -- flowers with programmable lean

package main

import (
    svglib "./svg"
    "time"
    "rand"
    "os"
    "flag"
    "fmt"
    "math"
)

var (
    width     = flag.Int("w", 500, "width")
    height    = flag.Int("h", 500, "height")
    ninter    = flag.Int("n", 75, "interations")
    stemlen   = flag.Int("l", 450, "stem length")
    stemwidth = flag.Int("st", 4, "stem thickness")
    numleft   = flag.Int("nl", 50, "percentage of left-leaning stems")
    thickness = flag.Int("t", 10, "max petal thinkness")
    np        = flag.Int("p", 15, "max number of petals")
    psize     = flag.Int("s", 30, "max length of petals")
    opacity   = flag.Int("o", 50, "max opacity (10-100)")
    svg       = svglib.New(os.Stdout)
)

const (
  flowerfmt = `stroke:rgb(%d,%d,%d); stroke-opacity:%.2f; stroke-width:%d`
  stemfmt = `stroke:green;stroke-opacity:0.3;stroke-width:%d`
)

func radial(xp int, yp int, n int, l int, style ...string) {
    var x, y, r, t, limit float64
    limit = 2.0 * math.Pi
    r = float64(l)
    svg.Gstyle(style[0])
    for t = 0.0; t < limit; t += limit / float64(n) {
        x = r * math.Cos(t)
        y = r * math.Sin(t)
        svg.Line(xp, yp, xp+int(x), yp+int(y))
    }
    svg.Gend()
}

func background(v int) { svg.Rect(0, 0, *width, *height, svg.RGB(v, v, v)) }

func random(howsmall, howbig int) int {
    if howsmall >= howbig {
        return howsmall
    }
    return rand.Intn(howbig-howsmall) + howsmall
}

func randrad(x, y int) {
    var r, g, b, o, s, t, p int
    r = rand.Intn(255)
    g = rand.Intn(255)
    b = rand.Intn(255)
    o = random(10, *opacity)
    s = random(10, *psize)
    t = random(2, *thickness)
    p = random(10, *np)
    radial(x, y, p, s, fmt.Sprintf(flowerfmt, r, g, b, float64(o)/100.0, t))
}

func init() {
    flag.Parse()
    rand.Seed(time.Nanoseconds() % 1e9)
}

func main() {
    svg.Start(*width, *height)
    background(255)
    svg.Gstyle(fmt.Sprintf(stemfmt, *stemwidth))
    var x, l, xe, offset int
    for i := 0; i < *ninter; i++ {
        x = rand.Intn(*width)
        l = rand.Intn(*stemlen)
        offset = rand.Intn(*width / 10)
        if rand.Intn(100) > *numleft {
            xe = x - offset
        } else {
            xe = x + offset
        }
        svg.Line(x, *height, xe, *height-l)
        randrad(xe, *height-l)
    }
    svg.Gend()
    svg.End()
}

Browser SVG Gallery - UI analysis

2010-03-21T10:20:00.002-04:00

Browser SVG Gallery
Originally uploaded by ajstarks

This grid shows four modern browsers: Chrome (5.0.307.11) , Firefox (3.6) , Opera (10.10) , and Safari (4.0.5) displaying SVG content generated by SVGo. The browser UI's have been muted to emphasize the content. Some observations:

1) Chrome: shows more "chrome" than any of the others. Lot's of head space with it's tab structure, with an emphasis on the omnibox, but with a minimal number of controls (back, forward, refresh)

2) Firefox: as minimal as it gets, except for the OS X furniture, only minimal tab controls are visible

3) Opera: lots of buttons with space for the address box, but all in a single row

4) Similar to Opera, but with fewer buttons (only back and forward). The big ol' Google box looks conspicuous.

Note that both Opera and Safari allow for all controls to be suppressed so that you are left with pure content:

Processing vs. SVG Go

2010-03-06T12:35:00.001-05:00

Processing vs. SVG Go
Originally uploaded by ajstarks

This screenshot compares developing the same graphic using Processing and the SVG Go Library.

Top row: from the left: Processing code, Go Code (in TextWrangler), Command lines to run the Go code

Bottom row: Processing output, Go-generated SVG rendered in Safari and Chrome

The workflow for Processing is familiar: enter code, hit the run button. For Go: edit/write code, move to a shell window, compile, open a browser.
For subsequent runs, just hit refresh in the browser to see the result.

Both methods support rapid prototyping and sketching -- the Go compiles are so fast it's almost the same as hitting the run button in Processing. As you can see the results are identical

In terms of code, both environments are similar -- there is almost a one-to-one correspondence, however SVG and Processing treat ellipse width differently.

I note that the syntax highlighting of graphics functions in Processing tells me at a glance what the program is doing -- this is less apparent with the Go program.

TweetFreq Go Language web Server (wtf): Browser, console, source

2009-12-09T23:03:00.006-05:00

wtf is a specialized web server that also acts as client to twitter. wtf acts as a "compiler" for the browser, responding to requests with optimized "pre-cooked" JavaScript canvas directives---think of it as assembly language for the code in the browser that interprets canvas tag data. For example:

C.beginPath();C.arc(799,167,8,0,p2,true);C.fill();

draw the markers. All computations like coordinate and object positioning is done by the wtf, not the browser.
Currently, wtf retrieves JSON from twitter. The XML results were over 2 times slower (with a slight loss of data, the JSON results lack the full name of the twitter user).

wtf takes about 5.6 seconds to build a tweetfreq from 10 users
(abcnews,nytimes,wsj,cbsnews,cnn,foxnews,techcrunch,techmeme,arstechnica,slashdot)
consisting of 500 tweets. The bulk of the time is waiting for the
twitter search engine to respond.

wtf responds to requests in this format:

 /users/{userlist}?b={date}&e={date}&c={count}&m={marker-width}&p={picture width}&l={line-width}&s={spacing}

which specify the users and search/graphics/layout parameters

/users/ajstarks,nytimes,%23golang,cnn?b=2009-12-05&e=2009-12-09&c=100&m=12&p=48&l=24&s=60

builds a visualization of the twitter update frequency for the users ajstarks, nytimes, cnn, and the mentions of the #golang hashtag during the period between December 5-9, 2009 (UTC), up to 100 tweets/user, with a marker width of 12 pixels, a picture width of 48 pixels,
line width of 24 pixels, with spacing of 60 pixels.

Planned updates include support for twitter lists in the form of

/lists/{user}/{listname}

(improved support for authentication in the http package would be
helpful here...)

Parsing JSON and ATOM Twitter Search Results in Go

2009-11-18T05:17:00.004-05:00

Parsing JSON and ATOM Twitter Search Results in Go
Originally uploaded by ajstarks

The top chart shows the time a Go program takes to search twitter, parse the results, and output the tweets, varying the number of search results from 10-100. The blue shows the results for JSON, the green for ATOM (XML).

Also depicted is the number of bytes to parsed, and the resulting parsing rate.

The conclusion is that JSON is more efficient; HTTP rates are constant, but JSON requires less data, with less complex data structures to unmarshal. Both methods deliver identical results:

ts -f json -n 10 '#golang'

RT @koizuka: RT @tokuhirom: #golang は C/C++ のかわりにつかうというよりは。python のかわりに使うという領域の方がおおきいんだとおもう
RT @tokuhirom: #golang は C/C++ のかわりにつかうというよりは。python のかわりに使うという領域の方がおおきいんだとおもう

Here is the program: it demonstrates command line parsing, error checking, http processing and unmarshaling both JSON and XML.


// ts -- twitter search
//
// Anthony Starks (ajstarks@gmail.com)
//

package main

import (
 "fmt"
 "http"
 "io"
 "io/ioutil"
 "flag"
 "os"
 "xml"
 "json"
)


type JTweets struct {
 Results []Result
}

type Result struct {
 From_user string
 Text      string
}

type Feed struct {
 XMLName xml.Name "http://www.w3.org/2005/Atom feed"
 Entry   []Entry
}

type Entry struct {
 Title  string
 Author Person
}

type Person struct {
 Name string
}

type Text struct {
 Type string "attr"
 Body string "chardata"
}

var (
 format   = flag.String("f", "atom", "Output format (json or atom)")
 nresults = flag.Int("n", 20, "Maximum results (up to 100)")
 since    = flag.String("d", "", "Search since this date (YYYY-MM-DD)")
)

const (
 queryURI  = "http://search.twitter.com/search.%s?q=%s&rpp=%d"
 outputfmt = "%s \u27BE %s\n"
)


func ts(s string, how string, date string, n int) {

 var q string
 if len(date) > 0 {
  q = fmt.Sprintf(queryURI+"&since=%s", how, http.URLEscape(s), n, date)
 } else {
  q = fmt.Sprintf(queryURI, how, http.URLEscape(s), n)
 }

 r, _, err := http.Get(q)
 defer r.Body.Close()
 if err != nil {
  fmt.Fprintf(os.Stderr, "%v\n", err)
  return
 }
 if r.StatusCode == http.StatusOK {
  switch how {
  case "atom":
   readatom(r.Body)
  case "json":
   readjson(r.Body)
  }
 } else {
  fmt.Fprintf(os.Stderr,
   "Twitter is unable to search for %s as %s (%s)\n", s, how, r.Status)
 }
}

func readatom(r io.Reader) {
 var twitter Feed
 err := xml.Unmarshal(r, &twitter)
 if err != nil {
  fmt.Fprintf(os.Stderr, "Unable to parse the Atom feed (%v)\n", err)
  return
 }
 for _, t := range twitter.Entry {
  fmt.Printf(outputfmt, t.Author.Name, t.Title)
 }
 
}

func readjson(r io.Reader) {
 var twitter JTweets
 b, err := ioutil.ReadAll(r)
 if err != nil {
  fmt.Fprintf(os.Stderr, "%v\n", err)
  return
 }
 jerr := json.Unmarshal(b, &twitter)
 if jerr != nil {
  fmt.Fprintf(os.Stderr, "Unable to parse the JSON feed (%v)\n", jerr)
  return
 }
 for _, t := range twitter.Results {
  fmt.Printf(outputfmt, t.From_user, t.Text)
 }
}

func main() {
 flag.Parse()
 for i := 0; i < flag.NArg(); i++ {
  ts(flag.Arg(i), *format, *since, *nresults)
 }
}

Go vs. Curl for twitter search

2009-11-16T21:19:00.001-05:00

Go vs. Curl for twitter search
Originally uploaded by ajstarks

Twitter Search Grid: #artandcode #4

2009-11-16T21:17:00.001-05:00

Twitter Search Grid: #artandcode #4
Originally uploaded by ajstarks

TweetFreq: Scobleblog followers

2009-11-16T21:16:00.001-05:00

TweetFreq: Scobleblog followers
Originally uploaded by ajstarks

TwitGrid: Public timeline

2009-09-24T00:58:00.001-04:00

TwitGrid: Public timeline
Originally uploaded by ajstarks

TweetFreq: 10 Most Influential Journalists on Twitter -- 2009-09-11

2009-09-19T23:31:00.003-04:00

TweetFreq: 10 Most Influential Journalists on Twitter -- 2009-09-11
Originally uploaded by ajstarks

TweetFreq: 10 Most Influential Journalists on Twitter -- 2009-09-04

2009-09-19T23:31:00.001-04:00

TweetFreq: 10 Most Influential Journalists on Twitter -- 2009-09-04
Originally uploaded by ajstarks

TweetFreq: @scoopingthenews 10 most influential journalists on Twitter: 2009-09-18

2009-09-19T23:18:00.002-04:00

TweetFreq: @scoopingthenews 10 most influential journalists on Twitter: 2009-09-18

@scoopingthenews 10 most influential journalists on Twitter, Sept 18, 2009 (sorted by Influence score)

2009-09-19T23:17:00.001-04:00

@scoopingthenews 10 most influential journalists on Twitter, Sept 18, 2009 (sorted by Influence score)
Originally uploaded by ajstarks

10 most influential journalists on Twitter, 2009-09-18 as defined by @scoopingthenews www.scoopingthenews.com/2009/09/10-most-influential-journ...

Data shown is the influence score (50% RT rate, 25% followers, 25% tweets)

10 most influential journalists on Twitter, Sept 18, 2009 (sorted by Influence score)

2009-09-18T16:16:00.001-04:00

10 most influential journalists on Twitter, Sept 18, 2009 (sorted by Influence score)
Originally uploaded by ajstarks

Documenting Processing Work

2009-07-15T16:52:00.013-04:00

I've created this deck to document my Processing work

How do you document your Processing work? (Flickr sets and groups seem to be popular)

Official Google Blog: Google accounts on Twitter

2009-07-12T17:22:00.004-04:00

Official Google Blog: Google accounts on Twitter here is a visualization of how much the Google Twitter accounts are used (posting frequency over the past week)

Do one thing -- 30 years between Unix and iPhone

2008-03-12T18:53:00.004-04:00

A similar philosophy is stated between classic UNIX and iPhone development:

"Make each program do one thing well, to do a new job build afresh rather than complicate old programs by adding new features "

-- Doug McIlroy, Bell System Technical Journal, 1978

"Remember that you application should address a specific need -- most of the built in applications to one thing very, very well, instead of trying to do many things only moderately well"

-- Vicki Murley, Key Practices for iPhone Application Development", 2008.

The Kindle

2007-11-20T14:02:00.001-05:00

With great fanfare Amazon just announced its dedicated ebook reader -- the Kindle. The web is "ablaze" with coverage: starting with the Newsweek cover story, "The Future of Reading", and continuing with the Kindle Product Page.

Overview

Ars Technica, CrunchGear, Gizmodo, PaidContent, and BoingBoing, have reviews. And there is no shortage of commentary:

Wired has a brief comparison of the Kindle vs. the Sony Reader, and Rothman's Teleread Blog , where I first heard about the device months ago, comments on the design and format issues.

The feature set is unremarkable (essentially the same hardware specs as the Sony reader; they both sport the 166.67 dpi E-Link screen), with the standard ebook functions: zoomable text, search, annotation, -- but what distinguishes the Kindle is the connection to Amazon -- easy ordering and downloading content from the Amazon via always-on EVDO wireless networking. The connectivity comes for free with the device, but subscriptions fees apply to magazines, newspapers, and surprisingly, blogs.

The Amazon connection also brings 90,000 titles at introduction--nice---but this leads to two questions: what about the other millions of titles, and at what rate will the Kindle-compatible titles grow. It seems to me that cutting deals with individual publishers is not going to cut it -- this is not Steve Jobs dealing with a few record labels -- what is needed is a simple method for generating content for a device, so that both mainstream and long-tail content are available. Today this is answered by PDF, and the .epub format also promises to fulfill this need -- neither of these formats is currently supported natively on the Kindle. The supported file formats are: Kindle (.azw), text (.txt), unprotected Mobipocket (.mobi, .prc), Audible (.aa) and MP3 (.mp3). Amazon supplies (for fee) conversion of content via an email address assigned to each Kindle, and other conversions can be managed via its Digital Text Platform.

The Design, Marketing and Positioning

The design of the Kindle is fugly but seemingly functional. It includes a scroll-select wheel, and a physical keyboard---this feature alone breaks the "bookiness" that the device is striving for. But a physical keyboard is probably required given that the slow responding E-ink screen probably won't adequately support an on-screen keyboard. It will be interesting to analyze the UI and general user experience of the interface, but the reviews and videos to date don't provide enough detail yet. I'm hoping any UI or functionality bugs can be fixed via software updates.

In the promotional videos, Bezo's introduction, and the Newsweek piece, there is an emphasis on the act of reading -- note that the spokespeople recruited to push the device, Toni Morrison, Michael Lewis, Daniel Handler are authors, not geeks. I also note that the demo video has a "Kindle Dude", similar to the "iPhone guy" to introduce its features -- but unlike the Apple personality, K.D. gets out of the studio, putting the device through its paces in the park, coffee shop, the airport, and at home.

The Rocket ebook , REB 1200 , Sony Librie , Sony Readers, Irex Iliad, Kindle. It feels like the period between 1977-1981 when the personal computer marketplace was shaping up. Lots of machines that taken individually had interesting features, but none that reached the sweet spot that defined a product category.

Kindle, sadly, you are no IBM PC.

Although it feels like a device Steve Jobs would have rejected, the basic concepts are sound (take-your-content with you, nice screen, easy connectivity, document conversion and self-publishing), there is so much that can be improved: the design, DRM/lock-in, a more open platform.

Where is my device that can order content from not just from Amazon, but from other outlets as well, one that can easily and beautifully display any blog, not just selected ones (yes, I know the limited web browser fixes this somewhat, but what about a Google Reader-like application built into the device? Whoever develops a beautiful, universal container for my purchased, commercial, web, and personal content will win, Kindle points the way, but does not quite reach it.

Programming Infographics with Processing

2007-11-13T19:51:00.001-05:00

In learning Processing, it occurred to me that it could be used to produce visuals previously done via traditional tools like spreadsheets and presentation software. Using Processing means that layout decisions are programmed in -- for example the placement of pictures to annotate a barchart; instead of tediously placing each graphic, the placement is automatic. To test this, I used this graphic, produced in Keynote as a basis:

Here is the same data, but produced by code in Processing:

In comparison to the original Keynote graphics, I toned down the bar chart color significantly, and also plotted the products pictures as data, instead of axis labels.

The Data

Of course, visuals must be data driven, and Processing can take input from files; in this case, tracking stock price with other events: Apple Stock price to personal purchases,(incidentally this kind of visualization can based from other things as well, Google products. vs. stock price, Fat content of foods, IMDb ratings, or a drug companies product launches vs. stock price, etc.

The relevant stock data can be obtained from Yahoo Finance -- select the relevant time interval, and then download the data as a CSV file. Note that these visualizations work best if there is not too many data points.

The data file that my Processing code uses consists of these fields:
date, stock price, picture file, product name (with the exception of the first line, used to specify the title

for example:

Apple Purchases vs. Stock Price
2002-04,12.4,tipb.jpg,Ti PowerBook
2002-08,7.38,jaguar.png,Jaguar
2003-04,7.11,itunes.jpg,iTunes
...
2007-10,172.75,leopard.jpg,Leopard

The code

The code (166 lines) is broken down into these sections:

Read in the data, with sanity checks
Define major object locations based on percentages relative to the graphic's width and height (for example, the title is defined to be located at 7% of the overall height)
Define scaling factors
Show Title
Show the Y axis label, and the Grid
Plot the data

The display of titles, bars, data values, labels, grid, images, names, and dates, can be independently toggled on or off, controlled via boolean variables.

Observations

The model visual has the traditional barchart, but invoking the option to remove the grid and bars allows the the product pictures and the stock values to serve as data points:

Here is a slightly different twist: Relating the introduction of "iconic" Apple products with the stock price:

Having complete control over every aspect of the graphic is freeing --- with Processing, you are not constrained by what a standalone software package can do --- if you can conceive it, you can easily program it in without waiting for Apple, Adobe, or Microsoft to add the feature or behavior. (Another useful side effect is that your graphics won't look like everyone else's).

Processing encourages this interactive "sketching" style. (in fact instead of "code" and "graphics", the Processing environment refers to these collectively as sketches.

I note that using percentages and not hard-wired "magic" numbers means that the graphic will scale when the height and width are adjusted. This means that you can pick the correct aspect ratio for a given data set and the various graphic elements remain in their proper places. Also, because each section of the graph is drawn and placed independently, re-arrangement and experimentation is easy: Need to move the product labels -- adjust one number -- hit the execution command, -- boom. Instant gratification.

I plan to continue this with other types of data -- See the flickr set for more.

Leopard: Fixing the translucent Menu

2007-10-30T20:32:00.000-04:00

On Mac OS X Leopard, the top menu is now translucent. I hate it. The effect is especially bad if you use a single color background:

To fix this I needed a very precise desktop background that has a strategically placed white stripe at the top. Processing to the rescue:

int menuHeight=22; // Height of the top Menu
size(1280,854); // size of my PowerBook screen
noStroke();
fill(255);
rect(0,0,width,menuHeight);
fill(58,103,173);
rect(0,menuHeight,width,height-menuHeight);
save("menufix.jpg");

After this runs, use the System preferences to use "menufix.jpg" as your background.

Leopard: Fixing the 3D Dock

2007-10-30T20:01:00.000-04:00

Some have complained about the appearance of the Dock on Leopard, and offered a bit of command line magic to fix it. Below is a shell script to automate the process of turning the dock "shelf" on and off.

When called as "flatdock" you get the non-3D version of the dock;

Called as "3ddock" the shelf is restored:

To install, place the script in your PATH, call it flatdock, make it executable, and then hardlink to 3ddock.


#!/bin/sh
state="YES"
name=`basename $0`
case $name in
3ddock) state="NO";;
flatdock) state="YES";;
esac
defaults write com.apple.dock no-glass -boolean $state
killall Dock

Processing...

2007-10-10T16:12:00.002-04:00

The Processing Environment is a nice one --- it's quite useful for prototyping graphical and typographical ideas.

The PDE (Processing Development Environment) is a study in simplicity and efficiency -- it abstracts much of the underlying Java-environment, and allows you to simply work with the expected graphical primitives (points, lines, rectangles, ellipses, curves, etc.), and coordinate system, using your own code to control the formation and interaction with the graphic.

I really like PDE -- it works the way you would expect: enter some code, press the run button, the result is displayed -- instant gratification. The environment supports the notion of "sketching" -- (in fact works in progress are called sketches). Want to change a color? tweak an RGB value in the code window, see the results immediately. The same applies to other areas of your graphic like sizes, primitives, loop indices, etc.

A new book on Processing has just been published: Processing: A Programming Handbook
for Visual Designers and Artists by Casey Reas and Ben Fry. The processing web site has excerpts and all the sample code from the book.

The New Amazon MP3 Store, Step by step

2007-09-25T14:29:00.000-04:00

Amazon has just launched their DRM-free MP3 store. Here is a step-by-step walkthrough on my Mac (for the impatient, it works, but in terms of usability, selection, and convenience, it's no iTunes, but I welcome the competition) One oddity: although you eventually wind up with MP3 files in iTunes, the process uses a standalone app to manage the downloads; you get a open a .amz file, and then the real MP3 is downloaded.

Step 1. Make your selection via the website, press the Buy MP3 button:

Step 2. Download and install the Amazon downloader application (one time only)

Step 3. Get confirmation of the purchase (via the web, and email)

Step 4. Double-click the .amz file, the Amazon downloader puts the MP3 file to your hard disk

Step 5. Press the "Reveal in Finder" button, double-click to import into iTunes -- this includes album art and appropriate metadata.
Step 6. Play in iTunes:

OLPC: 1+1

2007-09-24T15:05:00.000-04:00

The One Laptop per Child Project now allows individual purchase: Give one, Get one.

Stephen Fry on what distinguishes Apple

2007-09-19T19:13:00.000-04:00

Fry's long essay on iPhone competitors

That’s why Apple is Apple, they have people there (and of course it comes from the top) who say - “woah, not good enough, not cool enough, not simple enough, not fun enough, not sexy enough, not clever enough, not useful enough”.

Another use for the iPhone: redeeming coupons

2007-09-18T11:56:00.000-04:00

The other day, when I picked up "The Universal Principles of Design", I used a 20% off coupon from Borders. Instead of using a printed coupon, I handed my iPhone to the cashier, which had the PDF of the coupon from my email.

I wonder if the same thing will work for boarding passes?

Trends 2007: ERP 2.0

2007-09-17T19:39:00.000-04:00

Companies like SAP and Oracle have lock on ERP systems for enterprises. Why is there not Open Source alternatives to these large and comprehensive packages?

Trends 2007: The Rise of Apple

2007-09-17T16:08:00.001-04:00

Apple has firmly established itself as "the new cool". It's obvious that the company has hit its stride, with a relentless march of new products and continued evolution of its entire product line.

There is something to be said for a company that continues to delight its customers, create new classes of products, inspire such devotion (when was the last time you heard of a Windows fanboy), and manage to garner so much press, both mainstream and blogosphere.

I personally made the transition to Apple products in early 2002, and have not looked back. My personal five-year Apple timeline is:

2002/03: 15 inch Titanium PowerBook G4
2002/08: Mac OS 10.2, Jaguar
2003/04: First iTunes purchase ("Look to the Rainbow", Dinah Washington)
2003/10: Mac OS 10.3, Panther
2004/08: eMac, (college machine)
2004/08: 3G iPod, 15Gb
2005/03: 15 inch Aluminum PowerBook G4
2005/06: Mac OS 10.4, Tiger
2005/09: 4G Video iPod, 30Gb
2006/12: MacBook (college machine)
2007/06: iPhone
2007/08: MacBook (college machine)

During that period, Apple stock has grown from about $12.00 to over $130, I have purchased three generations of iPods, three versions of computers in two families (PowerBook G4, eMac, MacBook), three major operating system revisions.

Trends 2007: Scripting Language Web Frameworks

2007-09-17T15:49:00.000-04:00

As a platform for developing web applications, scripting language-based frameworks like Ruby on Rails and Django are gaining in popularity and mindshare. Rails is famously used by 37signals to produce its suite of products, Basecamp (project management), Campfire (group chat), Backpack (information organization), and Highrise (contact management). The approach advocated by this framework is rapid development, convention over configuration, and embracing the web including the REST architectural style.

Django is interesting to me because it's based on python, embraces the notions that URLs are meaningful, and like RoR, built based on real-world experience and issues (the framework built for perfectionists with deadlines), and uses a MTV (Model/Template/View) worldview that allows designers to concentrate on look and layout, leveraging data provided by developers.

Recently there has been a move away from bigger, heavier weight frameworks like J2EE, to more agile frameworks. Some have even said that Java is like COBOL -- your dad's language. This move is described in "Better Web Application Development", by Sean Kelly, where J2EE gets a real smackdown in the face of Ruby on Rails, Zope/Plone, TurboGears, and Django. This is a great example of "evidence-based" architecture.

Trends 2007: iPhone

2007-09-17T13:30:00.001-04:00

Yes, I was one of the early adopters (Los Angeles Century City Apple Store, 10 minute wait in line), I have an iPhone receipt dated June 29, 2007, and I have no regrets, and my $100 "gift" is also welcome.

This device illustrates everything I like about Apple products: a delightful, human-centric user-interface, a clean design that meets the "emotional" test -- you want to use it because it's fun to use, novel technology, and a hackable platform. Beyond the current round of hacks to route around AT&T lock-in, the iPhone as a web platform is extremely interesting. Here's Apple's developer strategy as I see it: Create a standards-based web platform, with mechanisms to optimize for the iPhone within that framework, followed by leveraging the existing Dashboard widget framework for "native" applications. You only need compare Stock and Weather iPhone applications to their desktop counterparts to see where this is going.

From a person that rarely even carried a mobile phone, this is the device that has me happily and easily texting my college-aged kids, showing my and taking photos, looking at videos, listening to music and podcasts (at home and in the car), browsing the web, and even experimenting with developing content.

Initial reports from analyst firms like Gartner advised caution on adopting the iPhone in the enterprise, but subsequent reports moderated that stance somewhat as more enterprise-friendly features or versions were predicted. In my view Apple should resists the enterprization of the iPhone, and enterprises should embrace the iPhone as is, and can leverage it now by taking advantage of its web and podcast capability --- it's a platform, just like the PC, and frankly the the arguments against sound like the old stuff I used to hear in the 198os about personal computers.

Trends 2007: Web 2.0 Exemplars

2007-09-17T10:38:00.000-04:00

At this point the notion of Web 2.0 is firmly established, if not endlessly debated and re-debated. Here is my take on organizations/sites that exemplify modern web technology and business models:

Google: the so called "heart of the web"
Amazon: leveraging infrastructure to provide a service-based computing grid
37Signals: Leveraging scripting language frameworks to create useful, web-centric applications
Wikipedia: proving that collective knowledge is useful
Flickr: combining photos with community on a web-scale platform
Facebook: connecting people and what they care about
del.icio.us: Preserving and sharing the journey through the web
digg: what is cool and interesting now
salesforce.com: challenging the big boys on web-based applications and platforms

Trends 2007: REST

2007-09-17T09:30:00.000-04:00

Much has been written about REST, the architectural style for distributed, networked applications, described by Roy Fielding. My first exposure to the concept was in from Paul Prescod's article published in 2002: Second Generation Web Services. After reading that piece, I "got it", and I've been an advocate ever since. Fast forward to 2007, and we have the the publication of "R&R" -- Richardson and Ruby's, "Restful Web Services". Along the way we have Ruby on Rails embracing REST along with Microsoft, Google, and others.

There has been a good bit of REST-theory and debate on the net, but also note that Roy Fielding is scheduled to provide some perspective at the next ApacheCon with the talk "A Little REST and Relaxation".

If you want to get beyond the theory and see real-world REST semantics, principles, and working code, you need go no further than the Atom Publishing Protocol.

What does it all mean? REST is at this point the default method for doing real "web services" -- any modern API is typically RESTful to some degree. Traditional IT shops and enterprise vendors should take heed and begin to adopt these methods.

Here's my linkdump on REST (maintaining these is what prompted me to use del.icio.us)

(thanks to Sam Ruby for the REST logo)

Trends 2007: Syndication

2007-09-15T17:02:00.001-04:00

The notion of syndication is now firmly entrenched in the post Web 1.0 world --- anything that matters can now has a feed; from web pages, blogs, news items, photos, you name it. Once you begin to use syndication (via built in browser facilities, web-based reader, or dedicated reader, it's impossible to go back. The ability to sift through a mountain of information (of your own choosing, much of it discovered from other feeds) is addictive --- in a good way. You can move at your own pace, fast or slow, through your own carefully constructed information space, consuming items as diverse as: Bullet Points, the Ten Greatest Advancements in Fast Food and Greenspan Is Critical Of Bush in Memoir.

Syndication changes the way you consume information -- you are now in control of how they receive your information via the subscription, and in turn you consume information on your terms --- by headline, by, summary, full. Moreover, the information space is exportable, portable and sharable via OPML files.

I believe that syndication is largely unexploited in mainstream IT shops -- however this could be changing, in fact some have elevated syndication to its own architecture.

Trends 2007: Megadata

2007-09-13T11:40:00.000-04:00

Today it's obvious we now have massive computing power, the amount of data we work on even for personal use is growing larger. How we will process the volume?

Joe Gregorio coined the term "megadata", that is large data sets, where traditional relational techniques (triggers, transactions, joins, are NOT used). For example, Google's use of BigTable.

Moreover, the common themes here are that the "structure" of structured data is different: it is inherently distributed, and de-normalized, (which brings to mind the Cal Henderson statement on normalized data)

Trends 2007: Next generation User Interfaces

2007-09-13T11:32:00.000-04:00

Starting with the interface in Minority Report, and culminating in the Apple iPhone, we are finally starting to see some movement away from the traditional keyboard/mouse/windows style of interaction to richer ones.

The grandaddy of these is Jeff Han's Mutitouch (which I saw demoed at Etech, amazing). Microsoft follows this with -- the Surface, and of course the iPhone. On a different note is the BumpTop -- which seeks to take the desktop metaphor to the next level my replicating real world structures like stacks coupled with an interaction model that duplicates real-world physics.

This notion of movement and direct manipulation makes for a more pleasant experience and is one reason for the appeal of the iPhone -- the idea of "touching your music" somehow brings you closer to the thing at hand -- finding something good to listen to -- in this sense the device is out of your way.

Trends 2007: OLPC

2007-09-13T11:11:00.000-04:00

The One Laptop Per Child project is interesting on a number of levels, its aim is to bring computing to a new set of people without the constraint of "installed base" and in doing so has the potential to improve computing for the rest of us. In effect the project asks the question -- "what should computing be like" instead of "we have this set of computing systems along with its built in constraints and conventions that you must deal with.

The OLPC has challenged the established PC players on two fronts, price, and function, and has met these challenges with a device that costs significantly less than current products, with features like an amazing dual mode display with higher resolution than 95% of the laptop displays on the market today (7.5 inch, 1200×900 pixel, TFT screen and self-refreshing display at 200 DPI), with 1/7th the power consumption. Add to this a new security model, mesh networking, and new UI, and you have a device that is pushing the boundaries of what a computing device should be.

For more on the OLPC design see: "How the OLPC Machine was Designed" this video which include details and demos of the software.

Trends 2007: Parallel and Concurrent Computing

2007-09-13T10:54:00.000-04:00

Following up and related to cloud computing is the notion of parallel and concurrent computing. (Rob Pike points out the distinction of these concepts in Advanced Topics in Programming Languages: Concurrency/message passing in Newsqueak)
He states;

Concurrency has nothing to do with Parallelism they are actually separable ideas and thats important

Another major point of this talk is:

The world is concurrent, but computers are not!

and that language support for concurrency is important. The increased attention to languages like Erlang, which has built-in support for concurrency, in indicative of this trend.

What does this mean? We can break out of the sequential/synchronous world and begin to think about problems in more natural ways. Pike makes the point that concurrency should not be something hariy and messy that "you just have to deal with", but concurrent programs can be "much cleaner and more beautiful" than their sequential counterparts.

Finally, Google has shown the world that for a certain class of problems, working it in parallel environment is highly effective, and using infrastructure, programming environments, and languages like Sawzall, and MapReduce makes it possible.

Trends 2007: Cloud Computing

2007-09-12T18:01:00.000-04:00

This trend is not so new, but has gained a lot of currency lately, from Google pushing both apps and data in the Internet to Microsoft responding in kind.

One way to summarize the two approaches is to note quotes from Google CEO Eric Schmidt:

Web 3.0 will ultimately be seen as applications that are pieced together which are relatively small, the data is in the cloud, apps can be run on any device (PC or mobile), apps are very fast, very customizable, apps distributed virally by social networks and email, not store bought. It's a very different application model from the mainframe era and PC industry, likely to be very large, low barrier to entry, new generation of tools being introduced by Google and others make it relatively easy to do, solves a lot of problems, and works everywhere.

Compare these statements to Ray Ozzie in his recent analysts meeting:

Seamless Office scenarios that span the PC, the Web and even the phone. Documents that go wherever you want them, news scenarios, sharing scenarios, meeting scenarios, note-taking, presentation scenarios that use PCs for they're really good for for document creation and editing and review. That use the web for what it's really good for: publishing and sharing and universal access. They use the phone for mobile access that extends your PC-based and Web-based activities to wherever you go....

Finally for developers, the services opportunity lies really in the breadth and character of the this new type of platform. At the back end, the promise of services for developers is the promise of utility computing in the cloud...

Both of these quotes reflect that applications and data are not just local, Google embracing the "everything in the cloud" approach more aggressively, looking to unseat the PC-centric status-quo, with Microsoft, still clinging a bit to its history and heritage, trying to have it both ways.Nevertheless, having data in a central place has large implications: chief among them is location independence, and as alluded to my both Google and Microsoft above, device independence.

Update: Cap Gemini is endorsing Google Apps

Add to this the notion of massive data centers, used by Google, copied by Microsoft, with Sun attempting to commericalize the concept. Amazon is making waves with its S3 and EC2 services, leveraging existing capacity to provide storage and computing power at very low barriers of entry.

This begs the question for individuals, startups, and large enterprises: how soon will true utility computing be a reality?

What's on my mind: Trends 2007

2007-09-12T17:36:00.000-04:00

I Recently a took stock of items in the world of technology and computing that were occupying my attention.

Looking at things like my feed subscription list, I attempted to boil the list to a managable few -- to further capture the ideas/items, I created a presentation: called "Trends 2007".

Of course this kind of introspection is not new, O'Reilly has his Radar, and recently people like Sam Ruby in his "Long Bets" posting have done similar exercises.

Here is the list; I plan to write about each of these in subsequent posts:

A Piece of my Mind?

2007-09-11T19:27:00.000-04:00

I'm coming late to the game of public blogging, but the time feels right. This will be my place to share -- to "give you a piece of my mind" -- not in the sense of raving or railing, but to show, to give something I know, and hopefully continue to learn.

Also, this will be a place to keep ideas from rotting.