collective intelligence

as a software engineer who works in the e-commerce space, i am not usually impressed by the positive ways a site works.  most sites are no more complex than an insurance salesman who takes their customer though a long profiling exercise, in an effort not to select the right product but to know how to best pitch the basic set of products he would have suggested before the interview.  there is one site the continuously impresses me, amazon the largest ecommerce site in the world and the one site that seems to be able to anticipate my interests pretty well.

i got to thinking about this when i saw the suggestion for the movie "in the line of fire".  clint eastwood plays "frank horrigan", a secret service agent with a long history and a jaded past.  through the movie there are scenes with a small hotel in DC i stayed in long long ago, and another in the westin bonaventure in LA where i stayed only long ago.  i begin to empathize for some of what clint's and john malkovic's  (the assassin, mitch leary, horrigan is trying to stop) are feeling as they play their cat and mouse.  they both have are aging and have histories that drive them.

horrigan, like many of the eastwood characters over the years, delivers great one-liners as the action unfolds.  one is, "i know things about people", which he uses through-out the film to explain how he can read into the behaviors of the people around him.  the other is after he tells a female agent (renee russo) that her character is window dressing out to court the female vote, she asks him what demographic he represents and he replies:

let's see... white, piano-playing heterosexuals over the age of fifty. there ain't a whole lot of us, but we do have a powerful lobby.

this always makes me wonder which demographic i represent.  the same thing that happens when i use amazon, which tries to know me and compare me to others to help make suggestions.  the idea is that the collective intelligence of the crowd being the best indication of the behavior or desires of the individual.  if others like me enjoy things then i should too.  but to be good at this they need the ability to put people into the groups and then use what they know about all the people in that group to make projections on individual behavior.

the ways you know things about people are to ask them questions and then to watch their behaviors.  amazon knows where someone lives if they have delivered purchases.  from this they get a zipcode that ties into a demographic sample, telling them the income level, ethnic profile, education level and political party affiliation if not of the individual, than the community around them. all good information, but not as good as the actions of the individual.  i for example are not a good match of either of the places i live; well maybe expat-malaysia but not exit-8.

so they also watch what someone does.  what does the user view?  what do they buy?  what items do they put into a wish-list? or what do they rate, even if they didn't buy it on the site?  this has all become standard ecommerce marketing behavior.  smart engineers use huge amounts of data to target people because they "know things about people".  apple uses former purchases, and itunes match information to make genius recommendations.  look at yours, mine are spooky good at knowing what i like.

all of this has me thinking, what do you need to figure other people out.  i know you start by asking questions, and then you watch behavior.  if you believe the house-ism that everyone lies, then you need to consider that too.  people may lie, they may spin, but if you watch carefully and use enough data, you can figure them out and be able to make projections.

so, what do you ask first?  how much data do you need?

what would you need to figure me out?  what about the first 30 buckets i would put myself into?

• male
• late-forties
• european-american
• liberal-arts education
• deep reader
• software engineer
• fallen catholic
• ENTJ
• east-coast
• traveller 
• asia
• father
• married and divorced
• libertarian free-thinker
• triathlon
• lifeguard
• inked
• beach-bum
• craft beer
• ethnic food
• non-halal
• cook
• leatherman
• iphone
• macbook air
• crumpler
• evolutionary theory
• anthropology and psychology
• politics and economics
• open-sharing

maybe i cheated keeping the list to 30 by doubling up a few, but there is the behavior you need to watch.  those are the clues on what is important and how life is approached.  i wanted to go back and reorder things.  it felt like there might be a ranking inherent to the list, but i didn't reorder because this is the way it came out of my head.

as i look at the list, i realize that if i met a new person on the beach i would share most of this, the rest they might be able to see if they looked.  but is this enough for them to know things about me?  could they make a projection on my interests?

apple knows the shows i will buy.  amazon can suggest books and music to me.  but if asked, which demographic would i represent?

let's see... white, open-minded metrosexual tattooed book-worms near the age of fifty.  there really aren't a lot of us, and i like it that way.