20 August 2010

Digital IQ?

Filed under Articles

If you haven’t seen it, there was a report that was released yesterday from NYU Stern and GWU Business Schools that sets out a methodology for determining the Digital IQ of Senators of the United States, and then proceeds to do so. You can find it here. They even go so far as to declare seven Senators as Digital Geniuses.

Unfortunately, the methodology is not well described in the report, but from the information available, it appears to be a bit shallow. The good news is that the researchers have asked for comments on it, so here goes. First, the methodology as described in the report (pg 4):

Facebook – 25%:

  • Presence
  • Number of Likes
  • Like Growth

Twitter – 25%:

  • Presence
  • Followers
  • Velocity of Tweets
  • Follower Growth

YouTube – 25%:

  • Presence
  • Number of Uploads
  • Number of Channel/Upload Views

Online Buzz: Blogs – 12.5%:

  • Velocity of Mentions on Blogs and Other 2.0 Sites
  • Sentiment

Site Traffic: – 12.5%:

  • Annual and Monthly Unique Visitors
  • Number of Visits

Not Enough Information

I’d really like to see the raw numbers and methodology here so I can better understand what’s going on.

  1. What is the scale that is being used? Clearly, they have established the Digital IQ to align with actual IQ numbers in terms of designating individual Senator’s capabilities, eg average is 100, over 140 is genius, but is this established by normalizing the distribution or is there a set scale this is being compared against?
  2. What is velocity? I can guess that it is the number of Tweets (or mentions) per time period, but it’s a term I’ve haven’t run across previously (perhaps I just haven’t been looking at the research closely enough).
  3. During what time frame was this analysis made?

Analysis of the Methodology

The self-stated goal of the study is (pg 4):

Digital IQ = A More Robust Democracy

Our thesis is that digital competence provides an opportunity for senators to authentically engage and mobilize voters and constituents. Key to managing and developing competence is an actionable metric. This study attempts to quantify the digital competence of the 100 U.S. senators. Our aim is to provide a robust tool to diagnose digital strengths and weaknesses and prioritize incremental investment in digital.

Now hold on a second. It seems to me that this methodology is primarily based on eyeballs, the traditional media gauge of effectiveness — to be explicit, the more people that see your stuff, the better your chance of converting them. Unfortunately, this is neither the goal nor the correct gauge to be applying if you are accurately attempting to assess a Senator’s effective ability “to authentically engage and mobilize voters and constituents.” Effective use of social media is about connecting, having conversations, and engaging in meaningful ways.

The majority of the factors in the methodology are nothing more than measures of how traditional campaign tactics have been applied to the digital world:

  1. Presence on Facebook, Twitter, and YouTube is important, but it only means you’ve shown up to the party – it doesn’t mean you know how to dance.
  2. Number of followers or likes (and the growth of these) is not much different from traditional polling methods. Are you making a lot of noise? People will follow you – sometimes they may not like you, but they want to know what you’re up to – there’s no commitment to even read anything afterwards. That measure in and of itself is almost meaningless. (In all fairness however, it is a mandatory pre-condition for being able to engage meaningfully.)Note: This shows the Facebook metric to be completely irrelevant to meaningful engagement – making only 75% of the Digital IQ valuable.
  3. Velocity of Tweets and Number of Uploads on YouTube are solely about the Senator’s ability to publish. Many of them will simply their press release rss feed to Twitter and push the same information through a new channel. This is not indicative of engagement.Note: This moves the Twitter metric into the same category as the Facebook metric – making only 50% of the Digital IQ relevant.
  4. The Online Buzz: Blogs section references a candidate’s ability to get press (not in the traditional sense, but it’s still getting written about) and takes into account sentiment – which I assume means if the writing is positive or negative about them. This has nothing to do with their ability to meaningfully engage their constituents and in fact doesn’t even measure anything that they would have to actively do themselves.Note: Making 37.5% of the score relevant to the stated goal.
  5. Site Traffic: This is web 1.0. It is possible to engage site visitors in meaningful engagement, but there is no measure of that going on here.Note: 25% relevant.
  6. The only factor in the methodology I have not berated is the Number of Channel/Upload Views on YouTube. Now this is not a complete metric for engagement, but at least it gets at the problem. This is tangible evidence that ideas and information being distributed by the Senator is actually being absorbed by the constituents. There is an implication here that if they’re watching the video, they care about what’s being said. This is a fundamental component of meaningful engagement.Note: Given that the YouTube metric has three components, I will give them all equal weight and arrive at a final relevance score of 8.3%. Not so good for something that’s being touted all over the political media and is representing the good name of New York University and the George Washington University.

Is this fair?

Well, not entirely. I have thus far completely demeaned the importance of the factors that have been measured: primarily the piece of mind to engage online and the ability to attract followers or likes or visitors to online spaces. This is the first step to being able to engage – you have to be there and you have to have constituents to engage with. Since the focus of the study is about the Senator’s abilities to “authentically engage and mobilize” however, I think accomplishing this first step should only account for 10% of the points that can be awarded in Digital IQ.

That means my relevance score has to go up from 8.3% to 17.5%. I still don’t think that means it passes.

What should be done?

I’m not going to pretend to have the answers, but I also know that studies like this are not actually helpful to improving citizen engagement.

Accomplishing what these researchers set out to do is not easy. Here are some thoughts:

  1. Better Metrics. I would investigate the metrics of companies like Klout, who claim to measure your influence on Twitter. There are a number of them, all with different methodologies that I haven’t spent much time looking at recently. I would imagine there are similar metrics or tools that could be used to analyze discussion on a Senator’s Facebook wall and YouTube channel. How often does the Senator (or their staff) respond to the messages there?
  2. Other Sites. There should be a category for effective use of sites beyond Facebook, Twitter, and YouTube. Some states may have a large following on MySpace or a local social network or discussion forum that the Senator uses very effectively. This needs to be considered.
  3. Distinguish between campaign and official use. There’s a difference for members of Congress and it’s important – without it, incumbents could use federal money and outreach for campaigns, which would unfairly imbalance elections. How effectively are they maintaining this distinction and what are they doing to move followers from one to the other. I don’t know how to accurately measure this factor, but it’s an important part of their digital literacy.

I’m sure there are many more people out there who have better ideas than I about how to establish the metrics that need to be created here, but I hope that it’s helpful in some way nonetheless.

If you’ve taken the time to read this, I’d really like to hear your opinion on it as well. Am I off-base or am I grasping some fundamental component of social media that was primarily unaccounted for in this study?

11 Replies to Digital IQ?

  1. Wayne,

    Thanks for the detailed feedback, several of your observations resonated and will impact the methodology for our next study. The next week is crazy, however maybe we can do a call the week after Labor Day to provide more color on our methodology and get additional feedback?

    Thanks again,

    Scott Galloway
    NYU Stern

    Posted by Scott Galloway on 20 August 2010 at 12:08pm

    • Thanks, Scott! It’s good to know that you are pro-actively working to improve the methodology. I’m happy to have that call.

      Posted by Wayne Moses Burke on 20 August 2010 at 12:08pm

  2. I think your analysis introduces a number of useful questions. Aside from the data and methods of the research itself, which are always worth considering in terms of the validity of the conclusions reached (see Politico’s headline that John McCain is a “Twitter Genius), your skepticism regarding the omission of engagement metrics is important, as the distinction between governance and campaigning.

    Deciding whether someone is actually listening, engaging and using sentiment analysis or leveraging new media for more participatory democracy requires more factors. I agree that deciding to “assess a Senator’s effective ability “to authentically engage and mobilize voters and constituent” based upon traffic or followers numbers is problematic.

    Social media allows a reciprocal relationship between the folks formerly known as the audience and the people sharing information. Measuring velocity, which correlates to frequency of posts, follower or fans numbers and reshares of information doesn’t reflect whether there are conversations or feedback loops on the platforms. The report, in many ways, applies a broadcast model of measurement to social media. I suspect that I’m not alone amongst other long-time users in finding that unsatisfactory as a gauge for anyone’s “digital IQ.” Yes, showing up and posting matters, but in some ways it’s akin to showing up to take a test and filling out your name and address on a standardized form.

    Posted by Alex on 20 August 2010 at 12:08pm

    • As is your way Alex, you have successfully summarized my thoughts into a much more succinct statement. Thanks.

      Posted by Wayne Moses Burke on 20 August 2010 at 12:08pm

  3. I think the most important measures of Digital IQ should focus on engagement – mentions, retweets, conversations, shares, likes, comments – and the extended reach those engagements create.

    These days, the effectiveness of your digital strategy is all about creating buzz about what you are doing by trusted, third-party validators/messengers.

    Thus, the point of your activities related to posting content/messages and growing your direct audience is to mobilize other people to extend your reach and engage with you.

    Contrary to popular belief, the web has ALWAYS been about social media. Core social media tools (chat, discussion boards) are older than the web. So anyone who thinks web 2.0 is the dawn of social media is about 10-15 years behind reality.

    Digital IQ is all about your ability to push out your ideas to an audience that will discuss those ideas with you and share those ideas with their audiences, creating a larger audience for you to discuss your ideas and more people to share them. Its all about communications and community.

    I often say those that call the internet an “information revolution” sell it short. It is a communication revolution and what we communicate is information and who we communicate it to is our extended community.

    Any measure of digital IQ that relegates community engagement to the bottom of the list or to omit it altogether (except for activities on blogs) is completely out of touch with reality.

    Wayne’s detailing of the Digital IQ breakdown from NYU Stern and GWU Business Schools reveals that their view severely limited. There measures reflect an assumption that the internet is an information revolution. They are woefully behind the curve.

    As for the 7 geniuses in Congress, their list is a joke. John McCain, for example, uses Twitter primarily as a one-way broadcast tool. Given that twitter is first and foremost an engagement tool, his use of the tool is a far from genius as it gets. Sure he has a lot of followers, but he only follows 160 people (mostly colleagues and news feeds). And while he has a large audience on Facebook, he doesn’t appear to participate in any of the conversations under his wall posts.

    As for integrating his web presence, his Twitter profile links to his website and his website links to his Twitter, but his website does not link to his Facebook page.

    I could go on, but I think my point is clear. The topped ranked genius according to this report doesn’t even do these most fundamental things. If that is genius, then we really do need to reassess our notion of education and intelligence.

    Posted by Alan Rosenblatt on 20 August 2010 at 1:08pm

  4. Thanks, Wayne! I appreciate the time you took to lend concrete analysis to what I was feeling–that this “study” had the rigor of, well, something not rigorous (mom always said to be nice).

    We’ve talked about a spectrum of social media adoption and engagement–from no engagement to broadcast to interaction to co-creation. I would offer that interaction then has two sub-spectrums. The first including allowing commenting and ranking, to audience interacting with each other, to a conversation between the principal/brand (Senator in this case) and audience. Another swath is place of engagement, only on their turf or on other’s turf–i.e. do they color outside the lines of their own Facebook page or blog?

    The Digital IQ measures primarily broadcast, as you have identified. But part of that is because it’s easy to count. I agree with Alan that engagement can be shown by people taking in the broadcast and passing it on and the reach and its echoes. Retweets and Facebook “shares” are ways to measure this–and are countable. Alex is right that we need to find additional markers that measure engagement.

    This begs the question–what is our definition of “engagement.” Without consensus here, we are all measuring well, stuff. More work to be done.

    Posted by Gwynne on 21 August 2010 at 4:08pm

  5. The report states, “Our thesis is that digital competence provides an opportunity for senators to authentically engage and mobilize voters and constituents.” Using digital tools this way is powerful, but it seems to be more useful for advocacy groups than senators; the focus of advocacy groups is to “engage and mobilize” citizens to call/email/fax their member of Congress on a frequent basis.

    The only time senators mobilize voters is to vote for him/her, and that type of mobilization isn’t enough to create a robust democracy because it happens only once every six years.

    I do agree that politicians who are proficient with digital tools are more likely to get elected, but the report falls short of it’s profound central premise: “Digital IQ = A More Robust Democracy”.

    An elected representative’s role is not to engage and mobilize constituents to get reelected but instead to represent (think re-present) their point of view in a national forum when making complex policy decisions.

    To the authors of the report, I’d suggest replacing “engage and mobilize” with “engage and listen”. Although they are all fascinating, the fourteen metrics listed above focus on mobilizing (# followers, # views, sentiment/approval, etc) rather than listening. A report that focuses on how well senators listen will more accurately measure how robust our democracy is.

    Posted by Lucas Cioffi on 22 August 2010 at 1:08am

  6. This is really interesting, both the article and your comments. I will definitely by linking to them from my site and twitter.

    Posted by Josh on 23 August 2010 at 10:08am

  7. Very nice post, Wayne. I hadn’t seen it before I wrote my own post at SECTOR: PUBLIC, but now I’m linking over to yours too.
    http://sectorpublic.com/2010/11/does-your-public-sector-digital-iq-measure-anything-important/

    Posted by Mark Drapeau on 29 November 2010 at 7:11am

  8. Pingback: Does Your Public Sector “Digital IQ” Measure Anything Important? | Publicyte

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>