Social web metrics: between the cracks of evidence and confidence


Assessing knowledge work is back on my menu for this year and I need to start somewhere simple(r): social web metrics.

Rather than focus on the high end of monitoring/evaluation (M&E) of knowledge work, I’d like to look into current web metrics in use and to understand what they are really capturing, what they fail to capture or what problems they pose and what links them together or what we could do with them.

The social media analytics framework below proposes a good entry point to this exploration. I will come back at a later stage to such analytical frameworks.

A framework for Social Media Analytics (Gaurav Mishra)

A framework for Social Media Analytics (Gaurav Mishra)

The social web metrics we have at our disposal to assess knowledge work are related along a chain from attention to action (the famous [social] AIDA again) – or from content to collective intelligence as suggested above. First comes discovering a particular resource and last comes using it, appreciating its use and being transformed (at scale) with it. Each resource, each piece of content,  hopes to tick as many of these goals. With it comes a potential useful insight, but also limitations… and mitigations.

Here is an overview of some objectives we might legitimately have with our content (which metrics try, mostly insufficiently, to capture).

Find me!

Sometimes we just bump into a site or resource on the web, while looking for something else… A page view is a reflection of that. Page views can thus be intentional views (effectively linked to your content/focus) or accidental views (someone ends on a web page after looking for term that is only vaguely connected with that content). An instance of this: I suspect quite a few visitors to my blog are actually looking for information about the gay podcast ‘the feast of fools’ but they end on this blog post – no connection other than the name. Then again, people do come across your content for good reasons too.

Limitations and mitigation: Page views are thus not entirely helpful. Oh, and in case you didn’t know by now, a hit really is not a useful metric, even though it’s all about finding stuff online.

The remedy is to properly (meta-)tag your content, use descriptions on your photos, add links to relevant related content. Linking is the currency of the time with search engine optimisation. The more you are linked to by others, the more likely people will find your content genuinely related to their focus, in relation with specific search terms.

Grab my attention!

Ending on a page or resource is the first step. Attracting our curiosity as online visitors is the second step, and it is not straightforward with our 8-second attention span. (Intentional) Page views are still the main metric here. But so are retweets on Twitter.

Limitations and mitigation:

This is where a good title comes in handy (one of the many useful tips of Ian Thorpe in sharing his blogging experience). But a retweet in particular doesn’t mean that the person re-tweeting the page/resource actually liked it… or even read it. These visitors just seemed to like your shop window’s look and feel! Mind that they like your content for the right reasons beyond that sweet first impression. All ‘Find me’ advices are applicable here too!

Like my content!

Ok, now people have checked your content. And they enjoy it! They ‘like’ it. Or they +1 it, or  they rate it… There are various ways to show appreciation for content. Perhaps the most valuable one is to comment on content and show appreciation this way. It’s useful feedback, provided it’s genuine.

Limitations and mitigation: The danger is that some people just ‘like’ because the like button is easy to push, with or without checking the content in the first place (see the shop window problem above). The other problem is that there is still no indication as to why they like your content (perhaps the tone, the image you chose, the serendipity effect that led them to your content at a moment when they were looking for something similar). Most liking metrics are only partly useful, unless a certain volume of these signals is aggregated throughout various collections and it starts indicating trends.

Focused comments, however, should be encouraged as they help find out why people liked your content and helps you engage with your audience one step further…

Pass it on to others!

If people liked your content, perhaps they didn’t rate it (most people find giving feedback a daunting step) but they might have shared it with others. Metrics here include: linking to your content, social shares (re-tweets are a point in case, but Facebook shares, Google+ shares and email shares are other examples), citations of your work etc. People might be sharing a link to your content or the full content (re-blogging content is an indirect metric of sharing here).

Limitations and mitigation: The same danger of people sharing without having checked your content is still looming. But sharing content is generally a better indication of appreciation for your content, especially when it is shared in quantity and quality. Pay attention to who shares your content. Trusted and valued sources are great indicators of the quality of your content. I am not aware of tools that track the sharing of content with a specific breakdown of the popularity of sharing sources but that would be useful.

Keep me for later!

People may keep track of your content for different reasons:

  • They haven’t read it yet but want to do so later when they find time for it;
  • They want to share it with others but haven’t gotten around it;
  • They like it so much – or find it useful enough – that they want to collect and curate your content.

At any rate, they seem attracted to your content enough to keep it for later.

Metrics here include: Bookmarks, favourites, downloads etc. These are possibly good measures of some following for your content.

Limitations and mitigation: Two out of three reasons above do not point to any particular appreciation. Resources could be put aside and never used again. Even when downloaded, their effective use depends on the discipline and willingness of the bookmarker to actually use his/her saved resources for another activity. Again here large numbers of these metrics can plot useful trends, but individual measurements or isolated bookmarks remain marginally useful.

(re-)Use me!

The objective of your content is to be used – and re-used. Directly or indirectly, now or later, as intended or otherwise, as direct inspiration or diffuse source of innovation. But this is very difficult to track. Only direct references in someone else’s work are (usually) straightforward indications that content is being used.

Readily available metrics thus include: reblogs, citations, links in other important writings and works. Testimonies (e.g. stories of change and the likes) are not a given in social media but are probably the best approach to hear about the use of content. Indirectly, comments may play a similar role, if they mention how the content is being applied somewhere else (as opposed to just reacting on the content itself).

Limitations and mitigation: It is very difficult to get such references and accounts of use – but from this point on it becomes really interesting and relevant. Aiming at collecting such testimonies and developing a culture of feedback and critical reflection (e.g. by means of comments, ratings etc.) all contribute to getting better at and closer to collecting interesting results about the use of content.

Let me make a better you!

One of the best results we can hope for any resource we develop is for it to contribute to changing behaviour. Using content doesn’t equate change. Change is very elusive and difficult to assess as it is an intimate matter, which perhaps requires the realisation of the person changing that they are changing. 

Among other metrics here, the most important one are testimonies, and to a lesser extent comments (provided these comments relate to the usefulness and effect of the resource itself, how it was used not just about the content of the resource). These are not available web metrics (yet?) and would be more typically part of process/outcome/impact monitoring efforts. But these results are worth tracking down.

Limitations and mitigation: As for the use of content, accounts of change brought about by resources or otherwise are very diffuse and hard to collect, even harder to attribute, unless  mentioned in the testimonies. The same approach as for the use matters here, it just goes one level deeper in the exploration.

Become a movement thanks to me

The ultimate goal of any resource is that it is so seminal that it is referred to over and over again and has the tendency to provoke a knock-over domino effect on the behaviour of many. What the Bible or the Coran or the little red book achieved. Tough job…

Limitations and mitigation: Frankly, if you are at that stage, you should be blogging about this instead of me 😉 I can only say that radical innovation, use of locally nested word-of-mouth conversion effects and tapping into the viral potential of some technologies and their disruptive nature might offer shorter paths to this holy grail.

 In conclusion…

What is difficult is that there is no linear following along these metrics. Furthermore, some of these metrics only become useful at a certain scale – or in combination with other metrics occur e.g. only when various people have downloaded and favourited a resource can one tell that it probably has a transformative effect on people. The only sure way to get a relatively sure account of evidence is through testimonies – if they are truthful and sufficiently marginally biased.

The table below summarises some of the metrics available to suggest evidence of any impact of your content/resources.

Direct metrics Indirect metrics
Finding Page views, hits
Liking Likes, +1’s, ratings, comments
Retweets and other social shares
Sharing Links, Social shares, citations Downloads, comments, reblogs
Keeping Downloads, bookmarks, favourites Re-tweets, Social Shares, (some) social ratings
Using Citations, links, testimonies, reblogs Comments
Being transformed by it N/A Comments, testimonies

All in all, what matters in those web metrics are a combination of: effective consumption of the content, appreciation of that content (its quality and relevance), intent to use it, effective use of it, transformation brought by that use, scale of that transformation.

There are many tools to collect these. But the tools only address the collection part (your demand for it as content provider wishing feedback). What is more difficult is the supply of such evidence, and that comes only progressively with a culture of feedback and critical inquiry… Until that culture is there, we always navigate between the cracks of evidence and personal confidence.

Related blog posts:

Research, KM and multi-stakeholder processes: cross interview with Cees Leeuwis and Mark Lundy


Cees Leeuwis (Credits: C.  Bilonda / E. Le Borgne)

Cees Leeuwis (Credits: C. Bilonda / E. Le Borgne)

Last week, I had the privilege of sitting with two people I’ve been following with interest over the past few years:

  • Cees Leeuwis, Professor of Communication and Innovation Studies at Wageningen University and a lead thinker on multi-stakeholder processes and social learning processes involving research.
  • Mark Lundy, senior researcher at the International Center for Tropical Agriculture (CIAT) and one of the forefront CGIAR thinkers and leaders on multi-stakeholder processes such as learning alliances (which later inspired my former employer IRC International Water and Sanitation Centre to a.o. develop this publication).
Mark Lundy (Credits: C.  Bilonda / E. Le Borgne)

Mark Lundy (Credits: C. Bilonda / E. Le Borgne)

They kindly accepted to answer a couple of questions about their current sources of (research) interest, knowledge management and multi-stakeholder processes.

What are you currently working on or interested in?

(Cees) I’m interested in so many things! The overarching theme in my work is around the relationships between technology and society, looking at innovation; it is about saying that innovation is more than technology alone, that it combines hard- soft- and org-ware and about thinking through the implications of that combination. This focus is very important and helps us explain why a lot of things go well or wrong and to rethink the role of science in the innovation process, how one can stimulate, organize and contribute to innovation.

(Mark) Two major things: (a) business models for sustainable trading relationships between small farmers and buyers (see: http://ciat-library.ciat.cgiar.org:8080/jspui/bitstream/123456789/6593/1/LINK_Methodology.pdf); and, (b) Research in development platforms building on CIAT’s experience with Learning Alliances and Innovation Platforms. I find these two topics fascinating and would happily give up my role in other programs I’m involved to dedicate myself to them.

What role do you see for knowledge management (if any) in the work you are doing and more broadly?

(Cees) KM is a problematic term. My real work on KM is related to how to embed research in society. I think that should be the role of KM: to help make people wait for research before it’s even finished. The idea is that you manage the production of research in such a way that there is some guarantee that people are waiting for it.

(Mark) KM is critical for nearly everything we do. My personal focus is on KM in the form of feedback loops for improved decision-making in business models and KM at the level of Research in Development platforms. I also see a critical role in regards to policy incidence which, historically, has not been the forte of the CGIAR.

Where do you see research on social learning and multi-actor initiatives go in the coming years?

(Cees) I think there will be more attention the dynamics of tension and conflict in these kinds of processes and the implications this has for facilitating such processes. In the end, change is about altering the status quo and usually many stakeholders are not very interested in that. And at the same time there may be competing initiatives for change. So tension and conflict are inherent to multi-actor initiatives, and I think we need to get better at dealing with this. There is a lot we can learn from studies in conflict management!

(Mark) From a CGIAR perspective, these topics need to be recognized as legitimate research topics in their own right. The CG can do brilliant upstream research but if we don’t find ways to effectively connect this to development demand in ways that add value to both research and development we will have negligible impact.

Related blog posts:

Harvesting insights (6): A checklist of comms/KM functions in any development (research) organisation or initiative


Just a post to gather my thoughts on this once and for all on this topic.

Together with some colleagues from WorldFish and the Inernational Water Management Institute (IWMI), we have been pondering about the profile of comms/KM profiles and positions in any organisation and/or project, since ILRI and IWMI held a really interesting workshop on knowledge management and communication in the CGIAR research programs.

The reflection on KM positions has been helpful to think about the profile of the people that might have to take care of comms and KS/KM/learning. Now, at a higher level, what are the organisational functions that take care of communication and/or knowledge sharing and management and learning? This is treading suspiciously close to the happy families of engagement, but here I want to think about the functional departments or units of work that any organisation might want to consider useful, rather than look at the fields of expertise as mentioned in the families of engagement.

Every organisation or project has its way of looking at these functions – rightly so – but what could a generic checklist of these functions look like?

I would think this works around different tiers of organizational importance (how this is perhaps currently assessed, not how it should be assessed) and relative recognition of those functions.

  • First tier
    • Public awareness and media engagement (i.e. communicating the organisation/initiative)
    • Dissemination of information (communicating the results -against stated objectives- of the initiative)
    • Marketing (for commercial companies or initiatives that promote a particular product or service)
    • Network engagement (and management) with critical stakeholders and partners
    • Policy engagement and support, advocacy
  • Second tier:
    • Internal communication
    • Data and information management
    • Knowledge management
  • Third tier:
    • Capacity development (around communication and knowledge work),
    • Monitoring and evaluation (of knowledge work and communication)
    • Process documentation (informal monitoring)

For projects and time-bound initiatives, these different functions follow a different lifespan which my colleague Peter Ballantyne drafted here. Let’s examine these functions one level down in granularity:

Public awareness and media engagement: promoting the intervention/organisation, getting public attention through the media and conveying it through more mainstream (and increasingly social) media. This is all about communicating about the project/organisation/team etc. and is usually the most recognised set of communication activities because it might be a requirement from donors but also a good way to get some visibility for the initiative (the quest for immortality shows its nose again).

Dissemination of information and results

Communicating agri-water research over time (credit: ILRI/Ballantyne)

Communicating agri-water research over time (credits: ILRI/Ballantyne)

Second in line, usually, after talking about the intervention or organisation itself is: talking about what comes out of information dissemination. In this other graph by Peter Ballantyne, this would be typically the second peak of communication activities in an otherwise ‘communication-empty’ initiative: PA at the project launch, and dissemination at the end when results are ready. The problem is: it’s not enough. But dissemination remains a crucial function of communication – even though we are increasingly moving towards an engagement-rich communication approach.

Marketing

The projects and organisations that have some products and services to offer to the public – pay-for or not – have an additional communication imperative around the marketing of these products and services. The approach changes a bit between pay-for and free/public products and services but the idea of attracting attention, creating a desire, informing the desiring customers and leading them to action (the AIDA model which is increasingly questioned and reexamined from a socialisation perspective – see graph) or the

AIDA socialisation (credits - CoffeeMarketing)

AIDA socialisation (credits – CoffeeMarketing)

4Ps (price, promotion, product, place) can come in handy to make sure products and services find their customers and users. But again there might be little engagement there. Hence…

(Practice-oriented) Network engagement and management with critical stakeholders and partners

As pure dissemination-based approacheds are finding their limits, network engagement and management (or rather facilitation) is becoming increasingly crucial. Communication is no longer about crafting documents in isolation and sending them to intended target audiences but more and more so about bringing those audiences in the (co-)creation process. Trust becomes an important currency in communication work and partner / stakeholder management. We analyse our social networks, map stakeholders, identify who are the key nodes in the network and work with them from the start.

Whether by means of visits, exchanges, workshops, training courses, brown bag seminars, informal and formal lunches, bilateral discussions, network engagement is becoming a central bone in the communication spine. The practice aspect of this function is to ensure that the engagement effectively leads to transforming and adapting discourses, ways of thinking, behaviours i.e. the formal and informal practices of these actors we are working with and for. It is the alter-ego of the next function…

Policy engagement and support, advocacy

A related field is that of policy engagement / support and advocacy. The objective here is to ensure that research and other activities inform and influence policies, support them, and advocate issues that might have been blind spots until now. Increasingly, policy engagement is moving away from conventional advocacy (the one that is following a PR approach of unilaterally targeting messages for audiences) to embrace a much higher degree of interactions with these policy-makers and political actors that should be influenced. In multi-stakeholder processes, these political actors are part of the co-creation process and that is a new way of engaging with policies.

The next three areas are less obvious functions of comms/KM but people talk about them and recognise their importance. They simply don’t act upon it systematically.

Internal communication and knowledge sharing

Perhaps this ought to just be part of regular communication but it has often been overlooked in the past, because internal teams were not a key ‘target audience’. As we are in the network era, the importance of communication, cooperation and coordination dawns on project managers, and as teams are increasingly decentralised and scattered across various countries and locations, internal communication and knowledge sharing are also increasingly recognised as an important area of comms/KM.

Data and information management:

Data and information management are typically an area whose importance is recognised. Lip service is frequently paid for it, but following through with elaborate and robust systems for data and information management are a mile further which many are not ready to run. Still this is an area of concern for communication because the documentation part of the work collects a lot of information and the platforms and channels are usually set up by the communication (or KM) team. In research organisations, this function is sometimes nested directly in the research teams – but the challenge remains intact: someone needs to ensure data are collected, tagged and meta-tagged properly, cleaned, archived and sorted. Information outputs and records should also follow this logic, at a higher level of processing.

Knowledge management

Maybe this also ought to be lumped up with its sub-components of knowledge sharing, dissemination and information management, but knowledge management ought to be a function (if not a formalised position) to ensure the integration of conversations with documentation and learning. It becomes the life and blood of reflexive communication in and outside the organisation or initiative.

Now we enter the obscure areas in comms/KM, those functions that are usually not accounted for, not paid lip service for nor even thought about much, if at all.

Capacity development

One of these obscure areas is capacity development for comms and KM. In any organisation or initiative, there are people writing, presenting, engaging, reflecting, questioning… but they’re not part of the comms/KM team. They are sometimes very strong in all these areas that are perhaps not typically in their portfolio of activities. But sometimes they are not and they should be trained, coached, sharing their perspective with peers to improve their own practice. And sadly, there isn’t much in store for them to do so. Organisations and initiatives of the future should include a capacity development aspect to their activities to make sure that everyone involved is strong at conversing, documenting and learning individually and collectively…

Monitoring and evaluation (M&E) of knowledge work and communication

Of course M&E is recognised in most development/research organisations but formally including the monitoring and evaluation of knowledge work, much less so. Yet a formal assessment of communication and KS/KM activities would help all parties get more effective at what they are doing. Simple reporting on outputs is far from reaching this goal and understanding dynamic relationships, use of knowledge, effects of learning, transformative consequences of engagement are subtle but critical areas of importance for all of us if we are to remain relevant over time and strong on adaptive/proactive management.

Process documentation (informal monitoring)

I’ve already blogged in the past about process documentation and its Latin and Francophone variants in the past. It seems to me (and to my former organisation IRC) a crucial area to learn by doing and to improve the way an initiative is unfolding against its theory of change. Alas too often people recognise the importance of processes but fail to monitor them, not even informally – documenting discussions, reflections, insights, questions is not the cup of tea of most people, but I do think it is absolutely essential to instil a learning culture and to support various other areas of work: communication, KM/KS, M&E. See this publication for more information on this topic.

Morphing these categories?

Communication is evolving. Social learning is blurring the boundaries as it tends to bring together a lot of these activities together. And every organisation is mixing these functions in its own ways, so there isn’t a fixed menu but rather a set of options that can be combined and recombined in any comms or KM strategy. The functions themselves are however relevant to think about.

What changes do you see happen in this field? What is missing among these organisational functions?

Related blog posts:

What is common knowledge about knowledge? A visual tour…


The brain of a knowledge worker - and that is just the beginning (Credits: unclear)

Knowledge from a KM perspective? (Credits: unclear)

Knowledge is the all elusive complex concept. And visuals help represent complex concepts.

So for a change I thought I’d give a commented visit on a number of images about knowledge, found on the net. I don’t necessarily agree with what these images convey, but I have chosen to comment these ones because they seem to be popular on the web and generally in common knowledge.

This first one (the KM phrenology) is one picture that I used in the past – an interesting image because it depicts a lot of issues related to knowledge in the field of knowledge management. But of course it’s not meant to really represent knowledge and the picture is dated (over emphasis on ‘best practices’, looking at portals etc.).

Knowledge coffee (not cafe mind you)

Knowledge coffee (not cafe mind you)

This second picture is much more related to knowledge itself and represents the diversity of attributes associated (the coffee cup) with knowledge, with a higher emphasis on know-how/what/who/why and the definition of knowledge as ‘justified true belief’. It’s a useful image to remember all the angles that people associate knowledge with. A mine field indeed, or a rather spicy cup of coffee.

A KM mindmap of knowledge

A KM mindmap of knowledge

This third picture is a mini mind map of knowledge from a KM perspective again. It brings together four concepts typically associated with knowledge: tacit knowledge, knowledge conversion, explicit knowledge and information. I have my doubts about a few of these nodes – information and explicit knowledge are the same for me, and the graph itself seems to have been around for a while.

Is the world knowledge tree really growing?  (credits - unclear)

Is the world knowledge tree really growing? (credits – unclear)

But this graph again puts knowledge at the centre of KM attention and is useful to understand what are some of the key KM concepts associated with it.

The next one (the tree on the world) I quite like, as it represents the proverbial knowledge tree while also resting on the entire world. Somehow the idea that our knowledge is growing as a tree on the basis of global and local interactions is compelling. Let’s just hope that the branches (the results of knowledge) are not outpacing the roots of that knowledge tree (the sources that lead to develop knowledge results).

Knowledge in the DIKW nonsense (Credits - David McCandless)

Knowledge in the DIKW nonsense (Credits – David McCandless)

The next one is the unavoidable depictions of knowledge: knowledge as part of the DIKW pyramid.

It is no less wrong at that though – and I already blogged about this. But here you go: old established fallacies die at long last.

The next selected image (Superman) is another very commonplace portrayal of knowledge and its associations: knowledge is power.

Knowledge is power (Credits - Tiffany and Lupus)

Knowledge is power (Credits – Tiffany and Lupus)

I think this image does true justice to question (by stupidity) this saying which deserves additional caution. Sharing knowledge is more powerful in the age of networks…

Next (the blue face) is one new entry for knowledge which mirrors the first of these images but really focuses on all the insights that lay inside our head and are ready to be called and acted upon.

Knowledge as the collection of insights ready for sense-making

Knowledge as the collection of insights ready for sense-making

To me this is perhaps the closest depiction of what knowledge is – insights that can be invoked and used as and when. The image still  misses the dimension of the capacity that knowledge brings to use these insights, as I blogged about it earlier, but it’s getting there in my eyes.

And finally, building upon the previous image, what – really – is knowledge without action? A former boss of mine used to always ask ‘knowledge to do what?’ and that’s a bang on question. For that matter, knowledge management has been useless in many cases for failing to answer this simple question. So from this gallery of images, perhaps the most important to retain is that without using knowledge we are not better off with it.

What is knowledge without action (credits - Hiking artist)

What is knowledge without action (credits – Hiking artist)

If you have other personal ‘knowledge visualisation favourites’, please share them with me and I will feature them here with due credits!

Related blog posts: