Network monitoring & evaluation: Taking stock

Another stock-taking post on the collection of network M&E resources (Photo credits: Hooverdust)

It was about time to prepare another of those stock-taking blog posts, don’t you think?

This time the topic is monitoring and evaluation (M&E) for networks, among others because there are a number of networks that I am involved in which will need to develop a solid M&E framework for themselves and for their respective donors so this post could help come up with a better approach. And, who knows, perhaps you will also find something useful in there. If this is all rubbish, please put me out of my misery and help me read some quality references on the topic, ok?

When it comes to M&E of networks, documents are a lot more scattered than for the capacity development stock-taking post I wrote earlier. And to spice things up, on Google, there is a hell of a lot of misleading resources pointing to LAN/WAN network monitoring – clearly the web is still the stronghold of a self-serving (IT) community.

Fair enough! But luckily there are also relevant resources among my documents, of which I would like to mention:

Guides, tools and methods for evaluating networks (direct link to a Word document)

(Amy Etherington – 2005)

As the title indicates, this paper focuses on evaluation rather than monitoring of networks – as a means for networks to remain relevant and adapt if need be. Three major considerations are taken into account here:

• measuring intangible assets (related to characteristics of networks such as social arrangements, adding value, creating forums for social exchange and joint opportunities);
• issues of attribution (linked to issues of geographic and asynchronous complexity of networks, joint execution of activities, broad and long term goals of networks);
• looking at internal processes: the very nature of networks renders internal processes – of mobilisation, interlinking, value-adding – very interesting. The further effects of the network on each individual member are also useful to look into.

And then follows a selection of nine evaluation methods (all dating from 1999 to 2005 though), very well documented, including checklists of questions, tables with dimensions of networks, interesting (or sometimes scary) models, innumerable steps referring to various maturity stages of communities. This seems one of the most relevant references to find at least practical methods to tackle network M&E.

Evaluating International Social Change Networks: A Conceptual Framework for a Participatory Approach (PDF)

(Ricardo Wilson-Grau and Martha Nuñez – 2006)

Among the most influential authors on the topic of M&E and networks, Wilson-Grau and Nuñez have been writing a lot of documents referred to in other papers mentioned here. This paper – which also focuses on the evaluation of networks – introduces the 8 or so functions that networks perform and considers four qualities and three operational dimensions. The result is a table of 56 criteria – shaped as questions – which ought to be answered by members of the network – with a careful eye for justification behind each criterion, because each network is different. The authors continue with the four types of achievements one can hope for social change networks: operational outputs, organic outcomes, political outcomes (judged as most useful by the authors themselves) and impact. Again the table is of great help and this document is a useful introduction to the author’s body of work.

A Strategic Evaluation of IDRC-Support to Networks (Word)

(Sarah Earl – 2004)

Epitomising the long term experience of the Canadian International Development Research Centre (IDRC) with monitoring and evaluation of networks, Sarah Earl presents, in this seven-page briefing note, a questioning process to evaluate the function of IDRC in supporting networks. In doing so, she stresses a series of questions pertaining to the coordination, sustainability and intended results / development outcomes of networks. She further explains the methodology used (literature review, key informant interviews and electronic survey of network coordinators, lesson learning sessions leading to writing stories from IDRC staff). This paper can be useful for actually setting up a methodology to collect evidence about the functioning of a network.

Network evaluation paper (Word).

(June Holley – 2007)

June Holley has been working for over 20 years on economic networks. This five-page paper  introduces a method that focuses on network maps and metrics, network indicators and outcomes. The paper suggests using scores and looking at awareness (of the network as a whole), influence, connectors, integration, resilience, diversity and core/periphery.

Network mapping and core-periphery (Image credits: Ross Dawson)

In terms of indicators, Ms. Holley recommends a series of questions that point to the self-organising and outcome-producing characteristics of the network, but also at questions of culture (as in shared norms and values) and evidence of skills that allow the network to change.

There are more (*) papers specifically focused on networks and their evaluation but I found them less relevant, often mostly because they are a bit dated.

Of course there are many other references on monitoring and evaluation in publications and resource sites about networks. Here is another, shorter, selection:

• The M&E section (pp. 87-91) of ‘Work the net’, GTZ’s guide to managing formal networks. This guide from 2006 is also available in other languages on the GTZ website – find the PDF documents here: French version – Spanish version – Portuguese version).
• In the water sector, CAP-Net has been working quite a bit on monitoring networks. On this page they provide an overview of several network management tools, including their own network monitoring strategy (2008).
• From ECDPM’s paper on mapping of approaches towards M&E of capacity and capacity development (2006), a number of things are mentioned on the monitoring of networks too, particularly

While on the topic of network M&E and its link with the specific monitoring of knowledge management, I would like to point to the summary of a discussion that took place in 2008 on the KM4Dev mailing list on the topic of M&E of KM: http://wiki.km4dev.org/wiki/index.php/Impact_and_M%26E_of_KM. This topic will probably remain interesting. It has been explored various times on the KM4DEV mailing list, it was recently touched upon in the francophone KM4DEV CoP SA-GE and it is likely to reappear as a topic of choice in 2010 on various platforms, not least because IKM-Emergent is planning to work more on this issue after having released the first of two commissioned papers on M&E of KM (this working paper on monitoring and evaluation of knowledge was written by Serafin Talisayon). I will certainly report about this in the coming weeks / months.

As ever with this series of stock-taking posts, I will try and keep this overview updated with any other interesting resource I get my hands on. So feel free to enlighten me with additional resources that go deeper, provide a lot of synthetic clarity or provide a refreshing perspective on the topic of network monitoring. What has worked for you in your work with networks? What have you found useful ways to measure their effectiveness and other dimensions? What would be your words of caution when assessing networks?

Networks are here to stay for a while so this discussion goes on…

(*)

I came across a number of other papers that all have something to say but are a bit out of date and I decided not to reference them here.

• Do networks really work? A framework for evaluating public-sector organizational networks (PDF) – (Keith G. Provan and H. Brinton Milward – 2001)
• Measuring while you manage: Planning, monitoring and evaluating knowledge networks (PDF) (Heather Creech – 2001)
• Measuring the Immeasurable Planning, Monitoring and Evaluation of Networks (PDF) (Oxfam/Novib – 1999)
• Organizational Network Analysis as a Tool for Program Evaluation (Merrill Eisenberg and Nancy Swanson – 1996)

Related posts:

• Other stock-taking posts: Assessing, measuring, monitoring knowledge (and KM) – Storytelling – Learning cycle basics and more – Capacity development

Une communauté de pratique francophone – duplicating my identity or celebrating our diversity?

That’s it, the francophone community of practice about knowledge management for development is standing on its legs: SA-GE was created on the back of the francophone discussion that took place in Brussels on 5 October 2009. It represents a combination between Savoirs and Gestions and that name itself is the result of a long discussion started in Brussels and that still keeps quite a few people ‘perplexes‘ (including me); yet, at the end of the day, what matters is what goes on on the platform, not what is going on with its title.

Anyway the discussion group site is here: http://dgroups.org/Community.aspx?c=532af1c1-1d65-419c-a52c-7af6c8b7451a; you can register on to it (provided you speak French and have some interest in knowledge management/sharing for development, that is 😉 following this link: http://su.pr/7h3yPF or alternatively contact me to get on it.

It’s an interesting initiative because there is a lot of knowledge to be shared between francophones – we have been – actively or not and consciously or not – sidelined by the global discourse in English (oh, am I writing in English here?). And perhaps because a lot of francophones are good at talking i.e. face-to-face rather than writing, not a hell of a lot has happened in terms of creating text-based communities of practice (CoPs). If I’m wrong, I’d be happy to hear about very successful large francophone CoPs in the field of development.

This time, SA-GE stems from the global KM4DEV community so it comes with some kind of legated identity but hopefully it will develop its own dynamics and discourse based on peculiar areas of interest etc. And that’s the fascinating side of being part of this new community of practice: observing perhaps entirely different patterns of discourse emerging. The very use of different words has an impact on one’s worldview. Insisting on ‘capitalisation’ and its integrated approach is not quite the same as the toolkit-approach of the Anglophone world… Let’s see what flares up.

As I said on a previous post, for me it will also mean developing a split personality between the global (English-speaking) and the Francophone groups. Though if discourses and agendas differ between the two there could be a really great opportunity to share across communities, to get different world views acquainted with each other (and hopefully to solve my multiple personality disorder, at least partly)…

The increasing specialisation in different language groups may well pose the threat of creating new silos, but bridging and connecting silos

Blending cereals from different silos to get a healthy mix...

(or bashing them down to create only one big market of grains happily mixing with each other on the edges) will become all the more essential for that reason. And if we’re good at bridging, we should be one step closer to recognising multiple knowledges and the beautiful diversity of this world. A natural diversity that reinforces us rather than an over-engineered conformity that weakens us, as testifies the impoverishment of nature in the face of our apprentice wizardry: think decreasing bio-diversity because of us engineering nature as a landscape and encouraging monoculture.

Alors à quand les premières discussions sur SA-GE ?



KM4DEV 2009, a few weeks later

I am just back from amazing holiday in Ethiopia – a country so rightly desperate to attract more tourists and shed once and for all its image of drought & famine-ridden block of sadness.  The whole of last month was therefore genuinely amazing and totally distracting for my blog routine, if I can still talk of such a thing as a routine.

Anyway after more than a month I can also reflect about the recent KM4DEV 2009 event (see the documentation of the event), which is usually one of my highlights of every year. This year didn’t fail to be, even though for a host of reasons I was not as deeply into it.

What I could observe though:

–          The community is growing. despite crisis and travel budget crunch. Over 85 people made it this year and on the Ning site there are – as we speak – 746 members. And at the event easily over 70% of the participants had joined only in the last year;

–          The community is expanding up to other communities – which is the reason why the francophone group (SAGE) is being set up, following the SIWA community;

–          The community is also expanding down to various sub-groups. Was it really an unconscious choice for this year’s event to focus on various sub-sectors (health, agriculture, water etc.)? And what about the recent KM4D journal issue on learning in the water sector, or the current exploration of  various sub-geographic groups (the Rome club, The Hague KM4DEV, Washington KM4DC, and the very recent request on the mailing list for a KM4DEV in Egypt…)?

–          Finally on the side line, there is perhaps a higher turnover among core group members. Perhaps this is actually stable and I am just noticing it as more people I know are slightly stepping down and others are joining – which is a healthy sign after all;

From one community to multiple communities of pollinators?

What this says to me: the community is going through a major shift in scale, bearing implications on the identity, unconscious choice of topics and up to governance (or stewarding, or gardening). Is the group now too big to keep focusing on cross-cutting issues of knowledge sharing and learning and should it not start looking into more specific challenges in more specific (thematic and geographic) contexts?

It is difficult to tell what direction this will go but this is always an interesting and opportunistic moment for any community to reinvent itself and take advantage of its (more) critical mass. So while I will try and continue blogging (perhaps with more than one this week), I look forward to the next months and for my part I will certainly try and play a bridging role between the SAGE group and the global KM4DEV community.