I’ve started working on a summary of two papers commissioned by the IKM-Emergent programme to unpack the delicate topic of monitoring (and evaluation) of knowledge management (1). This could be just about the driest, un-sexiest topic related to KM. Yet, it seems precisely one of the most popular topics and one that keeps resurfacing on a regular basis.
On the KM4DEV community alone, since the beginning of 2009, nine discussions (2) have focused on various aspects of monitoring of knowledge management, some of them generating a traffic of over 30 emails!! Are we masochistic? Or just thirsty for more questions?
M&E the phoenix of KM? (photo credits: Onion)
Anyway, this summary piece of work is a good opportunity to delve again into the buzz, basics, bells and whistles of monitoring knowledge management (as in the practice of observing/ assessing/ learning inherent to both M&E rather than on the different conditions in which M or E generally occur).
In attempting to monitor knowledge and/or knowledge management, one can look at an incredible amount of issues. This is probably the reason why there is so much confusion and questioning around this topic (see this good blog post by Kim Sbarcea of ‘ThinkingShift’, highlighting some of these challenges and confusion).
In this starting work – luckily supported by colleagues from the IKM working group 3 – I am trying to tidy things up a bit and to come up with a kind of framework that helps us understand the various approaches to M&E of KM (in development) and the gaps in this. I would like to introduce here a very preliminary half-baked framework that consists of:
And I would love to hear your views on these, to improve this if it makes sense, or to stop me at once if this is utter gibberish.
First, there could be various components to look at as items to monitor. These items could be influenced by a certain strategic direction or could happen in a completely ad hoc manner – a sort of pre-put. The items themselves could be roughly sorted as inputs, throughputs or outputs (understood here as results of the former two):
||Input (resources and starting point)
||Throughput (work processes & activities)
|– None (purely ad hoc)- Intent or objective
– Structured assessment of needs (e.g. baseline / benchmarking)
– Strategy (overall and KM-focused)
|– People (capacities and values)- Culture (shared values)
– Systems to be used
– Money / budget
|– Methods / approaches followed to work on KM objectives- (Co-)Creation of knowledge artefacts
– Use of information systems
– Relationships involved
– Development of a learning/innovation space
– Attitudes displayed by actors involved or concerned
– Rules, regulations, governance of KM
|– Creation of products & services- Appreciation of products & services
– Use/application of products & services
– Behaviour changes: doing different things, things differently or with a different attitude
– Application of learning (learning is fed back to the system)
– Reinforcement of capacities
All these components are then affected by the various levels at which a KM intervention (or strategy) is monitored, which could be:
- Individual level;
Different levels at which M&E of KM could take place
- Team level;
- Organisational level;
- Inter-organisational level i.e. communities of practice, multi-stakeholder processes, potentially verging on to sectoral level – though with the problem of defining ‘a sector’;
- Societal level affecting a society entirely.
And then of course comes perhaps the most crucial – yet implicit – element: the worldview that motivates the approach that will be followed with monitoring of knowledge management.
Because this is often an implicit aspect of knowledge-focused activities, this is largely a grey area in the way knowledge management is monitored. Yet on a spectrum of grey shades I would distinguish three world views that lead to three types of approaches on monitoring of knowledge (management). These approaches can potentially be combined in innumerable ways. The three strands would be:
- Linear approaches to monitoring of KM with a genuine belief in cause and effect and planned intervention;
- Pragmatic approaches to monitoring of KM, promoting trial and error and a mixed attention to planning and observing. I would argue this is perhaps the dominant model in the development sector, judging from the literature available anyhow (more on this soon).
- Emergent approaches to M&E of KM, stressing natural combinations of factors, relational and contextual elements, conversations and transformations.
In the comparative table below I have tried to sketch differences between the three groups as I see them now, even though I am not convinced that in particular the third category is giving a convincing and consistent picture.
||Linear approaches to M&E of KM
||Pragmatic approaches to M&E of KM
||Emergent approaches to M&E of KM
|Attitude towards monitoring
||Measuring to prove
||Learning to improve
||Letting go of control toexplore natural relations and context
||What you planned à what you did à what is the difference?
||What you need à what you do à what comes out?
||What you do à how and who you do it with à what comes out?
|Chain of key elements
||Inputs – activities –outputs – outcomes – impact
||Activities – outcomes – reflections
||Conversations – co-creations – innovations –transformations – capacities and attitudes
||Why, what and how?
||Logical framework and planning
||Trial and error
||Experimentation and discourse
|Attitude towards knowledge
||Capture and store knowledge (stock)
||Share knowledge (flow)
||Co-create knowledge and apply it to a specific context
||Information systems and their delivery
||Knowledge sharing approaches / processes
||Discussions and their transformative potential
|I, K or? What matters?
||Knowledge and learning
||Innovation, relevance and wisdom
|Starting point of monitoring cycle
||Expect as planned
||Plan and see what happens
||Let it be and learn from it
|End point of monitoring cycle
||Readjust same elements to the sharpest measure(single loop learning)
||Readjust different elements depending on what is most relevant(double loop learning)
||Keep exploring to make more sense, explore your own learning logic(triple loop learning)
The very practical issue of budgeting does not come in the picture here but it definitely influences the M&E approach chosen and the intensity of M&E activities.
Aside from all these factors, there are of course many challenges that are plaguing an effective practice of monitoring knowledge management, but this framework offers perhaps a more comprehensive approach to M&E of KM?
Again, I am inviting you to improve this half-baked cake or to reject it as plainly indigestible. So feel free to shoot!
(1) Knowledge management understood here as ”encompassing any processes and practices concerned with the creation, acquisition, capture, sharing and use of knowledge, skills and expertise (Quintas et al., 1996) whether these are explicitly labelled as ‘KM’ or not (Swan et al., 1999)”. This definition is extracted from the first IKM-Emergent working paper. Even though I don’t entirely agree with this definition, let’s consider it’s creating enough clarity for the sake of understanding this blog post.
(2) Previous discussions related to M&E of KM on KM4DEV:
- Managing community of practice: creative entrepreneurs (22/11/2009) with a specific message on the impact of communities of practice
- Value and impact of KS & collaboration (11/10/2009)
- Evaluation of KM and IL at SDC (08/07/2009)
- KM self-assessment (18/03/2009)
- Organisational learning indicators (13/12/2009)
- Monitoring and evaluating online information (05/02/2009)
- Monitoring and evaluating online information portals (03/02/2009)
- Evaluation of KM processes (30/01/2009)
- Evidence of sector learning leading to enhanced capacities and performances (05/01/2009)