Tag Archives: wales audit office

Being open by default

How might an audit office open up its systems so that information becomes open by default? Dyfrig Williams spoke with Tom Haslam about the approach of New Zealand’s Office of the Auditor-General.

The logo of the Office of the Auditor-General New Zealand

As part of the Wales Audit Office’s Cutting Edge Audit project, I am working on an Open Data prototype. During this work, colleagues told me that we could improve our approach to data. Not acquiring new data though – most colleagues said their biggest issue was better knowledge of, and access to, data that the office already held.

Our organisation has two specialist practices – financial audit and performance audit. This division facilitates specialism, so that we have colleagues with incredibly good knowledge in their fields of expertise. However, it also means that we have to work hard to break down organisational silos, sometimes reinforced by the systems we have in place.

Safeguarding data is an important feature of the way we have set up our information systems. Network folders are protected. Access is only available to specific teams and personnel, which means that the data within them is closed to others by default. Our SharePoint system is also set up in a similar way and the search functionality is not as good as it might be. All of this means that unless you know where the data is held, you’re unlikely to find it.

Learning from other audit offices

In my last post on the Queensland Audit Office’s work, I mentioned a well-travelled colleague called Tom Haslam. Tom has worked at the Office of the Auditor-General (OAG) in Wellington, New Zealand. And while there, the OAG identified similar problems with how they organised and held their data.

To address this, the OAG implemented a new SharePoint-based information system and complemented this with some pilot cross-office groups known as ‘iShare’. These groups were based around cross-cutting functional topics (for example the Transport iShare) with the aim of helping to break down organisational silos and promote a one-team approach across the office.

Adopting a new information system gave the OAG an opportunity to debate the relative merits of information systems being open or closed by default. This was discussed across the office through various channels.

The previous information systems had encouraged a mainly ‘closed until open’ approach. But the general feeling was that closed data might prevent the office from making the most of the information that they held. The natural tendency of all auditors is to be cautious, so under a ‘closed unless open’ approach, setting information as ‘open’ might be viewed as a risk best avoided, even if this approach wasn’t justified. On a practical level, having information closed off requires various permissions and access rights to be set up. This alone can be a barrier to sharing data.

The OAG structured its new information system so that information was ‘open unless closed’ with metadata to help staff find what they wanted. This approach facilitated sharing, encouraging staff to think about how they could add value by joining up information. A default setting of ‘open until closed’ made staff think more carefully about why they should want to close off access, for example material with national security implications or identifiable personal information.

On a technical level, a cleaner configuration of the IT system without endless permissions and restrictions made the system run more reliably. The improved reliability of the new SharePoint system led to time savings, and increased staff confidence and satisfaction with IT. The iShare pilots encouraged group members to look actively for opportunities to work jointly and share information.

As these pilots progressed and reported their successes to the wider office, they encouraged a more open outlook across teams – ‘look we shared stuff and worked together and it hasn’t all turned to custard’ as our kiwi cousins might say.
Tom also thought there was a trust dimension. Handling sensitive client information is part of an auditor’s day job. Therefore, opening up data was a clear signal that the OAG was a high trust environment.

However, change is a journey and the OAG report that its experience is no different. It continues to encourage and aim for an environment where information is open until closed. But it hasn’t always been plain sailing since introducing the new information system. Some staff have embraced the opportunity to openly share information. Others have been more hesitant in sharing information more or are yet to change what they have always done to be more open. The OAG has had to periodically promote and reinforce the new approach. It recognises that a change of this magnitude won’t happen overnight or without a sustained effort. But the end – using collective knowledge to influence improvement and improve accountability – justifies the effort.

How this fits with the work of the Good Practice Exchange

Our Good Practice Exchange work on effective data sharing shows that this relies on the principle of adopting proportionate steps when safeguarding data.

In a previous blog post on whether data sharing was a barrier to public service improvement, I included a quote from the Information Commissioner, which said ‘People want their personal data to work for them. They expect organisations to share their personal data where it’s necessary to provide them with the services they want. They expect society to use its information resources to stop crime and fraud and to keep citizens safe and secure.’ It’s also well worth watching Anne Jones, the Assistant Information Commissioner for Wales, outlining how data can be shared effectively.

The upcoming General Data Protection Regulation will ramp up the safeguarding of data a few notches, but it’s also an opportunity to reconsider how we can share data effectively. Particularly, how we make sure that auditors are confident enough to make the most of data collection and sharing.

Previously I have blogged about our staff trust event, where we heard that trust is essential if public services are to take well-managed risks, innovate and deliver public services that are truly fit for the 21st century.

Tom is leading on a separate project within the Wales Audit Office to look at how we’re using our information systems including SharePoint. One option we’re considering is the use of SharePoint Online, which would make it easier for us to develop an area that could be accessed by external bodies and partners – a portal. Leigh Dodds ‘s post provides a good overview of what a portal might contain.

A portal would allow us to share data with audited bodies and partners more effectively. We’re testing this concept with a SharePoint based prototype portal for some of our health colleagues. Learning from this will feed back into Tom’s project. And if working on the Cutting Edge Audit project has taught me anything, it’s that joined up and collaborative approaches are the best way to ensure we add real value to the work that we’re doing.

Acquiring data for a cutting edge audit office

How is the Wales Audit Office working to ensure that it provides audit that’s fit for the future? Dyfrig Williams blogs below on his work with the Cutting Edge Audit project.

Over the past few months, I’ve been working on the Cutting Edge Audit project, which looks at how the Wales Audit Office can challenge our existing use of data and technology and assumptions that we normally take for granted. We’re thinking radically about how we might use new technology to transform the way that we work.

It’s been a fantastic piece of work to undertake, which has really put that radical thinking into practice. The project’s being led by my colleague Steve Lisle, who is reporting directly to the Auditor General for Wales. This has meant that we’ve moved away from hierarchy into a much flatter structure. We’ve also been outcome focussed – we’ve been testing and prototyping as we go so that our risks are well managed and that we learn from failure.

I’ve been working on how the Wales Audit Office acquires data to give us deeper knowledge and fresh insight.

Data Maturity

data_maturityIt was helpful to think about Data Maturity when we were doing this work. Data Maturity is the journey towards improvement and increased capability in using data. Data Orchard have used this model (which I’ve nicked from a great post by Ben Proctor) of stages in an organisation’s development:

  1. Ad-hoc gathering of data in some areas
  2. Pulling data together centrally
  3. Starting to use data looking backwards
  4. Using data in real time to manage the organisation and move resources rapidly
  5. Modelling the future before making decisions to enable better decisions to be taken
  6. Modelling the future the organisation wants and working backwards to understand what needs to happen now to deliver that future

This is very much a journey for us as an organisation, but it has helped to inform my thinking. It’s helped me think about how we get to point 6, where we’re modelling the future that the organisation is working towards, and ensure that the things that I’m working on set us out on the right path beyond the lifespan of the Cutting Edge project.

My prototypes

I’ve been working on two different tests within this field. The first is an Open Data prototype, which has been more challenging than I expected because the Wales Audit Office is a secondary user of data. This means that we use data that is gathered by others, so we don’t always have the right to share it. I have managed to find a useful dataset though, so my next step is to set it free into the world and look at the challenges around how we can make it as useful as possible.

I’ve been putting the Good Practice Exchange’s principles into practice in this work by visiting other organisations to learn about the work they’re already doing because there’s no point reinventing the wheel. I’ve also been thinking about how we adapt rather than adopt their work to suit our organisational needs, because after all, a one-size-fits-all approach never works.

I’ve blogged before about why the public sector needs to start thinking about its approach to Open Data, and we subsequently ran a Google Hangout to look at why it’s an important topic. Hendrik Grothius has written an excellent blogpost on how organisations can start to publish Open Data, and it will be a brilliant starting point for me as I get to grips with this.

My second piece of work has been looking at how we enable our staff to make better use of data, thereby minimising the audit burden. I’m looking at how we can bring together data from public bodies in a way that makes it easy to access, open to everyone, and give us an improved insight into the performance of the Welsh public sector, and international comparators. I’ve been talking to our staff so that I can better understand what type of approach would be useful to them. I’ve developed personas to help guide our work in this area, which will shape the next phase of this work and ensure that my part of the final report is focused on user need.

Iteration

I’ll be writing future posts to share my approaches, what I’ve learnt and what I would do differently next time. We are working iteratively so that we learn from each development and how we can build on that learning going forwards. If those prototypes don’t work, we’ll be looking to learn from failure and see what the organisation can do differently in the future.

At all our Good Practice Exchange seminars we hear that public services can’t continue to work in the same way in these austere times. It’s been great working on a practical project at the Wales Audit Office, as we’re getting to grips with those same challenges and applying new thinking to our work.

The strategic importance of digital: a conference about culture change

What were the key messages from our recent events on digital? Kelly Doonan from Devon County Council reflects on the main learning points that she took away.

Image of speech bubble linking people to clouds, phonoes and documents

On 13 September I attended an event organised by the Wales Audit Office Good Practice Exchange called; Redesigning public services: The strategic importance of digital. Although I’ve referred to it as a conference for title alliteration purposes, it was actually a seminar event with interactive workshops – and some really fabulous catering – held at the SWALEC Stadium in central Cardiff.

This is my take on the event and the six key messages I came away with. Which, as the title suggests, aren’t actually about digital…

1. Digital means different things to different people… we need a clear understanding of what it means to us

The event kicks off with a speech from Auditor General, Huw Vaughan Thomas. In the speech he states; quite accurately, that: “Digital means different things to different people.”

It does and I think that is a huge problem. When he says that we need a clear understanding of what it means to ‘us’ I think we need one clear definition that everyone understands. It’s the only way that we can have aligned conversations and make aligned decisions.

Rotherham Metropolitan Borough Council has just released their new digital strategy (as a PDF) which explains that Rotherham is putting digital at the ‘forefront’ of their journey to become a modern authority. It links to local health digital strategies, but doesn’t seem to link to a wider culture change or service redesign strategy. Does digital mean the same to Rotherham MBC as it does to the WAO or to Devon County Council? Can we work together effectively if we don’t have an agreed definition?

2. Digital is not doing the same work, but digitally

Huw Vaughan Thomas goes on to clarify that: “Digital is not doing the same work, but digitally.”

Which begins to move us towards a definition of digital, and suggests that we’re starting to talk about culture change and service transformation, not creating a new digital strategy.

3. Mistakes are inevitable; we mustn’t shy away from that

Also from Huw Vaughan Thomas’ speech. This is an interesting one. If common sense was a thing this statement feels like it would be a classic example. Of course humans make mistakes; it’s one of our defining characteristics and how we know that we’re not actually machines surely? Still, it feels weirdly radical to have an auditor stand up and say this. It also feels hugely positive and (hopefully) liberating.

We have to move away from a culture that assumes all mistakes can be ‘policied’ out if only we policy hard enough. Instead we have to encourage reflection, learning and individual responsibility. Back to culture change again.

After the Auditor General’s speech there’s a quick fire question and answer session with the panel. The first questions are prepared by the organisers, but the rest are sourced from the audience – it’s a brilliantly engaging approach and works really well.

4. We can’t ‘do digital’ until we understand what citizens actually need

My cavalier approach to note-taking means that I don’t actually know which panellist said this, but it was definitely one of them.

I get an email every other day from a software development company telling me how their customer portal is going to revolutionise back office systems and save money. They’ve even got a snazzy customer testimonial video featuring a local authority IT manager explaining how this digital transformation has saved him pots of money and tidied up all his back office systems, and no-one ever ever mentions user needs.

We can’t put any digital tools in place until we know that we need them and that they’re solving the right problem – and surely we can only do that if we’re talking to our citizens? Surely we can only do that if we are clearly articulating our purpose and we understand why we’re doing anything at all? What we need is culture change and a different approach to understanding our citizens.

5. These things are not technology problems… digital is an enabler. Buying a load of iPads won’t change your culture.

Beautifully succinct quote from Professor Tom Crick in his workshop session, A digitally competent, digitally capable workforce. For me this session raises some really interesting questions about digital capabilities.

  • Is there a basic digital standard that our workforce needs to achieve?
  • If there is, then shouldn’t this be part of our job descriptions?
  • Do we have a hierarchy of digital capability in our workforce with a digital ‘elite’ who have lots of skills and are working in radically different ways to those further behind?
  • How do we make sure that staff are learning digital skills rather than learning how to use separate pieces of proprietary software?
  • Do we have senior leaders who know enough about digital to make these kinds of decisions?
  • Does every organisation essentially need a benevolent hacker at the top table wielding some real power?

Which is all to say that we probably need to look at changing our culture around staff training and recruitment.

Also in this workshop I share a story about a piece of work we did under the heading ‘ask forgiveness, not permission’ which literally makes another delegate’s mouth fall open in shock.

6. Can you build agile, interdisciplinary project teams that can work iteratively?

For the final session I attend the workshop Learning from the Digital Innovators Network run by Jess Hoare and Amy Richardson from Y Lab, which involves marshmallows and spaghetti.

Y Lab is an innovation lab for public service created by Nesta alongside Cardiff University. They have some wonderful, practical resources – most of which are available on the Nesta website.

The workshop involves a quickfire session answering some provocative questions such as ‘[In your organisation] What is the perceived role of IT?’ and ‘Can you build agile, interdisciplinary project teams that can work iteratively?’. We then identify a digital problem and use the Nesta tools, and Jess and Amy’s support and input, to work the issue through.

Fairly quickly we start talking about articulating the problem, identifying users, understanding needs and gathering evidence. We spend the rest of the session looking, essentially, at redesigning the service and the processes.

The problem with digital transformation

Every conversation I had at this event that started with digital transformation ended with looking at culture change and system transformation.

I think we do need to have an agreed definition of digital and it became clear through this event that many people – but definitely not all – understand that digital is an enabler and not an end in itself. I would say that we don’t need digital strategies (sorry Rotherham) rather we need system transformation strategies which include digital enablers. We need to start with purpose and start with users and understand what we’re for and what they need.

I think there’s a real opportunity here though. To start conversations about digital transformation and, through events like this, show how that conversation must move to one about system transformation.

WAO Good Practice Exchange are planning more events in this series and it would be great to see them challenging participants further to think about how we use digital as a catalyst for real organisational change – not just buying a load of iPads.

GovCamp Cymru 2016: Using behaviour change to improve public services

How can behaviour change theory help to embed ideas generated at unconferences into organisations? Dyfrig Williams outlines his pitch for GovCamp Cymru.

Logo GovCamp Cymru / GovCamp Cymru's Logo

This year will be my third GovCamp Cymru, which for the second year in a row will be held the National Assembly for Wales’ Pierhead Building.

For the uninitiated, GovCamp Cymru is an unconference, where attendees make the agenda by pitching what they’d like to talk about at the start of the day. I’ve avoided pitching so far, but having attended a few unconferences now I think that now’s the time for me to finally get involved.

Behaviour change

This year the Good Practice Exchange at the Wales Audit Office has been working on Behaviour Change Festivals across Wales, with the event in Swansea taking place in the run up to GovCamp Cymru. I’ve heard about some fantastic examples of behaviour change over the past few months – from the Chimp Shop App that helps people to cut down on their drinking to the WiFi that encourages people to move out of the sun.

I’m really interested in how Behaviour Change theory could be applied to help change to happen as a result of an unconference. I’ve found unconferences to be great events that enable people to develop their thinking and gain new contacts. Many unconferences are rightly proud that they attract passionate people who are prepared to give up their weekends to make public services better. But what happens when we get back to the office, get back to reality and have to persuade everyone else to buy into the brilliant ideas we’ve had or heard over the weekend? How do we persuade our colleagues to make that innovation a reality?

Some theory to get us started

This is what I’d like to examine in my proposed session. How do we bring all our colleagues along with us on the public service improvement journey? As a starter for ten, Chris Bolton has written a good post on getting ideas accepted. To break down his post to a very basic level (via a slightly brutal overview, sorry Chris!), people might:

  • Pretend they’re not a maverick
  • Get leaders on side
  • Wait until the organisation is likely to be receptive
  • Or find a host organisation that accepts you

Helen Bevan also has a great presentation which is directly aimed at change makers that suggests that people:

  1. Start with yourself
  2. Work out what might help others to change
  3. Build alliances
  4. Don’t be a martyr

So if these are starting points (come to my session if you disagree!), how can we enable positive behaviour and service improvement to take place as a result of unconferences? I’d also love to hear about examples of how people have got their colleagues to buy into changes in order to improve public services. I reckon that by pooling our experiences and our knowledge, we can go a long way to figuring out how we can better implement changes to improve our work.

What does modern Learning and Development look like?

How relevant is learning and development (L&D) within today’s workplace and does it have a positive influence? Russell Higgins of the Wales Audit Office recently completed a study to assess the impact of L&D within the workplace with particular focus on evaluation. For the study Russell used the Wales Audit Office as a case study organisation.

My research covered a variety of objectives which included how effectively and efficiently L&D needs were identified and delivered in the workplace, how to measure and quantify the relevance of L&D and how organisations can benefit from its effective measurement.

Identification of learning

The thing that struck me in the very beginning was that with financial budgets becoming tighter and tighter, it is essential that both public and private sector organisations deliver cost effective L&D solutions. In order to do this L&D professionals need to make sure that the learning is accurately identified and focuses on organisation’s priorities which in turn will bring a positive return on investment and expectation. The L&D solution should also aim to raise individual skills and motivate them to do things differently.

L&D needs can be identified in various ways – from an organisational point of view (a top-down process where the organisation is thinking about goals and vision) and via the appraisal process, where the line manager is key in identifying the right learning and development solution. The line manager therefore has a key role in the identification of L&D.

The role of line managers

Line managers have the opportunity to identify the L&D needs of the people they manage and can use this information to provide guidance and coaching.  Research findings suggest that this opportunity is frequently missed as managers do not always have the skills, confidence and / or motivation to identify and address the L&D needs. Indeed some research conducted by Penny Hackett stated that some line managers see all performance problems as training problems and expect trainers to provide solutions. If line managers are not knowledgeable about clear identification of L&D then it is likely that the learning identified will not be aligned to the organisational business strategy. Following my research I believe it is important that line managers have regular contact with members of staff throughout the year to discuss and review individual L&D requirements. Line managers should be skilled and knowledgeable enough to ensure that when L&D is identified it is delivered in the most appropriate manner and not only via the traditional classroom based manner.

The 70:20:10 model

A visiual representation of the 70:20:10 model, as described in Russell's postMy research found that learning and development was splitting onto 2 i.e. traditionalists and modern workplace learning. Traditionalists tend to focus on traditional classroom training or e-learning, whereas the modern workplace learning practitioner is more likely to work with line managers to develop the most appropriate way of learning, using the 70:20:10 model – 70% of the learning takes place in the workplace (on the job learning), 20% from other forms (like mentoring and coaching) and 10% through the traditional classroom method. This is a massive change for the way that staff learn and develop, and a big change for the L&D function so that they think about things in a different way. Modern workplace learning also puts the emphasis on getting people to take accountability for their own learning, rather than it being done to them.

A visual representation of the Kirkpatrick Model as a pyramid, as described in Russell's postThis therefore presents a challenge in terms of evaluating the impact of learning within your organisation. When thinking about the Kirkpatrick model of evaluation (there are loads of different models available, but this is the most common one used by L&D functions) which focuses on four key areas – reaction, learning, behaviour and results. My research found that very few organisations are actually looking at all four aspects, especially level four which is results – did the learning have any return on investment?

I also found that organisations tend to use a generic evaluation form, however quite often these should be tailored so that they fit the specific learning and development objective. In order to be useful to the organisation, the evaluation needs to go beyond the first two levels of the Kirkpatrick model (reaction and learning) and in order to do this, the line manager role is essential – have they seen a change in behaviour, has there been a return on investment on the activity?

I believe that L&D functions of the future need to be fully in touch with all departments to ensure that they are providing L&D interventions that are fully aligned to business requirements and organisational strategic objectives. There is also a joint dependency between the line manager and L&D function. They need to work together to ensure that all aspects of the Kirkpatrick evaluation model is followed.

To Charge or not to Charge?

Can councils make better use of their resources by charging for waste services? Gwyndaf Parry of the Welsh Local Government Association blogs for us on an event that they held with the Wales Audit Office.

Public services are coming under more and more pressure, with local authorities expected to deliver more with less. To deliver a wide range of services to residents and maintain that high quality of service, Councils in Wales have the legal powers to charge for a wide range of services, including Garden or green wastes, Bulky waste collections and replacement bins or recycling containers.

The event was hosted by the Welsh Local Government Association and facilitated by the Good Practice Team from the Wales Audit Office, to encourage and enable Welsh local authorities to share good practice when introducing a charge for waste collection services. Delegates were encouraged to participate in discussions and learn from others, helping to avoid making the same mistakes and to save costs and time.

Andy Phillips from the Wales Audit Office introduced the day and interestingly showed the income gained from waste services per resident, showing the difference between Wales, Scotland and England. Income earned in England has steadily being increasing since 2008/9 whilst reducing in Scotland and Wales.

A graph that shows that charging for Waste services has dropped in Wales and Scotland since 2008/9, but grown in England

Di Bradbury from Wirral Council shared her experience of introducing a charge for collection of garden waste, and how Wirral managed the introduction of a charge back in 2013/14. One of Di’s key pieces of advice was ensuring a robust IT system was in place from the outset, ensuring your IT system can handle customer registration, manage the customer database and manage payments. Di stressed that this was one of the most time consuming elements of introducing a charged service – people expect a high quality of service when they pay for a service. Missing a collection should be avoided, when building a reputable charged collection service.

Wirral acknowledged the importance of public consultation, as part of their waste planning they consulted with the public to gain their thoughts on charging. 51.7% of the respondents said they felt having to pay for a garden waste collection was completely unacceptable. However budget constraints meant the council had to push forward with a charge. However they ensured alternative options to residents including:

  • Online subscription discount of £5 (89% of customers use this option),
  • Shared bin option with neighbor, and
  • Promotion of home composting bins

A valuable lesson learned by Wirral was to offer a 14 day cooling off period, residents under law must be offered a 14 day period where they can receive a refund. In Wirral they only offered this refund if the resident had not received a collection.

As would be expected the tonnage of garden waste collected at the kerbside reduced in the Wirral, however the HWRCs saw a considerable increase in garden waste throughput. Overall garden waste tonnage reduced by 11% over a two year period. Contribution to overall MSW recycling rate decreased for two years post introduction of a charge, however in the third year the rate is seen to be increasing to just under its original state. Interestingly over the same period number of fly tipping incidents have continuously decreased.

Is the future green?

This first Workshop breakout session was hosted by Jim Espley from Denbighshire Council. Having introduced a charge for garden waste service on 30th March 2015, Denbighshire are the latest LA in Wales to charge, therefore had some valuable tips for other councils.

Since getting approval to introduce a charge in September 2014, Denbighshire had relatively short time period to introduce the service, key activities to ensure success were:

  1. Communicating with residents – introducing the new service available,
  2. Setting up a suitable IT system including payment processing system,
  3. Dealing with complaints and setting up suitable processes, and
  4. Buy suitable barcode and scanners for the bins.

12,500 (30%) properties signed up to the new service initially, by the end of the year this went up to 17,000 (40%) properties. Customers could sign up online (with a discount) or face to face, over the phone and at One Stop Shops. Denbighshire worked closely with their IT department to ensure a fit for purpose system was in place. A purpose built Database allowed them to capture, address details, collection day, assisted collection info, as well as other collection history. Every bin is issued with a barcode sticker and this is linked to the customer database. The database is also liked to a ‘Trackyou’ software system that has in-cab technology allowing the crew to monitor and record customer details in real-time. Helping Denbighshire offer their residents a top quality service.

A photo of the Track You device used by Denbighshire County Council

A valuable lesson that Jim shared with the group was that on-line subscriptions would ensure high quality data was fed into the IT database. Whilst a number of errors were experienced in customer details when hard copy paper work was completed. Therefore Denbighshire is working towards encouraging more and more to subscribe online.

How green is your valley?

A second workshop hosted by Carl from Monmouthshire was all about sharing lessons learnt, Monmouthshire have a well-established charged garden waste collection service. From July 2013 Monmouthshire have been charging for the collection of Garden waste. Having an initial charge of £8 per collection of a 90litres hessian reusable sack, by 2016/17 the charge has increased to £14 per sack.

Residents are issued with a free sack and must pay for the permit that is tagged on to the sack. Once again Monmouthshire reinforced the importance of having an IT system that was fit for purpose – this can make or break a successful service.

Number of residents signing up to the charged service has increased year on year in Monmouthsire, with households that tend to have larger gardens purchasing an increased number of bags. Carl also emphasized that an increased tonnage of garden waste was going into their HWRCs, therefore having suitable HWRCS in place that could manage the increased capacity was important.

Top Tip

A key message from the day was to invest and allocate time and resources into a suitable and fit for purpose IT system that can manage payments and manage customer data and information, making it easier to know who is signed up for the service and if their collections have been delivered or not. Customers expect a high quality service when they pay for it.

Learning from failure in complex environments

In his second blog post on the Learning from Failure workshop, Dyfrig Williams looks at failure in a complex environment.

It’s now been a few weeks since the Learning from Failure workshop, and my subsequent admission that I haven’t been very good at learning from my own failure. The event took place at the Wales Audit Office, so it was perhaps inevitable that we discussed the role of audit in learning from failure.

Systematic failure

James Reason Swiss Cheese Model. Source: BMJ, 2000 Mar 18:320(7237): 768-770

James Reason Swiss Cheese Model. Source: BMJ, 2000 Mar 18:320(7237): 768-770

Chris Bolton’s presentation was on the James Reason Swiss Cheese Failure Model, which compares human systems to layers of Swiss Cheese. Reason chose Swiss Cheese for a reason (see what I did there), as each layer is a defence against mistakes and errors, and things go badly wrong when the holes line-up. There’s an interesting critique of the model in the comments by Matt Wyatt of Complex Wales.

After a good Twitter conversation on the merits of different types of cheese as defence (I went patriotic and chose Caerphilly – ‘I crumble in the face of failure’), I looked at a model that Matt has developed, called the ‘Timeline of Inevitable Failure.’

Whereas the Swiss Cheese Model is a reflective model (you look back and check out the failure after Timeline of Inevitable Failureit’s occurred), Matt’s model is interesting as it offers opportunities to reflect on failure and its consequences at different stages, which fits in with a systematic approach to failure and chimes with some of the thinking in my last post on examining failure rigorously.

To be able to rectify failures at the early stage of the timeline, we have to be open and frank about failure, or issues will escalate and become bigger problems. By being comfortable with minor instances of failure, we’ll also be better prepared for when things go drastically wrong. As Matt says in another comment, ‘complex living systems will always fail, so instead of trying to make them failsafe, it’s much more useful to make them safe to fail.’ It’s well worth reading Chris’ post on Trojan Mice, which are safe to fail pilots, before delving in to a video of Dave Snowden discussing them as part of the Cynefin Framework.

You can see this approach in action through the work of the Bromford Lab and Dublin City Council’s Beta Projects. In terms of the latter, it’s worth checking out how their painting of traffic signal boxes led to less tagging and graffiti.

What does this mean for audit and audited bodies?

Aside for the recommendation  in the workshop to take your auditor out for lunch to better understand their approach to failure (which I’m completely on board with by the way!), this all relates to the complex environment in which public services are delivered and audited.

In Wales, this environment is about to change fundamentally with the Wellbeing of Future Generations Act. It’ll need a shift in thinking for organisations, as they’ll have to improve people’s wellbeing without compromising the ability of future generations to meet their needs. It’ll also be a challenge for us at the Wales Audit Office – it’s difficult to measure success when you don’t know what the future will look like. There’s a great post on the Wales Audit Office blog that outlines these challenges by Ann Webster, Assistant Auditor-General of New Zealand.

We’ve already shared steps that organisations can take to report effectively, including integrated reporting, at a seminar we held with the Sustainable Futures Commissioner. But in terms of this event, I was struck by some simple steps that organisations can take to evidence improvement. Jonathan Flowers gave a great example of how a Neighbourhood Network Scheme Manager asked for two instances a month of how the service had improved people’s lives. These narratives show that the service is moving in the right direction and can be used at the project evaluation stage.

Where now?

When it comes to evaluating our project, we’ve been gathering examples of how our work has led to organisations adopting good practice. These aren’t often measures in themselves, but complex case studies of how services have changed.

And in terms of our work, it’s important that we continue to have these conversations about failure, so that it’s normalised and people can be honest about it. And if we can do that, we’re in a better place to help organisations take further steps to improve their services.