Tag Archives: audit

Being open by default

How might an audit office open up its systems so that information becomes open by default? Dyfrig Williams spoke with Tom Haslam about the approach of New Zealand’s Office of the Auditor-General.

The logo of the Office of the Auditor-General New Zealand

As part of the Wales Audit Office’s Cutting Edge Audit project, I am working on an Open Data prototype. During this work, colleagues told me that we could improve our approach to data. Not acquiring new data though – most colleagues said their biggest issue was better knowledge of, and access to, data that the office already held.

Our organisation has two specialist practices – financial audit and performance audit. This division facilitates specialism, so that we have colleagues with incredibly good knowledge in their fields of expertise. However, it also means that we have to work hard to break down organisational silos, sometimes reinforced by the systems we have in place.

Safeguarding data is an important feature of the way we have set up our information systems. Network folders are protected. Access is only available to specific teams and personnel, which means that the data within them is closed to others by default. Our SharePoint system is also set up in a similar way and the search functionality is not as good as it might be. All of this means that unless you know where the data is held, you’re unlikely to find it.

Learning from other audit offices

In my last post on the Queensland Audit Office’s work, I mentioned a well-travelled colleague called Tom Haslam. Tom has worked at the Office of the Auditor-General (OAG) in Wellington, New Zealand. And while there, the OAG identified similar problems with how they organised and held their data.

To address this, the OAG implemented a new SharePoint-based information system and complemented this with some pilot cross-office groups known as ‘iShare’. These groups were based around cross-cutting functional topics (for example the Transport iShare) with the aim of helping to break down organisational silos and promote a one-team approach across the office.

Adopting a new information system gave the OAG an opportunity to debate the relative merits of information systems being open or closed by default. This was discussed across the office through various channels.

The previous information systems had encouraged a mainly ‘closed until open’ approach. But the general feeling was that closed data might prevent the office from making the most of the information that they held. The natural tendency of all auditors is to be cautious, so under a ‘closed unless open’ approach, setting information as ‘open’ might be viewed as a risk best avoided, even if this approach wasn’t justified. On a practical level, having information closed off requires various permissions and access rights to be set up. This alone can be a barrier to sharing data.

The OAG structured its new information system so that information was ‘open unless closed’ with metadata to help staff find what they wanted. This approach facilitated sharing, encouraging staff to think about how they could add value by joining up information. A default setting of ‘open until closed’ made staff think more carefully about why they should want to close off access, for example material with national security implications or identifiable personal information.

On a technical level, a cleaner configuration of the IT system without endless permissions and restrictions made the system run more reliably. The improved reliability of the new SharePoint system led to time savings, and increased staff confidence and satisfaction with IT. The iShare pilots encouraged group members to look actively for opportunities to work jointly and share information.

As these pilots progressed and reported their successes to the wider office, they encouraged a more open outlook across teams – ‘look we shared stuff and worked together and it hasn’t all turned to custard’ as our kiwi cousins might say.
Tom also thought there was a trust dimension. Handling sensitive client information is part of an auditor’s day job. Therefore, opening up data was a clear signal that the OAG was a high trust environment.

However, change is a journey and the OAG report that its experience is no different. It continues to encourage and aim for an environment where information is open until closed. But it hasn’t always been plain sailing since introducing the new information system. Some staff have embraced the opportunity to openly share information. Others have been more hesitant in sharing information more or are yet to change what they have always done to be more open. The OAG has had to periodically promote and reinforce the new approach. It recognises that a change of this magnitude won’t happen overnight or without a sustained effort. But the end – using collective knowledge to influence improvement and improve accountability – justifies the effort.

How this fits with the work of the Good Practice Exchange

Our Good Practice Exchange work on effective data sharing shows that this relies on the principle of adopting proportionate steps when safeguarding data.

In a previous blog post on whether data sharing was a barrier to public service improvement, I included a quote from the Information Commissioner, which said ‘People want their personal data to work for them. They expect organisations to share their personal data where it’s necessary to provide them with the services they want. They expect society to use its information resources to stop crime and fraud and to keep citizens safe and secure.’ It’s also well worth watching Anne Jones, the Assistant Information Commissioner for Wales, outlining how data can be shared effectively.

The upcoming General Data Protection Regulation will ramp up the safeguarding of data a few notches, but it’s also an opportunity to reconsider how we can share data effectively. Particularly, how we make sure that auditors are confident enough to make the most of data collection and sharing.

Previously I have blogged about our staff trust event, where we heard that trust is essential if public services are to take well-managed risks, innovate and deliver public services that are truly fit for the 21st century.

Tom is leading on a separate project within the Wales Audit Office to look at how we’re using our information systems including SharePoint. One option we’re considering is the use of SharePoint Online, which would make it easier for us to develop an area that could be accessed by external bodies and partners – a portal. Leigh Dodds ‘s post provides a good overview of what a portal might contain.

A portal would allow us to share data with audited bodies and partners more effectively. We’re testing this concept with a SharePoint based prototype portal for some of our health colleagues. Learning from this will feed back into Tom’s project. And if working on the Cutting Edge Audit project has taught me anything, it’s that joined up and collaborative approaches are the best way to ensure we add real value to the work that we’re doing.

How Queensland Audit Office uses data analysis to improve its auditing

What can we learn from the way that the Queensland Audit Office uses data analytics in their audits? Dyfrig Williams had an early morning phone conversation to find out.

As I previously mentioned on my post on the Wales’ Audit Office’s Cutting Edge Audit project, I’m looking to identify ways of making better use of data and technology in order to transform the way that we work. I’d come across this article on how Queensland Audit Office are using audit data analytics to gain greater insight, and through my well-travelled colleague Tom Haslam I managed to organise an early morning conversation between myself and my colleagues (Steve Lisle and Nigel Blewitt) and Daniele Bird, the Assistant Auditor-General for Performance Audit in Queensland, Ben Jiang the Assistant Director for Audit Analytics and David Toma, their Assistant Director for Performance Audit Services.

What is the Queensland Audit Office doing?

Back in 2014, they decided to look at opportunities to use more sophisticated data visualisation tools in order to get a better insight into their audited bodies. They trialled and tested Qlikview, which can be accessed via the web or desktop software. They set up a data analytics team in 2015, all of whom were former auditors branching out into the world of data analytics. The team undertook initial training, but their development from there was centred on self-learning. The team first planned to work on Performance Audit. However the focus moved to Financial Audit, as the recurring datasets meant they could work more efficiently and effectively, compared to the very different types of datasets that are needed for each performance study.

The importance of relationships

Our seminars on early closure of accounts in 2015 and 2016 have made it clear to me that both the auditor and the audited body have to work together closely in order for audit to work more effectively. Whilst the Queensland Audit Office had built processes and automatic systems to collect data, they still faced questions from clients about security and what they were going to do with the data. They built an internal Frequently Asked Questions page so that audit team leaders could re-assure organisations and build trust, and they also took the burden of change away from the audited bodies by enabling them to dump the unformatted data with the office. The analytics team did the work to transform and clean up the data, and they’re now able to offer unique insights as they gather data from such a wide range of bodies. They can quickly identify and access an overview of issues, as well as benchmark clients’ performance and show how they compare to their peers (which you can see in the slide below). They also use the tools to undertake exercises like Benford’s Analysis, which clearly identifies outliers. This is particularly useful in identifying fraud.

A graph showing lots of dots togther and a clear outlier on the value of bad debt

How Queensland Audit Office gains insight to the value of bad debt

Performance Audit

The Data Analytics Team is now also supporting Performance Audit staff. The team is using text mining, for example to see what teachers are identifying as development goals. This has enabled them to produce Wordclouds and text summaries of the top 10 statements. They also use these tools to sweep the text of Hansards.

The team is also looking at how it can help staff to choose performance audit topics. It’s running sentiment analysis tools on Twitter feeds in order to support the strategic planning process. This helps them to understand the issues that are out there in the wider world and what people are saying about them. Although the technology has been built for Twitter, it could also be used for networks like LinkedIn, which would enable auditors to access different users and perspectives. They’ve also built social media analysis tools that rank tweets. It grades them and enables auditors to focus resources on important messages.

Creating a positive working environment

Daniele, Ben and David acknowledged the importance of leadership from the top in creating the right environment for the work of the Data Analysis Team to flourish. They’ve received high level support from the previous Auditor General and the current Acting Auditor General. This is something that we’ve seen from our own Auditor General in the development of the Cutting Edge Audit project, and hopefully the findings from our own work can help to provide a base for us to develop our own ways of working. It was really interesting to hear how the organisation is continuously looking to develop their work to build on current practices. They’re currently looking at whether moving from Qlikview to Qlik Sense could provide further advantages.

The work of the Data Analytics Team has achieved efficiencies, mainly around saving time that can be reinvested in future work, but they’re also looking to save more to do more. The rapidly changing environment and limited public finances presents a challenge for every public service. Queensland Audit Office have faced that challenge head on and are working in a different way to provide a better service. Not only that, but their staff’s work is now more meaningful and interesting, and auditors are further developing their critical thinking skills. I’ll be thinking about our Good Practice Exchange principles as we reflect on our conversation with the Queensland Audit Office, so that we think about how we might adapt this approach in order to ensure that our work, like the services we audit, are delivering the best possible outcomes for the people of Wales.

Developing personas for a user focused service

How can personas help to ensure that projects are user focused? Dyfrig Williams reflects on how they’ve been used for his work on acquiring data effectively.

An image of personas for Performance Audit staff

As part of my Cutting Edge Audit work, I’ve been working on a test project to gather data for auditors so that we can work more efficiently.

I’ve blogged before on why I’m really interested in the digital delivery of public services because of the relentless focus on user need. This focus isn’t new to me – my previous work on citizen engagement made it clear to me that properly involving the people who access services at an early stage results in a service that is effective at targeting its resources.

Now that I’m four months in to a six month project you could argue that I’m leaving it a bit late to start on the practical part of the work. But in my public engagement role I heard time and time again that staff that didn’t involve people properly delivered their work according to existing preconceptions about what people wanted or needed. I was determined not to make the same mistake with my work, so I developed personas for the main users of data to ensure that we are focusing on building something that people would actually find useful.

Personas are representations of different types of customers or users. They answer the question “Who are we designing our work for?” They help to align strategy and goals to specific user groups. As we’re testing this work with the Health Performance Audit Team, I developed personas ranging from Performance Support Officers to Audit Managers, as well as complimentary work with members of the Financial Audit Technical Team and a Financial Auditor working on the frontline with health boards.

Where to start?

I was quite excited to start on the personas so that I could see how my public engagement mindset fitted with digital practice. My colleague Louise Foster-Key, who is the Digital Comms Officer at the Wales Audit Office gave me a few pointers from her work on the Wales Audit Office Intranet, and I also looked at wider good practice. There’s a great post on user personas on the Office for National Statistics’ Digital Blog, but I ended up basing my work on the Government Digital Service (GDS)’s Guide to User Stories. I had a punt at using Trello, but as this was a largely solo exercise (and it was only a small project to manage) I didn’t find it to be as useful as I expected. Instead I used Xtensio, a free(ish) tool that Louise recommended to me.

I stuck with the GDS format as the structure of my user stories, which I put together from interviews and emails with staff in key roles:

  1. As a… (Who is the user? A biography of the person accessing data)
  2. I need/want/expect to… (What does the user want to do?)
  3. So that… (Why does the user want to do this?)
  4. It’s done when… (This is the GDS acceptance criteria, which is a list of outcomes that you use as a checklist to confirm that your service has done its job and is meeting that user need)

Has it been useful?

By talking to auditors in depth about each of the roles that I’ve mapped, I’ve got a much better understanding of their work. This will be useful to me far beyond the lifespan of the Cutting Edge Audit project, as I look at how the Good Practice Exchange can work more effectively with audit teams, especially the Health Audit Team, who volunteered to work with us on this piece of work.

The main use of the personas will be to sense check our work and to ensure that the purpose of our test matches what our auditors want and expect. Now that they’ve helped us run through some hypothetical tests, we’ll use their thoughts to avoid scope creep for the first stage of our iteration.

Their thoughts will also be incredibly useful for my section of our final report. It’s been said that “The pen is mightier than the sword”, and in this sense we as the report authors have all the power – it is us who provide the recommendations for future priorities. This work will ensure that my recommendations are grounded in the realities of our staff’s day to day work.

The final personas are now online and have been split into two sections because of the limitations of the free version of Xtensio. The first set of personas are based on the Performance Audit roles, and the second set include personas based on the Financial Audit and Technical Team roles.

These personas now give us a level of expectation for our work. If the test fails, they give us a rigorous criteria to examine our output against, and a clear vision to check our delivery against. On the other side of things, should the work (as we hope) be a success, we have an opportunity to think about whether personas can help us to be more focused and responsive as we ensure that public services really are delivering value for money for the people of Wales.

Acquiring data for a cutting edge audit office

How is the Wales Audit Office working to ensure that it provides audit that’s fit for the future? Dyfrig Williams blogs below on his work with the Cutting Edge Audit project.

Over the past few months, I’ve been working on the Cutting Edge Audit project, which looks at how the Wales Audit Office can challenge our existing use of data and technology and assumptions that we normally take for granted. We’re thinking radically about how we might use new technology to transform the way that we work.

It’s been a fantastic piece of work to undertake, which has really put that radical thinking into practice. The project’s being led by my colleague Steve Lisle, who is reporting directly to the Auditor General for Wales. This has meant that we’ve moved away from hierarchy into a much flatter structure. We’ve also been outcome focussed – we’ve been testing and prototyping as we go so that our risks are well managed and that we learn from failure.

I’ve been working on how the Wales Audit Office acquires data to give us deeper knowledge and fresh insight.

Data Maturity

data_maturityIt was helpful to think about Data Maturity when we were doing this work. Data Maturity is the journey towards improvement and increased capability in using data. Data Orchard have used this model (which I’ve nicked from a great post by Ben Proctor) of stages in an organisation’s development:

  1. Ad-hoc gathering of data in some areas
  2. Pulling data together centrally
  3. Starting to use data looking backwards
  4. Using data in real time to manage the organisation and move resources rapidly
  5. Modelling the future before making decisions to enable better decisions to be taken
  6. Modelling the future the organisation wants and working backwards to understand what needs to happen now to deliver that future

This is very much a journey for us as an organisation, but it has helped to inform my thinking. It’s helped me think about how we get to point 6, where we’re modelling the future that the organisation is working towards, and ensure that the things that I’m working on set us out on the right path beyond the lifespan of the Cutting Edge project.

My prototypes

I’ve been working on two different tests within this field. The first is an Open Data prototype, which has been more challenging than I expected because the Wales Audit Office is a secondary user of data. This means that we use data that is gathered by others, so we don’t always have the right to share it. I have managed to find a useful dataset though, so my next step is to set it free into the world and look at the challenges around how we can make it as useful as possible.

I’ve been putting the Good Practice Exchange’s principles into practice in this work by visiting other organisations to learn about the work they’re already doing because there’s no point reinventing the wheel. I’ve also been thinking about how we adapt rather than adopt their work to suit our organisational needs, because after all, a one-size-fits-all approach never works.

I’ve blogged before about why the public sector needs to start thinking about its approach to Open Data, and we subsequently ran a Google Hangout to look at why it’s an important topic. Hendrik Grothius has written an excellent blogpost on how organisations can start to publish Open Data, and it will be a brilliant starting point for me as I get to grips with this.

My second piece of work has been looking at how we enable our staff to make better use of data, thereby minimising the audit burden. I’m looking at how we can bring together data from public bodies in a way that makes it easy to access, open to everyone, and give us an improved insight into the performance of the Welsh public sector, and international comparators. I’ve been talking to our staff so that I can better understand what type of approach would be useful to them. I’ve developed personas to help guide our work in this area, which will shape the next phase of this work and ensure that my part of the final report is focused on user need.

Iteration

I’ll be writing future posts to share my approaches, what I’ve learnt and what I would do differently next time. We are working iteratively so that we learn from each development and how we can build on that learning going forwards. If those prototypes don’t work, we’ll be looking to learn from failure and see what the organisation can do differently in the future.

At all our Good Practice Exchange seminars we hear that public services can’t continue to work in the same way in these austere times. It’s been great working on a practical project at the Wales Audit Office, as we’re getting to grips with those same challenges and applying new thinking to our work.

Who knew that Faster Closure of Local Government Accounts would benefit front line service provision?

How can faster closure of local government accounts improve public services? Ena Lloyd examines what we can learn from English Local Authorities.

Faster closure of local government accountsCurrently, Local Government Accounts have to be signed off by 30 September in Wales. The Treasury in Whitehall would like to enable the Whole of Government Accounts to be made available by that date, and so there is every likelihood that the sign off date is going to move forward to 30 July in a few years’ time.

I’ve been chatting to a number of Local Authorities across England where they have already brought the date forward of Faster Closure of Local Government accounts, to hear their thoughts on their approaches. There seem to be three themes emerging from the discussions:

Technology

The reality is, technology has evolved, and by now it’s reasonable to expect that most financial processes are automated. So, at worst, ‘ faster closure’ can be seen as an opportunity to review current financial systems. Supporting financial teams to work smarter, not harder. Also it’s a great opportunity to pause and reflect to think about building in resilience into the process too.

The Band Aid Approach

The opening thoughts seemed to have a clear theme: there is no hiding from the fact that change is tough. Especially, when financial colleagues have been used to the same processes and worked to the same timescales for a number of years. It’s understandable. I can hear the argument that, if it’s not broken…………..

The idea of change was worse than the reality. That may sound rather a glib comment to make. Many reflected on the fact many financial colleagues spent more than half of their working year on ‘closing down the accounts’. The impact of this on colleagues left them feeling as if they were working in the past and in a kind of bubble too.

The benefit of ‘faster closure’ for financial staff is that they spend far less time focusing on the past, and, given that I work for the Wales Audit Office, I say the next bit with a smile on my face, the external audit is over much quicker. The relationship with the external auditor has changed in that they are involved much more in year, sharing their thinking and potential changes. Adopting a ‘no surprises approach’.

Benefits Realisation

Probably the biggest benefit of ‘faster closure’ is the improved service financial colleagues now provide front line services. They are able to devote far more time to budget setting and monitoring. More importantly, by engaging at a much earlier stage with Service Managers about what management information they need to run their service, can only be of benefit to public services.

And isn’t that what it’s all about at the end of the day?

Learning from failure in complex environments

In his second blog post on the Learning from Failure workshop, Dyfrig Williams looks at failure in a complex environment.

It’s now been a few weeks since the Learning from Failure workshop, and my subsequent admission that I haven’t been very good at learning from my own failure. The event took place at the Wales Audit Office, so it was perhaps inevitable that we discussed the role of audit in learning from failure.

Systematic failure

James Reason Swiss Cheese Model. Source: BMJ, 2000 Mar 18:320(7237): 768-770

James Reason Swiss Cheese Model. Source: BMJ, 2000 Mar 18:320(7237): 768-770

Chris Bolton’s presentation was on the James Reason Swiss Cheese Failure Model, which compares human systems to layers of Swiss Cheese. Reason chose Swiss Cheese for a reason (see what I did there), as each layer is a defence against mistakes and errors, and things go badly wrong when the holes line-up. There’s an interesting critique of the model in the comments by Matt Wyatt of Complex Wales.

After a good Twitter conversation on the merits of different types of cheese as defence (I went patriotic and chose Caerphilly – ‘I crumble in the face of failure’), I looked at a model that Matt has developed, called the ‘Timeline of Inevitable Failure.’

Whereas the Swiss Cheese Model is a reflective model (you look back and check out the failure after Timeline of Inevitable Failureit’s occurred), Matt’s model is interesting as it offers opportunities to reflect on failure and its consequences at different stages, which fits in with a systematic approach to failure and chimes with some of the thinking in my last post on examining failure rigorously.

To be able to rectify failures at the early stage of the timeline, we have to be open and frank about failure, or issues will escalate and become bigger problems. By being comfortable with minor instances of failure, we’ll also be better prepared for when things go drastically wrong. As Matt says in another comment, ‘complex living systems will always fail, so instead of trying to make them failsafe, it’s much more useful to make them safe to fail.’ It’s well worth reading Chris’ post on Trojan Mice, which are safe to fail pilots, before delving in to a video of Dave Snowden discussing them as part of the Cynefin Framework.

You can see this approach in action through the work of the Bromford Lab and Dublin City Council’s Beta Projects. In terms of the latter, it’s worth checking out how their painting of traffic signal boxes led to less tagging and graffiti.

What does this mean for audit and audited bodies?

Aside for the recommendation  in the workshop to take your auditor out for lunch to better understand their approach to failure (which I’m completely on board with by the way!), this all relates to the complex environment in which public services are delivered and audited.

In Wales, this environment is about to change fundamentally with the Wellbeing of Future Generations Act. It’ll need a shift in thinking for organisations, as they’ll have to improve people’s wellbeing without compromising the ability of future generations to meet their needs. It’ll also be a challenge for us at the Wales Audit Office – it’s difficult to measure success when you don’t know what the future will look like. There’s a great post on the Wales Audit Office blog that outlines these challenges by Ann Webster, Assistant Auditor-General of New Zealand.

We’ve already shared steps that organisations can take to report effectively, including integrated reporting, at a seminar we held with the Sustainable Futures Commissioner. But in terms of this event, I was struck by some simple steps that organisations can take to evidence improvement. Jonathan Flowers gave a great example of how a Neighbourhood Network Scheme Manager asked for two instances a month of how the service had improved people’s lives. These narratives show that the service is moving in the right direction and can be used at the project evaluation stage.

Where now?

When it comes to evaluating our project, we’ve been gathering examples of how our work has led to organisations adopting good practice. These aren’t often measures in themselves, but complex case studies of how services have changed.

And in terms of our work, it’s important that we continue to have these conversations about failure, so that it’s normalised and people can be honest about it. And if we can do that, we’re in a better place to help organisations take further steps to improve their services.

Simple questions can make a difference

How can board members help to improve waiting times? Verity Winn from the Performance Audit Health Team looks back at our ‘Asking the Right Questions’ seminar.

NHS Waiting Times ReportPutting on an event to publicise a checklist sounds like an auditors dream… But how do you make it interesting for Independent Members of health boards with busy schedules and no shortage of important issues on their plate? The answer is in making it relevant and focused on practical things they can do to make a difference for patients. And also to create a safe space where they can feel comfortable sharing and working through the issues with people who are in the same boat.

For me a lot was riding on our ‘Asking the right questions’ event on 21st May. I spent most of last year buried in information about the NHS in Wales as one of the team working on our report on NHS Waiting Times. Our findings were worrying – waiting times are getting longer, some patients are coming to harm and more could be done about it. We did a lot of work to understand why waiting times are getting worse including the pressures on the NHS in terms of money, staff, beds and other resources. We also looked at what could be done differently and identified some significant opportunities for health boards to make better use of their existing resources.

All this work filled four reports – but we had the sneaking suspicion that not everyone would have time to read through all of it! We wanted to get our message across in the simplest way to the people who can use it to make a difference. Independent Members have a vital role in scrutinising health boards’ performance and holding them to account. We wanted to create a tool to help them ask the right questions to challenge their health boards – not just about whether waiting times are getting longer, but to understand what they’re doing to bring them down. The tool is our checklist. It sets out a series of questions to understand how the health board is performing on waiting times and how strong their plans to improve are.

The event was a new experience for me. Working in the central national studies team, we tend to move onto a new topic once our reports are published. That means we don’t often get to see the local impact our work has, or meet the people who can use our reports work to make change happen. But this event brought us face to face with Independent Members and reminded me that sometimes the simplest way to get a message across is by talking to people.

The event started with the spotlight on Dave Thomas, Director of the Performance Audit Health Team at the Wales Audit Office and Helen Birtwhistle, Director of the Welsh NHS Confederation. They answered questions about some of the challenges and opportunities facing the NHS in Wales and what needs to be done to improve waiting times for patients. You can watch the full discussion below. The rest of the day was filled with practical workshops based on the topics in our checklist.

It was great to see people sharing experiences and advice in such an open and supportive way. And really positive to see the commitment of Independent Members to putting the patient first and not shying away from difficult questions to make sure this happens. I was particularly interested to hear the suggestions of questions to ask health boards to understand what they’re doing to improve waiting times. At the end of the event, one person told us that the key thing he learnt was to ask questions in ‘the most simple language, hoping to get to the nub of the point’ – for me, he hit the nail on the head.

I was really pleased that the feedback we had from the event was overwhelmingly positive. Not that there isn’t always something to learn about how we could do things a little better next time. We are holding another event on ‘Asking the right questions’ in the Metropole Hotel in Llandrindod Wells on 13 July. If you’d like a place, please email good.practice@audit.wales. I look forward to meeting more of you there.