Tag Archives: cutting edge audit

Innovative audit: Learning and sharing with the Netherlands Court of Audit

Flags from the Netherlands' different provinces line the waterside in the Hague

Darllenwch y flogbost yn Gymraeg

What can the Wales Audit Office learn from the Netherlands Court of Audit’s approach to innovation? Dyfrig Williams visited The Hague to see how they work and to share the work of the Good Practice Exchange.

In May I had the opportunity to visit the Netherlands Court of Audit to learn more about their Doen project (which means ‘Doing’) and their approach to innovation. Their whole approach stems from learning by doing, which is a fantastic mantra to have. Throughout the visit, staff openly shared their successes and failures from very practical change projects. Their safe to fail space mindset is key for providing the space for that learning and improvement.

The month before I had shared the work of the Good Practice Exchange with Mark Smolenaars and Sanne Kouwenhoven from the Netherlands Court of Audit in Cardiff. Myself, Steve Lisle (who is leading on the Cutting Edge Audit project) and Mike Usher (our Sector Lead for Health and Central Government, and who also leads our Investigative Studies work) were fortunate enough to be invited to the Netherlands to share our work with their colleagues, and also to learn more about their approach to innovation, so that we can adapt their approaches to suit our needs.

Looking at audit differently

At the Good Practice Exchange we always say that there is no “one-size fits all” approach. We may have many things in common with the Netherlands Court of Audit (we have about the same number of staff and therefore roughly the same challenges in identifying capacity and scaling up change initiatives), but there are also some differences too (the Dutch public sector includes casinos – I can’t even imagine what that audit looks like!).

When we started looking at their approach to innovation, one of the key things that struck me was that they have a variety of staff that contribute very different things to the organisation. Not only does this avoid a groupthink mentality, but it also brings a lot of different skillsets to their work.

We had a presentation from Linda Meijer, who is an auditor who has become a designer. It was fascinating to hear how her design skills led to her asking different questions of data as she illustrated their findings. This was particularly useful on an audit of products that have the CE mark, which shows that products are safe and can be sold in the European market.

We also discussed how the Netherlands Court of Audit have an Investigative Journalist working for them, which was particularly useful for us at the Wales Audit Office as we have our own Investigative Studies team. Jaco Alberts’ insight was fascinating as he talked about how he applies his expertise to his role.

We also heard about how they shared their expertise through a participatory audit of higher education institutions. They asked members of each student council to check pre-investments and to look at how the institution’s budget framework is working, which meant that they had the opportunity to make the most of the knowledge that is available within institutions. This fascinating approach gave us food for thought as we thought aloud about working to the Wellbeing of Future Generations Act, as one of the five ways of working that’s outlined is involvement.

Data

One of the areas that really fascinated me was the Netherlands Court of Audit’s use of data, as I have been leading on the Wales Audit Office’s work on acquiring data as part of our Cutting Edge Audit work. Their involvement of people with different backgrounds even filters into their data work, as we heard how they make the most of data interns.

As with any practice sharing, it was fascinating to hear how the Netherlands Court of Audit are dealing with challenges as well. Roline Kamphuis shared how the organisation have set up communities of practice around R, which they hope will help the organisation to free up the capacity of its data staff. We heard how they currently spend much of their time cleaning data, which means that they can make shared data available for wider use. By cleaning the data effectively at the start of the audit, they make work much easier for staff as it progresses.

They also spoke about why they particularly use R and SPSS, as they have a script that can be used for an audit trail. By developing a pre-programmed script in R, they can ensure that outputs fit house style. Also because R generates word and excel documents, it means that auditors can see easily see and understand the steps that have been taken when working on that data.

A photo of Steve Lisle presenting the Cutting Edge Audit work to the NCA

Steve Lisle presenting the Cutting Edge Audit work to the Netherlands Court of Audit

Rudi Turksema shared how the Netherlands Court of Audit had run an accountability hack, which meant that they were able to involve external stakeholders in their data work. By working in partnership with a range of organisations, they were able to open up access to a wide variety of datasets on the day. I mentioned that the Good Practice Exchange have supported the NHS Hack Day in Cardiff, but that I wasn’t sure that we had enough capacity as an organisation to run such an event yet. Yet they encouraged us to look at putting a similar event on as it had helped them to share their work and involve a wide range of people. We’ve already seen what happens when you make data open and accessible, as Ben Proctor and the Open Data Institute have already started using Google Fusion Tables to create maps and add value to our first open dataset. This certainly gave me food for thought for how such an event might help us to further develop our use of data and our own data maturity.

I learnt so much from my trip to the Netherlands. It was fascinating to see the parallel journeys that both of our organisations are taking to ensure that audit is fit for purpose in the twenty first century. Thanks to everyone at the Netherlands Court of Audit for your welcome and your hospitality – you’ve certainly helped inform our learning journey and your expertise has been invaluable in helping us to move forward with our own innovation work.

Making use of Open Data

Darllenwch y flogbost yn Gymraeg

The Wales Audit Office recently released our first Open Dataset. What happened next? Ben Proctor of Open Data Institute Cardiff talks us through how he made use of the data.

A screenshot of a dynamic map created by Ben Proctor to show levels of Council Tax per head of population in Wales

A screenshot of a dynamic map created by Ben Proctor to show levels of Council Tax per head of population in Wales

Oooo! new data

I was excited to see that the Wales Audit Office had released a set of data as open data. Open data is data that anyone can find access and use and it is the most useful sort of data.

Dyfrig Williams wrote about the process they’d gone through to release this data set (a summary of the audit data from each local authority in Wales for each year). The data is a simple table and you can download it as a CSV file (essentially a file that will work in any spreadsheet programme) here.

But there are problems

I downloaded the file and quickly spotted some problems. These are not errors exactly but just things that are missing or inconsistent and will make some uses of the data a bit harder. But this is not a complaint, because one of the attractive features of open data is that I could resolve these problems. I can do this because the Wales Audit Office have released the data under the Open Government Licence. This tells me I don’t need their permission to do anything with the data and there are no limits to what I can do with it (apart from I have to make it clear where it came from).

I can fix the problems

These are the things I did to my copy of the data.

I changed the format of the “financial year column” because in the Wales Audit Office file some of these are numbers and some are text.

I added a column of GSS codes. GSS codes are codes that are used to identify local authorities (and other boundaries). Having the GSS code means you don’t have to worry about whether the data says Anglesey Council, or Isle of Anglesey Council or Ynys Môn. And with the GSS code I could add “polygons” for each council. Polygons are basically instructions on how to draw the outline of each council and information about where to put the drawing on a map.

With these changes I was able to draw a series of maps showing the level of council tax per head in each local authority and how this has changed over time.

And given the Wales Audit Office an improved file

And I’ve been able to hand back to the Wales Audit Office a KML file. This is a file suitable for use in mapping software. Anyone who wants to visualise the Wales Audit Office data on a map can just open the KML file and get going.

You can download this mapping file yourself.

Why did I do this?

I’m part of the core team at ODI-Cardiff so I get excited about open data.
It took me a very few minutes.
I’m trying to get better at using a Google service called Fusion Tables and this is a good opportunity to experiment.
I’m actually quite interested in what this data might tell us.

Releasing Open Data

Darllenwch y flogbost yn Gymraeg

How can organisations release Open Data? Dyfrig Williams looks at the process of making the data behind the Wales Audit Office’s Local Government Financial Statements report freely available, and shares the dataset at the bottom of this post.

I’ve previously blogged on how the Wales Audit Office is looking to challenge our existing use of data and technology as part of the Cutting Edge Audit project. My role on the project has been to look at how we acquire data.

How does Open Data fit into acquiring data?

The below diagram shows the rationale behind my work. For me, we need to share the data that we have in order to develop our relationships with our client bodies so that we can gather data effectively. Part of this is about “being the change that you want to see.” Auditors are sometimes seen as being risk averse, but in the Good Practice Exchange we’ve seen that when we work differently, we enable others to do the same. A number of local authorities have reported back to us about how they’ve been able to challenge the limitations of the websites that they’re able to visit and the social media that they can access because of how we share knowledge. By making data openly available, we can demonstrate that there is little risk, as long as the process is well managed.

A cycle illustrating that by open data results from how we share data and how it feeds into acquiring data

As I mentioned in my original post, finding an appropriate dataset was more challenging than I thought that it might be. As we often don’t have the right to share the data that we collect from clients during our audits. However after a bit of research, we found the data behind the Local Government Financial Statements report, which is a report on local government bodies’ accounts. This was safe data to release because it’s already available on each council’s website as part of their accounts, but we are the only organisation that collates this data. The data within the report is analysed on a national basis, but by releasing the dataset we can enable councils and other interested stakeholders to look at the data on a county by county basis and to compare and contrast their accounts against others. The data is used by the Wales Audit Office to support local audit work and for general benchmarking. The report itself looks at the quality of accounts, and is based on the data that’s released before amendments – we don’t keep track of prior-period adjustments.

How did we go about making this data open?

Our starting point was a spreadsheet that we use internally that contains the datasets dating back to 2008-09. We were a bit disappointed to learn that the requirements for local authorities to provide this data in this structure has now changed, so there won’t be comparable data available next year. However, this dataset served as a good test for a future approach. In the longer term it would be worth us looking at how we could make continuous data available in order to reduce the burden of reporting requirements. Lucy Knight from Devon County Council has a really useful example that we can draw on in her lunchtime lecture for the Open Data Institute on Making open data happen in local government

We used Hendrik Grothuis’s post on making data open and the Open Data Institute’s Consumers Checklist as rough guides for the process. Our first step in cleaning up the data was to look at which data was ours to share, and which data was already available from other sources. We decided to remove the data that was already made available through StatsWales to avoid duplication, but should you want to think about using this dataset with some of the ones that we used internally, these may provide a good starting point:

We then used CSV Lint to check whether the file was readable. We were pleased to discover that we had a valid file, but we also found ways that we could improve it. We turned the dataset around so that the data items go horizontally and the years go vertically. We also created a null value to indicate where the data was unavailable. A quick Google search was enough for us to discover how to note empty cells.

As a Welsh public sector organisation, we are required to make the data available bilingually, so we sent it to the translators to make sure that we got each technical term exactly right.

Publishing the data

When it came to publishing the data, we decided to publish it as part of this post on the Good Practice Exchange blog. It would be a very lonely looking dataset on an Open Data platform at the moment, but the hope is that we can identify other datasets that we can release going forward. We looked at potential platforms that could be used, including open source options like CKAN and DKAN (both of which would integrate with our Drupal Content Management System), as well as cloud based platforms like Socrata. As an organisation we’re moving to the cloud when it makes sense, but there may be things that we could learn from Audit Scotland’s Innovation Zone, which has been set up to allow their staff to test new software and platforms in a lightly regulated space. This gives staff the opportunity to test new ways of working.

As per our recent webinar on Open Standards, we’ve chosen to publish the data in CSV instead of a proprietary format like Excel. This means that it can be used by a wide variety of software, and hopefully as wide a variety of people as possible.

It’s now up to us to ensure that this data is discoverable by tagging it effectively, and we will also publicise the dataset through the networks that we’ve built through our prior work on Open Data. Our next challenge is to track how the data is used, so if you do use the dataset, we’d love to have your feedback about the format and what you used it for.

Learning from the Welsh Government

The Welsh Government were a great help throughout my work on the Cutting Edge Audit project. They shared learning from their approaches, and we also attended meetings together to learn more about Cardiff and Monmouthshire Councils’ approaches. It was fascinating to hear that the Welsh Government’s own staff use StatsWales to share and gather data as it’s open and transparent. This is something for us to think about in our own journey forward – how we can make data more accessible for both internal and external stakeholders.

We ended up using the Welsh Government’s approach to Metadata as a template for our own work. Metadata is a set of data that describes and gives information about other data, and it’s really important because it gives context around the data that is being shared. You can find the metadata at the bottom of this post alongside a link to the data itself.

Feedback

Your feedback on our approach here is really important. As this is an initial test of how we might make data open and shareable, your feedback will be used to shape how this might progress. As an organisation, we’re very keen to look at how we can make better use of data to help public services improve, and also to walk the talk in terms of our own digital practice. The Auditor General for Wales talks about enabling innovation through well managed risks before every one of our shared learning events. We’re looking to share our own learning so that people can learn from our experiences, be they good or bad. We always say that there’s no point reinventing the wheel. By working openly and transparently, we hope that organisations can build on what we’re doing so that they can share data as effectively as possible in order to improve the services that they provide.

Dataset: Local Government Financial Statements

Metadata

Title Data from Local Government Financial Statements 2008-09 to 2015-16
Last update 01/04/2017
Next update No longer updated.
Publishing organisation Wales Audit Office
Source Local Government Financial Statements 2008-09 to 2015-16
Contact email good.practice@audit.wales
Lowest level of geographical disaggregation Local authorities
Geographical coverage Local authorities
Languages covered English and Welsh
Data licensing You may use and re-use this data free of charge in any format or medium, under the terms of the Open Government License – see http://www.nationalarchives.gov.uk/doc/open-government-licence

Glossary

Enabling staff to make better use of data

Darllenwch y flogbost yn Gymraeg

How can we enable Wales Audit Office staff to make better use of data? Dyfrig Williams reflects on learning from our Cutting Edge Health Audit prototype.

A screenshot of our intranet, which shows the latest health news and data

In my work on the Cutting Edge Audit project, I’ve been looking at how we can bring together data from public bodies in a way that’s open to everyone and easy to access so that we can get further insight as auditors. The Health Team at the Wales Audit Office kindly volunteered to work with us so that we could look at what this might mean in practice.

In order to develop the parameters for the work, I developed personas for staff in various roles so that we could be better informed about what work we needed to undertake. This really helped us to identify how research time is used and who is doing that research.

The feedback that I received was that data is difficult to access, so we’ve developed a prototype to bring useful health data and information together in one place.

Testing approaches

Our initial attempt to bring this data together in one place is very much a proof of concept to see how this could be progressed further. Our initial thinking was to try and create a data one-stop shop for each health board, as well as a page for national work that covers the whole of Wales or the UK. When we thought about what a functioning prototype might look like, we decided to use a national site as our test.

We initially decided to use Sharepoint Online because it gave us the opportunity to look at how we could develop our use of the service to make data more accessible internally. Unfortunately whilst this worked as an initial test, we could only make the site available to a selected number of users, as we’re currently testing it with a small user group. We really wanted to share the results throughout the organisation so that staff could think about whether this type of approach would be useful in their work, so we decided to host the information on our intranet (The Hub), which is a Drupal site.

We used feedback from the Performance Support Officer to bring together information feeds in order to save research time. We created widgets from RSS feeds (Really Simple Syndication feeds, which deliver regularly changing web content from news sites, blogs and other online publishers) where available. We also generated our own widget (a small application that can be installed within a web page) for a Twitter feed that we generated from a Good Practice Exchange Twitter list. We embedded these feeds as IFrames within the site.

Making data more user friendly

We wanted to make data easier to understand and use. There was a strong feeling from the staff that I interviewed that knowledge and understanding of data shouldn’t be siloed, so we looked at how we could make data more accessible.

We decided to use Microsoft’s Power BI (a suite of business analytics tools to develop insight) to make health data sets more accessible and easy to understand. This meant that we didn’t have to buy any software, and that we could host the data directly through the Power BI service. We didn’t use any sensitive data for our test so it wasn’t a problem to publish it directly to the web. There was mixed feedback from staff that I interviewed as to whether the site should include private data, so we will need to look at our options again should we choose to go down this route.

The data sets that we used are publicly available and use APIs (Application Programming Interfaces, which access the features or data of an operating system, application, or other web service). This means that the Power BI data tools are linked directly to StatsWales for the data so that there’s no manual downloading of data after its set up. It also means that we’re always using the most current data that’s available.

Where do we go from here?

The use of our prototype has now been developed and extended so that it can be used as a data tool for a Primary Care project in health so that colleagues can use it to analyse data for their own Health Board, so it’s great to know that the work has already been of practical value.

Our use of widgets and APIs mean that the amount of work needed to maintain the current information that the site holds is very limited. However, if we want to develop the data that it holds, we need to think about who might be responsible for its upkeep, should it be seen as valuable.

The next step for us as an organisation is to use the personas that we generated, as well as informal feedback from this work, to look at what an effective data site and service might look like, and how that might be adapted for other parts of the organisation. That will enable us to learn from any mistakes that we’ve made so that we do things differently in the future, and also to build on our successes. And if we can build on that learning, we’ll be well placed to develop our work in order to be the cutting edge audit office that we aspire to be.

Being open by default

How might an audit office open up its systems so that information becomes open by default? Dyfrig Williams spoke with Tom Haslam about the approach of New Zealand’s Office of the Auditor-General.

The logo of the Office of the Auditor-General New Zealand

As part of the Wales Audit Office’s Cutting Edge Audit project, I am working on an Open Data prototype. During this work, colleagues told me that we could improve our approach to data. Not acquiring new data though – most colleagues said their biggest issue was better knowledge of, and access to, data that the office already held.

Our organisation has two specialist practices – financial audit and performance audit. This division facilitates specialism, so that we have colleagues with incredibly good knowledge in their fields of expertise. However, it also means that we have to work hard to break down organisational silos, sometimes reinforced by the systems we have in place.

Safeguarding data is an important feature of the way we have set up our information systems. Network folders are protected. Access is only available to specific teams and personnel, which means that the data within them is closed to others by default. Our SharePoint system is also set up in a similar way and the search functionality is not as good as it might be. All of this means that unless you know where the data is held, you’re unlikely to find it.

Learning from other audit offices

In my last post on the Queensland Audit Office’s work, I mentioned a well-travelled colleague called Tom Haslam. Tom has worked at the Office of the Auditor-General (OAG) in Wellington, New Zealand. And while there, the OAG identified similar problems with how they organised and held their data.

To address this, the OAG implemented a new SharePoint-based information system and complemented this with some pilot cross-office groups known as ‘iShare’. These groups were based around cross-cutting functional topics (for example the Transport iShare) with the aim of helping to break down organisational silos and promote a one-team approach across the office.

Adopting a new information system gave the OAG an opportunity to debate the relative merits of information systems being open or closed by default. This was discussed across the office through various channels.

The previous information systems had encouraged a mainly ‘closed until open’ approach. But the general feeling was that closed data might prevent the office from making the most of the information that they held. The natural tendency of all auditors is to be cautious, so under a ‘closed unless open’ approach, setting information as ‘open’ might be viewed as a risk best avoided, even if this approach wasn’t justified. On a practical level, having information closed off requires various permissions and access rights to be set up. This alone can be a barrier to sharing data.

The OAG structured its new information system so that information was ‘open unless closed’ with metadata to help staff find what they wanted. This approach facilitated sharing, encouraging staff to think about how they could add value by joining up information. A default setting of ‘open until closed’ made staff think more carefully about why they should want to close off access, for example material with national security implications or identifiable personal information.

On a technical level, a cleaner configuration of the IT system without endless permissions and restrictions made the system run more reliably. The improved reliability of the new SharePoint system led to time savings, and increased staff confidence and satisfaction with IT. The iShare pilots encouraged group members to look actively for opportunities to work jointly and share information.

As these pilots progressed and reported their successes to the wider office, they encouraged a more open outlook across teams – ‘look we shared stuff and worked together and it hasn’t all turned to custard’ as our kiwi cousins might say.

Tom also thought there was a trust dimension. Handling sensitive client information is part of an auditor’s day job. Therefore, opening up data was a clear signal that the OAG was a high trust environment.

However, change is a journey and the OAG report that its experience is no different. It continues to encourage and aim for an environment where information is open until closed. But it hasn’t always been plain sailing since introducing the new information system. Some staff have embraced the opportunity to openly share information. Others have been more hesitant in sharing information more or are yet to change what they have always done to be more open. The OAG has had to periodically promote and reinforce the new approach. It recognises that a change of this magnitude won’t happen overnight or without a sustained effort. But the end – using collective knowledge to influence improvement and improve accountability – justifies the effort.

How this fits with the work of the Good Practice Exchange

Our Good Practice Exchange work on effective data sharing shows that this relies on the principle of adopting proportionate steps when safeguarding data.

In a previous blog post on whether data sharing was a barrier to public service improvement, I included a quote from the Information Commissioner, which said ‘People want their personal data to work for them. They expect organisations to share their personal data where it’s necessary to provide them with the services they want. They expect society to use its information resources to stop crime and fraud and to keep citizens safe and secure.’ It’s also well worth watching Anne Jones, the Assistant Information Commissioner for Wales, outlining how data can be shared effectively.

The upcoming General Data Protection Regulation will ramp up the safeguarding of data a few notches, but it’s also an opportunity to reconsider how we can share data effectively. Particularly, how we make sure that auditors are confident enough to make the most of data collection and sharing.

Previously I have blogged about our staff trust event, where we heard that trust is essential if public services are to take well-managed risks, innovate and deliver public services that are truly fit for the 21st century.

Tom is leading on a separate project within the Wales Audit Office to look at how we’re using our information systems including SharePoint. One option we’re considering is the use of SharePoint Online, which would make it easier for us to develop an area that could be accessed by external bodies and partners – a portal. Leigh Dodds ‘s post provides a good overview of what a portal might contain.

A portal would allow us to share data with audited bodies and partners more effectively. We’re testing this concept with a SharePoint based prototype portal for some of our health colleagues. Learning from this will feed back into Tom’s project. And if working on the Cutting Edge Audit project has taught me anything, it’s that joined up and collaborative approaches are the best way to ensure we add real value to the work that we’re doing.

How Queensland Audit Office uses data analysis to improve its auditing

What can we learn from the way that the Queensland Audit Office uses data analytics in their audits? Dyfrig Williams had an early morning phone conversation to find out.

As I previously mentioned on my post on the Wales’ Audit Office’s Cutting Edge Audit project, I’m looking to identify ways of making better use of data and technology in order to transform the way that we work. I’d come across this article on how Queensland Audit Office are using audit data analytics to gain greater insight, and through my well-travelled colleague Tom Haslam I managed to organise an early morning conversation between myself and my colleagues (Steve Lisle and Nigel Blewitt) and Daniele Bird, the Assistant Auditor-General for Performance Audit in Queensland, Ben Jiang the Assistant Director for Audit Analytics and David Toma, their Assistant Director for Performance Audit Services.

What is the Queensland Audit Office doing?

Back in 2014, they decided to look at opportunities to use more sophisticated data visualisation tools in order to get a better insight into their audited bodies. They trialled and tested Qlikview, which can be accessed via the web or desktop software. They set up a data analytics team in 2015, all of whom were former auditors branching out into the world of data analytics. The team undertook initial training, but their development from there was centred on self-learning. The team first planned to work on Performance Audit. However the focus moved to Financial Audit, as the recurring datasets meant they could work more efficiently and effectively, compared to the very different types of datasets that are needed for each performance study.

The importance of relationships

Our seminars on early closure of accounts in 2015 and 2016 have made it clear to me that both the auditor and the audited body have to work together closely in order for audit to work more effectively. Whilst the Queensland Audit Office had built processes and automatic systems to collect data, they still faced questions from clients about security and what they were going to do with the data. They built an internal Frequently Asked Questions page so that audit team leaders could re-assure organisations and build trust, and they also took the burden of change away from the audited bodies by enabling them to dump the unformatted data with the office. The analytics team did the work to transform and clean up the data, and they’re now able to offer unique insights as they gather data from such a wide range of bodies. They can quickly identify and access an overview of issues, as well as benchmark clients’ performance and show how they compare to their peers (which you can see in the slide below). They also use the tools to undertake exercises like Benford’s Analysis, which clearly identifies outliers. This is particularly useful in identifying fraud.

A graph showing lots of dots togther and a clear outlier on the value of bad debt

How Queensland Audit Office gains insight to the value of bad debt

Performance Audit

The Data Analytics Team is now also supporting Performance Audit staff. The team is using text mining, for example to see what teachers are identifying as development goals. This has enabled them to produce Wordclouds and text summaries of the top 10 statements. They also use these tools to sweep the text of Hansards.

The team is also looking at how it can help staff to choose performance audit topics. It’s running sentiment analysis tools on Twitter feeds in order to support the strategic planning process. This helps them to understand the issues that are out there in the wider world and what people are saying about them. Although the technology has been built for Twitter, it could also be used for networks like LinkedIn, which would enable auditors to access different users and perspectives. They’ve also built social media analysis tools that rank tweets. It grades them and enables auditors to focus resources on important messages.

Creating a positive working environment

Daniele, Ben and David acknowledged the importance of leadership from the top in creating the right environment for the work of the Data Analysis Team to flourish. They’ve received high level support from the previous Auditor General and the current Acting Auditor General. This is something that we’ve seen from our own Auditor General in the development of the Cutting Edge Audit project, and hopefully the findings from our own work can help to provide a base for us to develop our own ways of working. It was really interesting to hear how the organisation is continuously looking to develop their work to build on current practices. They’re currently looking at whether moving from Qlikview to Qlik Sense could provide further advantages.

The work of the Data Analytics Team has achieved efficiencies, mainly around saving time that can be reinvested in future work, but they’re also looking to save more to do more. The rapidly changing environment and limited public finances presents a challenge for every public service. Queensland Audit Office have faced that challenge head on and are working in a different way to provide a better service. Not only that, but their staff’s work is now more meaningful and interesting, and auditors are further developing their critical thinking skills. I’ll be thinking about our Good Practice Exchange principles as we reflect on our conversation with the Queensland Audit Office, so that we think about how we might adapt this approach in order to ensure that our work, like the services we audit, are delivering the best possible outcomes for the people of Wales.

Developing personas for a user focused service

How can personas help to ensure that projects are user focused? Dyfrig Williams reflects on how they’ve been used for his work on acquiring data effectively.

An image of personas for Performance Audit staff

As part of my Cutting Edge Audit work, I’ve been working on a test project to gather data for auditors so that we can work more efficiently.

I’ve blogged before on why I’m really interested in the digital delivery of public services because of the relentless focus on user need. This focus isn’t new to me – my previous work on citizen engagement made it clear to me that properly involving the people who access services at an early stage results in a service that is effective at targeting its resources.

Now that I’m four months in to a six month project you could argue that I’m leaving it a bit late to start on the practical part of the work. But in my public engagement role I heard time and time again that staff that didn’t involve people properly delivered their work according to existing preconceptions about what people wanted or needed. I was determined not to make the same mistake with my work, so I developed personas for the main users of data to ensure that we are focusing on building something that people would actually find useful.

Personas are representations of different types of customers or users. They answer the question “Who are we designing our work for?” They help to align strategy and goals to specific user groups. As we’re testing this work with the Health Performance Audit Team, I developed personas ranging from Performance Support Officers to Audit Managers, as well as complimentary work with members of the Financial Audit Technical Team and a Financial Auditor working on the frontline with health boards.

Where to start?

I was quite excited to start on the personas so that I could see how my public engagement mindset fitted with digital practice. My colleague Louise Foster-Key, who is the Digital Comms Officer at the Wales Audit Office gave me a few pointers from her work on the Wales Audit Office Intranet, and I also looked at wider good practice. There’s a great post on user personas on the Office for National Statistics’ Digital Blog, but I ended up basing my work on the Government Digital Service (GDS)’s Guide to User Stories. I had a punt at using Trello, but as this was a largely solo exercise (and it was only a small project to manage) I didn’t find it to be as useful as I expected. Instead I used Xtensio, a free(ish) tool that Louise recommended to me.

I stuck with the GDS format as the structure of my user stories, which I put together from interviews and emails with staff in key roles:

  1. As a… (Who is the user? A biography of the person accessing data)
  2. I need/want/expect to… (What does the user want to do?)
  3. So that… (Why does the user want to do this?)
  4. It’s done when… (This is the GDS acceptance criteria, which is a list of outcomes that you use as a checklist to confirm that your service has done its job and is meeting that user need)

Has it been useful?

By talking to auditors in depth about each of the roles that I’ve mapped, I’ve got a much better understanding of their work. This will be useful to me far beyond the lifespan of the Cutting Edge Audit project, as I look at how the Good Practice Exchange can work more effectively with audit teams, especially the Health Audit Team, who volunteered to work with us on this piece of work.

The main use of the personas will be to sense check our work and to ensure that the purpose of our test matches what our auditors want and expect. Now that they’ve helped us run through some hypothetical tests, we’ll use their thoughts to avoid scope creep for the first stage of our iteration.

Their thoughts will also be incredibly useful for my section of our final report. It’s been said that “The pen is mightier than the sword”, and in this sense we as the report authors have all the power – it is us who provide the recommendations for future priorities. This work will ensure that my recommendations are grounded in the realities of our staff’s day to day work.

The final personas are now online and have been split into two sections because of the limitations of the free version of Xtensio. The first set of personas are based on the Performance Audit roles, and the second set include personas based on the Financial Audit and Technical Team roles.

These personas now give us a level of expectation for our work. If the test fails, they give us a rigorous criteria to examine our output against, and a clear vision to check our delivery against. On the other side of things, should the work (as we hope) be a success, we have an opportunity to think about whether personas can help us to be more focused and responsive as we ensure that public services really are delivering value for money for the people of Wales.