Tag Archives: data

Innovative audit: Learning and sharing with the Netherlands Court of Audit

Flags from the Netherlands' different provinces line the waterside in the Hague

Darllenwch y flogbost yn Gymraeg

What can the Wales Audit Office learn from the Netherlands Court of Audit’s approach to innovation? Dyfrig Williams visited The Hague to see how they work and to share the work of the Good Practice Exchange.

In May I had the opportunity to visit the Netherlands Court of Audit to learn more about their Doen project (which means ‘Doing’) and their approach to innovation. Their whole approach stems from learning by doing, which is a fantastic mantra to have. Throughout the visit, staff openly shared their successes and failures from very practical change projects. Their safe to fail space mindset is key for providing the space for that learning and improvement.

The month before I had shared the work of the Good Practice Exchange with Mark Smolenaars and Sanne Kouwenhoven from the Netherlands Court of Audit in Cardiff. Myself, Steve Lisle (who is leading on the Cutting Edge Audit project) and Mike Usher (our Sector Lead for Health and Central Government, and who also leads our Investigative Studies work) were fortunate enough to be invited to the Netherlands to share our work with their colleagues, and also to learn more about their approach to innovation, so that we can adapt their approaches to suit our needs.

Looking at audit differently

At the Good Practice Exchange we always say that there is no “one-size fits all” approach. We may have many things in common with the Netherlands Court of Audit (we have about the same number of staff and therefore roughly the same challenges in identifying capacity and scaling up change initiatives), but there are also some differences too (the Dutch public sector includes casinos – I can’t even imagine what that audit looks like!).

When we started looking at their approach to innovation, one of the key things that struck me was that they have a variety of staff that contribute very different things to the organisation. Not only does this avoid a groupthink mentality, but it also brings a lot of different skillsets to their work.

We had a presentation from Linda Meijer, who is an auditor who has become a designer. It was fascinating to hear how her design skills led to her asking different questions of data as she illustrated their findings. This was particularly useful on an audit of products that have the CE mark, which shows that products are safe and can be sold in the European market.

We also discussed how the Netherlands Court of Audit have an Investigative Journalist working for them, which was particularly useful for us at the Wales Audit Office as we have our own Investigative Studies team. Jaco Alberts’ insight was fascinating as he talked about how he applies his expertise to his role.

We also heard about how they shared their expertise through a participatory audit of higher education institutions. They asked members of each student council to check pre-investments and to look at how the institution’s budget framework is working, which meant that they had the opportunity to make the most of the knowledge that is available within institutions. This fascinating approach gave us food for thought as we thought aloud about working to the Wellbeing of Future Generations Act, as one of the five ways of working that’s outlined is involvement.


One of the areas that really fascinated me was the Netherlands Court of Audit’s use of data, as I have been leading on the Wales Audit Office’s work on acquiring data as part of our Cutting Edge Audit work. Their involvement of people with different backgrounds even filters into their data work, as we heard how they make the most of data interns.

As with any practice sharing, it was fascinating to hear how the Netherlands Court of Audit are dealing with challenges as well. Roline Kamphuis shared how the organisation have set up communities of practice around R, which they hope will help the organisation to free up the capacity of its data staff. We heard how they currently spend much of their time cleaning data, which means that they can make shared data available for wider use. By cleaning the data effectively at the start of the audit, they make work much easier for staff as it progresses.

They also spoke about why they particularly use R and SPSS, as they have a script that can be used for an audit trail. By developing a pre-programmed script in R, they can ensure that outputs fit house style. Also because R generates word and excel documents, it means that auditors can see easily see and understand the steps that have been taken when working on that data.

A photo of Steve Lisle presenting the Cutting Edge Audit work to the NCA

Steve Lisle presenting the Cutting Edge Audit work to the Netherlands Court of Audit

Rudi Turksema shared how the Netherlands Court of Audit had run an accountability hack, which meant that they were able to involve external stakeholders in their data work. By working in partnership with a range of organisations, they were able to open up access to a wide variety of datasets on the day. I mentioned that the Good Practice Exchange have supported the NHS Hack Day in Cardiff, but that I wasn’t sure that we had enough capacity as an organisation to run such an event yet. Yet they encouraged us to look at putting a similar event on as it had helped them to share their work and involve a wide range of people. We’ve already seen what happens when you make data open and accessible, as Ben Proctor and the Open Data Institute have already started using Google Fusion Tables to create maps and add value to our first open dataset. This certainly gave me food for thought for how such an event might help us to further develop our use of data and our own data maturity.

I learnt so much from my trip to the Netherlands. It was fascinating to see the parallel journeys that both of our organisations are taking to ensure that audit is fit for purpose in the twenty first century. Thanks to everyone at the Netherlands Court of Audit for your welcome and your hospitality – you’ve certainly helped inform our learning journey and your expertise has been invaluable in helping us to move forward with our own innovation work.

Enabling staff to make better use of data

Darllenwch y flogbost yn Gymraeg

How can we enable Wales Audit Office staff to make better use of data? Dyfrig Williams reflects on learning from our Cutting Edge Health Audit prototype.

A screenshot of our intranet, which shows the latest health news and data

In my work on the Cutting Edge Audit project, I’ve been looking at how we can bring together data from public bodies in a way that’s open to everyone and easy to access so that we can get further insight as auditors. The Health Team at the Wales Audit Office kindly volunteered to work with us so that we could look at what this might mean in practice.

In order to develop the parameters for the work, I developed personas for staff in various roles so that we could be better informed about what work we needed to undertake. This really helped us to identify how research time is used and who is doing that research.

The feedback that I received was that data is difficult to access, so we’ve developed a prototype to bring useful health data and information together in one place.

Testing approaches

Our initial attempt to bring this data together in one place is very much a proof of concept to see how this could be progressed further. Our initial thinking was to try and create a data one-stop shop for each health board, as well as a page for national work that covers the whole of Wales or the UK. When we thought about what a functioning prototype might look like, we decided to use a national site as our test.

We initially decided to use Sharepoint Online because it gave us the opportunity to look at how we could develop our use of the service to make data more accessible internally. Unfortunately whilst this worked as an initial test, we could only make the site available to a selected number of users, as we’re currently testing it with a small user group. We really wanted to share the results throughout the organisation so that staff could think about whether this type of approach would be useful in their work, so we decided to host the information on our intranet (The Hub), which is a Drupal site.

We used feedback from the Performance Support Officer to bring together information feeds in order to save research time. We created widgets from RSS feeds (Really Simple Syndication feeds, which deliver regularly changing web content from news sites, blogs and other online publishers) where available. We also generated our own widget (a small application that can be installed within a web page) for a Twitter feed that we generated from a Good Practice Exchange Twitter list. We embedded these feeds as IFrames within the site.

Making data more user friendly

We wanted to make data easier to understand and use. There was a strong feeling from the staff that I interviewed that knowledge and understanding of data shouldn’t be siloed, so we looked at how we could make data more accessible.

We decided to use Microsoft’s Power BI (a suite of business analytics tools to develop insight) to make health data sets more accessible and easy to understand. This meant that we didn’t have to buy any software, and that we could host the data directly through the Power BI service. We didn’t use any sensitive data for our test so it wasn’t a problem to publish it directly to the web. There was mixed feedback from staff that I interviewed as to whether the site should include private data, so we will need to look at our options again should we choose to go down this route.

The data sets that we used are publicly available and use APIs (Application Programming Interfaces, which access the features or data of an operating system, application, or other web service). This means that the Power BI data tools are linked directly to StatsWales for the data so that there’s no manual downloading of data after its set up. It also means that we’re always using the most current data that’s available.

Where do we go from here?

The use of our prototype has now been developed and extended so that it can be used as a data tool for a Primary Care project in health so that colleagues can use it to analyse data for their own Health Board, so it’s great to know that the work has already been of practical value.

Our use of widgets and APIs mean that the amount of work needed to maintain the current information that the site holds is very limited. However, if we want to develop the data that it holds, we need to think about who might be responsible for its upkeep, should it be seen as valuable.

The next step for us as an organisation is to use the personas that we generated, as well as informal feedback from this work, to look at what an effective data site and service might look like, and how that might be adapted for other parts of the organisation. That will enable us to learn from any mistakes that we’ve made so that we do things differently in the future, and also to build on our successes. And if we can build on that learning, we’ll be well placed to develop our work in order to be the cutting edge audit office that we aspire to be.

How Queensland Audit Office uses data analysis to improve its auditing

What can we learn from the way that the Queensland Audit Office uses data analytics in their audits? Dyfrig Williams had an early morning phone conversation to find out.

As I previously mentioned on my post on the Wales’ Audit Office’s Cutting Edge Audit project, I’m looking to identify ways of making better use of data and technology in order to transform the way that we work. I’d come across this article on how Queensland Audit Office are using audit data analytics to gain greater insight, and through my well-travelled colleague Tom Haslam I managed to organise an early morning conversation between myself and my colleagues (Steve Lisle and Nigel Blewitt) and Daniele Bird, the Assistant Auditor-General for Performance Audit in Queensland, Ben Jiang the Assistant Director for Audit Analytics and David Toma, their Assistant Director for Performance Audit Services.

What is the Queensland Audit Office doing?

Back in 2014, they decided to look at opportunities to use more sophisticated data visualisation tools in order to get a better insight into their audited bodies. They trialled and tested Qlikview, which can be accessed via the web or desktop software. They set up a data analytics team in 2015, all of whom were former auditors branching out into the world of data analytics. The team undertook initial training, but their development from there was centred on self-learning. The team first planned to work on Performance Audit. However the focus moved to Financial Audit, as the recurring datasets meant they could work more efficiently and effectively, compared to the very different types of datasets that are needed for each performance study.

The importance of relationships

Our seminars on early closure of accounts in 2015 and 2016 have made it clear to me that both the auditor and the audited body have to work together closely in order for audit to work more effectively. Whilst the Queensland Audit Office had built processes and automatic systems to collect data, they still faced questions from clients about security and what they were going to do with the data. They built an internal Frequently Asked Questions page so that audit team leaders could re-assure organisations and build trust, and they also took the burden of change away from the audited bodies by enabling them to dump the unformatted data with the office. The analytics team did the work to transform and clean up the data, and they’re now able to offer unique insights as they gather data from such a wide range of bodies. They can quickly identify and access an overview of issues, as well as benchmark clients’ performance and show how they compare to their peers (which you can see in the slide below). They also use the tools to undertake exercises like Benford’s Analysis, which clearly identifies outliers. This is particularly useful in identifying fraud.

A graph showing lots of dots togther and a clear outlier on the value of bad debt

How Queensland Audit Office gains insight to the value of bad debt

Performance Audit

The Data Analytics Team is now also supporting Performance Audit staff. The team is using text mining, for example to see what teachers are identifying as development goals. This has enabled them to produce Wordclouds and text summaries of the top 10 statements. They also use these tools to sweep the text of Hansards.

The team is also looking at how it can help staff to choose performance audit topics. It’s running sentiment analysis tools on Twitter feeds in order to support the strategic planning process. This helps them to understand the issues that are out there in the wider world and what people are saying about them. Although the technology has been built for Twitter, it could also be used for networks like LinkedIn, which would enable auditors to access different users and perspectives. They’ve also built social media analysis tools that rank tweets. It grades them and enables auditors to focus resources on important messages.

Creating a positive working environment

Daniele, Ben and David acknowledged the importance of leadership from the top in creating the right environment for the work of the Data Analysis Team to flourish. They’ve received high level support from the previous Auditor General and the current Acting Auditor General. This is something that we’ve seen from our own Auditor General in the development of the Cutting Edge Audit project, and hopefully the findings from our own work can help to provide a base for us to develop our own ways of working. It was really interesting to hear how the organisation is continuously looking to develop their work to build on current practices. They’re currently looking at whether moving from Qlikview to Qlik Sense could provide further advantages.

The work of the Data Analytics Team has achieved efficiencies, mainly around saving time that can be reinvested in future work, but they’re also looking to save more to do more. The rapidly changing environment and limited public finances presents a challenge for every public service. Queensland Audit Office have faced that challenge head on and are working in a different way to provide a better service. Not only that, but their staff’s work is now more meaningful and interesting, and auditors are further developing their critical thinking skills. I’ll be thinking about our Good Practice Exchange principles as we reflect on our conversation with the Queensland Audit Office, so that we think about how we might adapt this approach in order to ensure that our work, like the services we audit, are delivering the best possible outcomes for the people of Wales.

Sut mae Swyddfa Archwilio Queensland yn dadansoddi data i archwilio’n fwy effeithiol

Beth allwn ni ddysgu o’r ffordd mae Swyddfa Archwilio Queensland yn dadansoddi ei data ar gyfer eu harchwiliadau? Cafodd Dyfrig Williams sgwrs ffôn yn hynod o gynnar yn y bore er mwyn darganfod mwy.

Fel rydw i wedi dweud o’r blaen yn fy mlogbost ar brosiect Archwilio Sydd ar Flaen y Gad, rwy’n edrych i nodi ffyrdd o wneud defnydd gwell o ddata a thechnoleg i drawsnewid y ffordd rydyn ni’n gweithio. Fe wnes i ddod ar draws yr erthygl hon ar sut mae Swyddfa Archwilio Queensland yn dadansoddi data i gael mewnwelediad gwell iddo. Trwy fy nghydweithiwr Tom Haslam trefnais sgwrs yn gynnar iawn yn y bore rhyngof i a fy nghydweithwyr (Steve Lisle a Nigel Blewitt) a Daniele Bird, yr Archwilydd Cyffredinol Cynorthwyol ar gyfer Archwilio Perfformiad yn Queensland, Ben Jiang, y Cyfarwyddwr Cynorthwyol dros Ddadansoddi Data Archwilio a David Toma, y Cyfarwyddwr Cynorthwyol ar gyfer Gwasanaethau Archwilio Perfformiad.

Beth mae Swyddfa Archwilio Queensland yn gwneud?

Edrychodd y swyddfa ar gyfleoedd i ddefnyddio offer delweddu data mewn ffordd fwy soffistigedig yn ôl yn 2014 er mwyn cael dealltwriaeth well o’r cyrff maen nhw’n archwilio. Fe wnaethon nhw dreialu a phrofi Qlikview, sef meddalwedd sy’n gallu cael ei ddefnyddio trwy’r we neu ar ben desg. Fe wnaethon nhw sefydlu tîm dadansoddi data yn 2015, ble roedd pob un ohonynt yn gyn-archwilwyr. Fe wnaeth y tîm derbyn hyfforddiant i ddechrau, ond ers hynny mae eu datblygiad wedi dod o hunan-ddysgu. Y cynllun ar y dechrau oedd gweithio ar Archwilio Perfformiad. Fodd bynnag, symudodd y ffocws i Archwilio Ariannol gan fod yr un setiau o ddata yn cael ei ddefnyddio tro ar ôl tro, felly roedd yn haws i alluogi gwaith effeithlon ac effeithiol o’i gymharu â’r mathau gwahanol iawn o ddata sydd eu hangen ar gyfer astudiaethau Archwilio Perfformiad.

Y pwysigrwydd o berthnasau

Fe wnaeth ein seminarau yn 2015 a 2016 ar gau cyfrifon yn gynt dangos i mi fod rhaid i’r archwilydd a’r corff sy’n cael ei archwilio i gydweithio’n agos i weithio’n effeithiol. Adeiladodd Swyddfa Archwilio Queensland prosesau a systemau awtomatig i gasglu data, ond roeddwn nhw’n dal i gael cwestiynau o gleientiaid am y diogelwch o beth roedden nhw’n gwneud a beth roedden nhw’n mynd i wneud  gyda’r data. Rhoddir tudalen Cwestiynau a Ofynnir yn Aml at ei gilydd yn fewnol fel y gallai arweinwyr y timau archwilio tawelu meddyliau’r sefydliadau ac adeiladu ymddiriedaeth. Fe wnaethon nhw hefyd cymryd y baich o newid i ffwrdd o’r cyrff drwy eu galluogi i anfon y data heb ei fformatio. Gwnaeth y Tîm Dadansoddi Data y gwaith o drawsnewid a glanhau’r data, ac maen nhw’n awr yn gallu cynnig mewnwelediad unigryw i’r data gan eu bod nhw’n casglu data o gyrff amrywiol. Maen nhw’n gallu ffurfio trosolwg o faterion yn gyflym, ac maen nhw hefyd yn gallu meincnodi perfformiad cleientiaid er mwyn dangos sut maen nhw’n cymharu â’u cyfoedion (cewch gip ar y sleid isod). Maen nhw hefyd yn gallu defnyddio’r meddalwedd i gynnal ymchwiliadau fel Dadansoddiad Benford, sy’n nodi’n glir pwy sydd yn bell o’r norm. Mae hyn yn arbennig o ddefnyddiol wrth nodi twyll.

Graff sy'n dangos llawer o ddotiau at ei gilydd ac un sydd glir ymhell i ffwrdd ar werth dyledion drwg

Sut mae Swyddfa Archwilio Queensland yn cael mewnwelediad i werth dyledion drwg

Archwilio Perfformiad

Mae’r Tîm Dadansoddi Data hefyd yn cefnogi staff Archwilio Perfformiad. Mae’r tîm yn defnyddio cloddio testun i archwilio pynciau amrywiol, er enghraifft i weld beth mae athrawon yn nodi fel eu nodau datblygu. Mae hyn wedi galluogi iddynt gynhyrchu Wordclouds a chrynodebau testun o’r 10 datganiad uchaf. Maen nhw hefyd yn defnyddio’r dechneg yma i ysgubo trwy Hansards.

Mae’r tîm hefyd yn edrych ar sut y gall hyn helpu staff i ddewis pynciau i archwilio iddynt. Maen nhw’n dadansoddi teimlad ar Twitter, sy’n bwydo i mewn i’r cynllunio strategol. Mae hyn yn eu helpu nhw i ddeall y materion sydd yn bwysig i’r byd ehangach a beth mae pobl yn dweud amdano’n nhw. Er bod y dechnoleg wedi cael ei hadeiladu ar gyfer Twitter, gallai hefyd gael ei ddefnyddio ar gyfer rhwydweithiau fel LinkedIn, a fyddai’n galluogi archwilwyr i gael gafael ar wahanol ddefnyddwyr a safbwyntiau. Maen nhw hefyd wedi adeiladu offer sy’n graddio pwysigrwydd y trydar. Mae hyn yn galluogi archwilwyr i ganolbwyntio adnoddau ar y negeseuon pwysig.

Creu amgylchedd gwaith cadarnhaol

Fe wnaeth Daniele, Ben a David gydnabyddi’r pwysigrwydd o arweinyddiaeth gref i greu amgylchedd ble mae gwaith y Tîm Dadansoddi Data yn ffynnu. Maen nhw wedi derbyn cefnogaeth o’r lefel uchaf o’r Archwilydd Cyffredinol blaenorol a’r Archwilydd Cyffredinol Dros Dro cyfredol. Mae hyn yn rhywbeth ni’n gweld o’n Harchwilydd Cyffredinol ein hunain yn natblygiad y prosiect Archwilio Sydd ar Flaen y Gad, a gobeithio y gall canfyddiadau ein gwaith ni ein helpu ni i ddarparu sylfaen i ddatblygu ein ffordd o weithio. Roedd e’n ddiddorol i glywed sut mae’r sefydliad yn edrych yn barhaus i ddatblygu eu gwaith ac adeiladu ar arferion presennol. Ar hyn o bryd maen nhw’n edrych i weld os all symud o Qlikview i Qlik Sense rhoi manteision pellach iddynt.

Mae gwaith y Tîm Dadansoddi Data wedi cyflawni arbedion effeithlonrwydd, yn bennaf trwy arbed amser y gellir ei ail-fuddsoddi mewn ei gwaith, ond maen nhw hefyd yn edrych i arbed mwy i wneud mwy. Mae’r amgylchedd ehangach yn newid yn gyflym ac mae cyllid cyhoeddus cyfyngedig yn rhoi her i bob gwasanaeth cyhoeddus. Mae Swyddfa Archwilio Queensland wedi wynebu’r her yma ac maen nhw’n gweithio mewn ffordd wahanol i ddarparu gwasanaeth gwell. Mae gwaith ei staff yn awr yn fwy ystyrlon a diddorol, ac mae archwilwyr yn datblygu sut maen nhw’n meddwl yn feirniadol. Fe fyddai’n defnyddio ein hegwyddorion Cyfnewidfa Arfer Da wrth i ni fyfyrio dros ein sgwrs â Swyddfa Archwilio Queensland, fel ein bod ni’n meddwl am sut rydyn ni’n gallu addasu’r dull gweithredu hwn fel ein bod ein gwaith ni, fel y gwasanaethau rydyn ni’n archwilio, yn cyflawni’r canlyniadau gorau posibl i bobl Cymru.

Developing personas for a user focused service

How can personas help to ensure that projects are user focused? Dyfrig Williams reflects on how they’ve been used for his work on acquiring data effectively.

An image of personas for Performance Audit staff

As part of my Cutting Edge Audit work, I’ve been working on a test project to gather data for auditors so that we can work more efficiently.

I’ve blogged before on why I’m really interested in the digital delivery of public services because of the relentless focus on user need. This focus isn’t new to me – my previous work on citizen engagement made it clear to me that properly involving the people who access services at an early stage results in a service that is effective at targeting its resources.

Now that I’m four months in to a six month project you could argue that I’m leaving it a bit late to start on the practical part of the work. But in my public engagement role I heard time and time again that staff that didn’t involve people properly delivered their work according to existing preconceptions about what people wanted or needed. I was determined not to make the same mistake with my work, so I developed personas for the main users of data to ensure that we are focusing on building something that people would actually find useful.

Personas are representations of different types of customers or users. They answer the question “Who are we designing our work for?” They help to align strategy and goals to specific user groups. As we’re testing this work with the Health Performance Audit Team, I developed personas ranging from Performance Support Officers to Audit Managers, as well as complimentary work with members of the Financial Audit Technical Team and a Financial Auditor working on the frontline with health boards.

Where to start?

I was quite excited to start on the personas so that I could see how my public engagement mindset fitted with digital practice. My colleague Louise Foster-Key, who is the Digital Comms Officer at the Wales Audit Office gave me a few pointers from her work on the Wales Audit Office Intranet, and I also looked at wider good practice. There’s a great post on user personas on the Office for National Statistics’ Digital Blog, but I ended up basing my work on the Government Digital Service (GDS)’s Guide to User Stories. I had a punt at using Trello, but as this was a largely solo exercise (and it was only a small project to manage) I didn’t find it to be as useful as I expected. Instead I used Xtensio, a free(ish) tool that Louise recommended to me.

I stuck with the GDS format as the structure of my user stories, which I put together from interviews and emails with staff in key roles:

  1. As a… (Who is the user? A biography of the person accessing data)
  2. I need/want/expect to… (What does the user want to do?)
  3. So that… (Why does the user want to do this?)
  4. It’s done when… (This is the GDS acceptance criteria, which is a list of outcomes that you use as a checklist to confirm that your service has done its job and is meeting that user need)

Has it been useful?

By talking to auditors in depth about each of the roles that I’ve mapped, I’ve got a much better understanding of their work. This will be useful to me far beyond the lifespan of the Cutting Edge Audit project, as I look at how the Good Practice Exchange can work more effectively with audit teams, especially the Health Audit Team, who volunteered to work with us on this piece of work.

The main use of the personas will be to sense check our work and to ensure that the purpose of our test matches what our auditors want and expect. Now that they’ve helped us run through some hypothetical tests, we’ll use their thoughts to avoid scope creep for the first stage of our iteration.

Their thoughts will also be incredibly useful for my section of our final report. It’s been said that “The pen is mightier than the sword”, and in this sense we as the report authors have all the power – it is us who provide the recommendations for future priorities. This work will ensure that my recommendations are grounded in the realities of our staff’s day to day work.

The final personas are now online and have been split into two sections because of the limitations of the free version of Xtensio. The first set of personas are based on the Performance Audit roles, and the second set include personas based on the Financial Audit and Technical Team roles.

These personas now give us a level of expectation for our work. If the test fails, they give us a rigorous criteria to examine our output against, and a clear vision to check our delivery against. On the other side of things, should the work (as we hope) be a success, we have an opportunity to think about whether personas can help us to be more focused and responsive as we ensure that public services really are delivering value for money for the people of Wales.

Datblygu personas ar gyfer gwasanaeth sy’n canolbwyntio ar y defnyddiwr

Sut all personas helpu i sicrhau bod prosiectau yn canolbwyntio ar y defnyddiwr? Isod mae Dyfrig Williams yn adlewyrchu ar sut maen nhw wedi bod yn rhan o’i waith ar fynd i’r afael â data yn effeithiol.

Llun o bersonas staff Archwilio Perfformiad

Rydw i wedi bod yn gweithio ar arbrawf i gasglu data ar gyfer archwilwyr fel y gallwn weithio’n fwy effeithlon fel rhan o fy ngwaith ar brosiect Archwilio Sydd ar Flaen y Gad.

Rydw i wedi blogio o’r blaen ar fy niddordeb mewn darparu gwasanaethau cyhoeddus digidol achos y ffocws di-baid ar angen y defnyddiwr. Dyw hwn ddim yn safbwynt sy’n newydd i mi – roedd e’n amlwg yn fy nghwaith yn fy swydd flaenorol ar gynnwys dinasyddion i sicrhau bod adnoddau’n cael eu targedu’n effeithiol.

Nawr rwy’n dechrau ar ran ymarferol fy gwaith ar ôl 4 mis o brosiect 6 mis. Ond yn fy hen rôl fe wnes i glywed tro ar ôl tro bod gwasanaethau’n cael eu darparu at hen ragdybiaethau pan nad yw staff yn cael yr amser i ymgysylltu â phobl mewn modd priodol. Doeddwn i ddim eisiau gwneud yr un camgymeriad yn fy ngwaith, felly datblygais personas ar gyfer y prif ddefnyddwyr o ddata mewn timau archwilio er mwyn sicrhau ein bod ni’n canolbwyntio ar adeiladu rhywbeth y byddai pobl yn wir ddefnyddio.

Mae personas yn gynrychioliadau o wahanol fathau o gwsmeriaid neu ddefnyddwyr. Maen nhw’n helpu ni i ateb y cwestiwn o “Pwy ydyn ni’n cynllunio ein gwaith ar gyfer?” Maen nhw’n helpu i alinio strategaeth a nodau i grwpiau o ddefnyddwyr penodol. Gan ein bod ni’n profi’r gwaith yma gyda’r Tîm Archwilio Perfformiad Iechyd, datblygais personas a oedd yn amrywio o Swyddogion Cefnogi Perfformiad i fyny i Reolwyr Archwilio, a hefyd aelod o’r Tîm Technegol Archwilio Ariannol ac Archwilydd Ariannol sy’n gweithio gyda’r byrddau iechyd.

Ble i ddechrau?

Roeddwn i’n edrych mlaen i ddechrau fy ngwaith ar y personas er mwyn gweld sut mae fy meddylfryd ymgysylltu yn ffitio gydag arferion digidol cyfredol. Fe wnaeth fy nghydweithiwr Louise Foster-Key, Swyddog Cyfathrebu Digidol Swyddfa Archwilio Cymru, rhoi ychydig o awgrymiadau i mi o’i waith ar fewnrwyd Swyddfa Archwilio Cymru, ac fe wnes i hefyd edrych ar arfer da ehangach. Mae yna blogbost da ar personas defnyddwyr ar Flog Digidol y Swyddfa Ystadegau Gwladol, ond yn y diwedd fe wnes i benderfynu defnyddio Canllaw Gwasanaeth Digidol y Llywodraeth (GDS) ar Storiau Defnyddwyr fel seilwaith i’m gwaith. Ceisiais ddefnyddio Trello, ond gan fy mod i’n gwneud y gwaith yma ar ben fy hun (a dim ond prosiect bach ydyw) doedd e ddim mor ddefnyddiol ag on i’n ddisgwyl. Felly defnyddiais Xtensio, gwefan sy’n cynnig gwasnaeth a wnaeth Louise awgrymu i mi.

Penderfynais aros gyda fformat GDS ar gyfer strwythur y gwaith, a chafodd y personas ei roi at ei gilydd trwy gyfweliadau a negeseuon e-bost gyda staff mewn rolau allweddol:

  1. Fel rhywun sy’n… (Pwy yw’r defnyddiwr? Ychydig o eiriau am y person sy’n ceisio mynd i’r afael a’r data)
  2. Rydw i angen / Rydw i eisiau / Rwy’n disgwyl … (Beth mae’r defnyddiwr eisiau gwneud?)
  3. Fel bod … (Pam mae’r defnyddiwr eisiau gwneud hyn?)
  4. Mae’r gwaith wedi gorffen pan… (dyma feini prawf y GDS, sef rhestr o ganlyniadau sy’n cael ei ddefnyddio fel rhestr wirio i gadarnhau bod y gwasanaeth wedi gwneud ei waith ac mae’n cwrdd ag anghenion y defnyddiwr)

Ydy’r ymarfer wedi bod yn ddefnyddiol?

Mae gen i ddealltwriaeth gwell o waith archwilwyr ar ôl siarad gyda staff sydd yn y rolau a gafodd eu mapio. Bydd hyn yn ddefnyddiol i mi ymhell tu hwnt i brosiect Archwilio Sydd ar Flaen y Gad, yn enwedig wrth i mi edrych ar sut y gall y Gyfnewidfa Arfer Da gweithio’n fwy effeithiol gyda thimau archwilio, yn enwedig y Tîm Archwilio Iechyd, a wirfoddolodd i weithio gyda ni ar y darn o waith yma.

Prif ddefnydd y personas fydd i wirio pwrpas ein gwaith ac i sicrhau bod ein arbrawf yn cyfateb i’r hyn mae ein harchwilwyr yn dymuno ac yn disgwyl. Nawr maen nhw wedi ein helpu ni i redeg drwy arbrofion damcaniaethol, byddwn yn defnyddio’r personas i stopio ni rhag crwydro’n bell o’r weledigaeth yma.

Bydd eu meddyliau hefyd yn hynod o ddefnyddiol ar gyfer fy rhan i o’r adroddiad terfynol. Gan mai ni yw awduron yr adroddiad, mae’r holl bŵer yn ein dwylo ni – ni sy’n rhoi’r argymhellion ar gyfer blaenoriaethau’r dyfodol. Bydd y gwaith yma’n sicrhau bod fy argymhellion i wedi’u seilio ar waith dydd i ddydd ein staff.

Mae’r personas terfynol bellach ar-lein ac maen nhw wedi cael eu rhannu’n ddwy ran achos cyfyngiadau’r fersiwn rhad ac am ddim o Xtensio. Mae’r set gyntaf o personas yn seiliedig ar rolau Archwilio Perfformiad, ac mae’r ail set yn cynnwys personas sy’n seiliedig ar rolau’r Tîm Technegol ac Archwilio Ariannol.

Mae’r personas yn rhoi lefel o ddisgwyliad i ni ar gyfer ein gwaith. Os yw’r gwaith yn fethiant, maen nhw’n rhoi meini prawf i ni archwilio ein cynnyrch a ddarpariaeth yn erbyn. Ar ochr arall y geiniog, os mae’r gwaith yn llwyddiant fel ni’n gobeithio, mae gennym gyfle i weld os all personas rhoi ffocws glir i ni mewn agweddau arall o’n gwaith wrth i ni ceisio sicrhau bod gwasanaethau cyhoeddus yn wir yn rhoi gwerth am arian cyhoeddus.

Acquiring data for a cutting edge audit office

How is the Wales Audit Office working to ensure that it provides audit that’s fit for the future? Dyfrig Williams blogs below on his work with the Cutting Edge Audit project.

Over the past few months, I’ve been working on the Cutting Edge Audit project, which looks at how the Wales Audit Office can challenge our existing use of data and technology and assumptions that we normally take for granted. We’re thinking radically about how we might use new technology to transform the way that we work.

It’s been a fantastic piece of work to undertake, which has really put that radical thinking into practice. The project’s being led by my colleague Steve Lisle, who is reporting directly to the Auditor General for Wales. This has meant that we’ve moved away from hierarchy into a much flatter structure. We’ve also been outcome focussed – we’ve been testing and prototyping as we go so that our risks are well managed and that we learn from failure.

I’ve been working on how the Wales Audit Office acquires data to give us deeper knowledge and fresh insight.

Data Maturity

data_maturityIt was helpful to think about Data Maturity when we were doing this work. Data Maturity is the journey towards improvement and increased capability in using data. Data Orchard have used this model (which I’ve nicked from a great post by Ben Proctor) of stages in an organisation’s development:

  1. Ad-hoc gathering of data in some areas
  2. Pulling data together centrally
  3. Starting to use data looking backwards
  4. Using data in real time to manage the organisation and move resources rapidly
  5. Modelling the future before making decisions to enable better decisions to be taken
  6. Modelling the future the organisation wants and working backwards to understand what needs to happen now to deliver that future

This is very much a journey for us as an organisation, but it has helped to inform my thinking. It’s helped me think about how we get to point 6, where we’re modelling the future that the organisation is working towards, and ensure that the things that I’m working on set us out on the right path beyond the lifespan of the Cutting Edge project.

My prototypes

I’ve been working on two different tests within this field. The first is an Open Data prototype, which has been more challenging than I expected because the Wales Audit Office is a secondary user of data. This means that we use data that is gathered by others, so we don’t always have the right to share it. I have managed to find a useful dataset though, so my next step is to set it free into the world and look at the challenges around how we can make it as useful as possible.

I’ve been putting the Good Practice Exchange’s principles into practice in this work by visiting other organisations to learn about the work they’re already doing because there’s no point reinventing the wheel. I’ve also been thinking about how we adapt rather than adopt their work to suit our organisational needs, because after all, a one-size-fits-all approach never works.

I’ve blogged before about why the public sector needs to start thinking about its approach to Open Data, and we subsequently ran a Google Hangout to look at why it’s an important topic. Hendrik Grothius has written an excellent blogpost on how organisations can start to publish Open Data, and it will be a brilliant starting point for me as I get to grips with this.

My second piece of work has been looking at how we enable our staff to make better use of data, thereby minimising the audit burden. I’m looking at how we can bring together data from public bodies in a way that makes it easy to access, open to everyone, and give us an improved insight into the performance of the Welsh public sector, and international comparators. I’ve been talking to our staff so that I can better understand what type of approach would be useful to them. I’ve developed personas to help guide our work in this area, which will shape the next phase of this work and ensure that my part of the final report is focused on user need.


I’ll be writing future posts to share my approaches, what I’ve learnt and what I would do differently next time. We are working iteratively so that we learn from each development and how we can build on that learning going forwards. If those prototypes don’t work, we’ll be looking to learn from failure and see what the organisation can do differently in the future.

At all our Good Practice Exchange seminars we hear that public services can’t continue to work in the same way in these austere times. It’s been great working on a practical project at the Wales Audit Office, as we’re getting to grips with those same challenges and applying new thinking to our work.