TRANSCRIPTION 🔗
Conor:
0:00
Hello and welcome to the performance audit report. My name's Conor McGarrity.
Yusuf:
0:03
And I'm Yusuf Moolla. This episode is an older episode. It comes from The Assurance Show, which is a podcast that has been going for a little while. That podcast was focused a little bit more broadly. So internal auditors, performance auditors, and a few other people that have primarily data as a focus in their work. But this episode is specifically performance audit related. There's about 10 of them that will be putting onto the show from time to time. They are a bit older. Some of them are interviews with performance audit professionals in Supreme Audit Institutions. They will be relevant directly relevant to you as a performance audit report listener. Just to know that they are a bit older. So if you have listened to the assurance show before you may have come across this episode, and so apologies for the duplication, we just think that it's worthwhile bringing that across, into this podcast because of its relevance. Hope you enjoy it. When we talk about open data, what are we actually talking about?
Conor:
0:55
For our purposes, we talk about data that's fully available, can be used and accessible. So generally that means stuff that we can get online can be reused and distributed by anyone. Although sometimes there's obviously some requirements to attribute the owners of the original data. The open data handbook that's available online and it recognizes 3 million components, first one being available and access. So it must be available and at no more than a reasonable reproduction cost, and preferably free. Second one needs to. Able to be reused and redistributed the terms , of the data itself must mean , that it can be used , and even mixed with other data sets or linked and joined and analyzed with other data sets , for its purposes. And the third one is universal participation, and that's where everyone, regardless of where you are, who you are and what you're doing, when it should be able to use it for non-commercial purposes or where you use for commercial purposes, you need to attribute that correctly. So that's in essence, the three main things. Needs to be available and accessible, needs to be able to be reused and redistributed, and there must be no discrimination in who can use it.
Yusuf:
2:05
Why is open data important to performance auditors?
Conor:
2:08
It's another evidentiary source, although it's only come to light probably in the past decade.
Yusuf:
2:14
What are the potential outcomes for auditors in using open data?
Conor:
2:19
When, for example, you're doing a performance audit, you've got an opportunity to educate some of the audience you're dealing with about what open data is available and how useful it is. The second thing is that open data is valuable often to challenge prevailing assumptions. So from a performance audit perspective, one of the main things governments try to do is deliver services and programs based on assumptions. At a point in time when that service or program was developed, by the time that's rolled out could be one to three, even four years after the original decision was made the way in which the recipients or beneficiaries of that program, mainly being the public, consume those services or will use that program may have differed. Open data gives you some opportunity to test some of the original assumptions
Yusuf:
3:08
If you already have hypotheses or at a high level criteria that you need to evaluate, then open data can help you determine whether those have been met.
Conor:
3:15
Open data is, one of the evidentiary sources you would go to. So you would probably, in the first instance, rely on information or data available within the entity, you're performing the performance order over. But open data is definitely a strong point of triangulation to try and prove or disprove whether a criterion has been carried out in accordance with this design or whether a hypothesis, is proved or disproved. Open data gives you that ability, certainly from an external Auditor-General's or Comptroller-General's viewpoint to be able to take not just the entities data that you're, looking at, but to join that with anything that's available outside and beyond that, that may not actually be readily available within that entity or being looked at within that entity itself. Governments are moving towards transparency and putting data out there. And that's something we need to tap into.
Yusuf:
4:13
Interestingly, there's a move within private sector as. There are a number of industries where data is going to be needed to be put into the open domain. So it may not be completely open given the definitions that you spoke about earlier, there may need to be quite a bit of privacy and security associated with, the way in which that data is then accessed. But the open banking regime that's coming in which will be followed by a range of other industries. I think starting with energy and then moving onto telecommunications. And that's where proprietary data from individual institutions that can be associated with consumers will need to be shared between those organizations and the few others that are coming through newer sort of startup fintech and other players.
Conor:
4:56
You've touched on a bigger point there, and it's more of a strategic opportunity as opposed to concern for internal and performance audit with all this more open data becoming available. That means that the power is going to vest in the hands of the users, which is you and me and members of the public. That creates a question in the minds of performance, auditors. If the public has that information that data available to them, then they're gonna expect us as auditors to be using that wisely and giving them insights from what's available.
Yusuf:
5:30
Internal audit don't generally have a very direct customer
Load full transcript
facing relationship. Most of internal audit work will be within the organization and they won't really. Lias directly with the organization's customers. So yes, there'll be an expectation that the organization does something with the data. I think that expectation is going to play out more as a commercial reality than as a consumer expectation. If you're not doing anything with the data and you can, when somebody else is and is able to provide that value, you might just go to them because they can do something for you that your current organization can't. Or they might use their data to figure out what it is that you need, and then lure you away.
Conor:
6:08
Well speak briefly about the types of open data that are out there. Some of 'em are quite obvious, but yet overlooked by some of the teams. So the most obvious one, one that we've relied on extensively over the past few years with our work is statistical data and not obtaining through, for example, a census process, you know, households and individuals. And that is very useful data.
Yusuf:
6:28
Quite a few of the statistical agencies, even though the actual census. So the, you know, the physical census is only would only be done every five to 10 years in the intervening years, they would have projected estimates. You can generally get population estimates and those population estimates will go down to so in Australia and the UK, The post code in the us, the zip code, and then various countries will have variations of that. You can get down to a fairly granular level and that's useful for both performance auditors and internal auditors, that sort of information. So we've used it. For example, for internal audit, we've used it to identify the level of sales and the level of complaints and a whole range of matters that are customer focus. Where we have information on the customer. So if you have the customer's individual customer's addresses, or if you have customer addresses and customer data sort of summarized at postcode level, or even at suburb level, then you can use that population data and related population statistics to do comparisons between services that we provide, and products that we provide and the level of potential that exist within those areas. So a little bit different to, we're not looking for fraud, we're just looking for levels of penetration there.
Conor:
7:40
The other lovely thing, the public sector statistical agencies do is they often break down their data by industry type, which can be quite useful depending on, which sector you're located in. So it could be for example, In the housing sector or could be in the motor industry or property industry or whatever. And that that's often really useful to be able to, do some benchmarking for either your organization's performance or, your jurisdiction's performance.
Yusuf:
8:07
Other types of open data that we use that are fairly low in terms of complexity. So two example. First of those is where you have businesses that you deal with, either as suppliers or as clients, you can actually get information from central business agencies that tell you whether those individual companies that you deal with are legitimate, are registered with them, have actually maintained their registration over the years. Whether they registered for certain types of tax in the UK and Australia to be GST, South Africa would be VAT, the U.S. maybe other types of local taxes. So there's a lot of information that you can get around, particularly suppliers. So you wouldn't use it as much for customers, but particularly suppliers to understand whether you're dealing with suppliers that are legitimate and that we should be dealing with. The other example of fairly simple data is where you have information on where. Money is going to, or money is coming from where you have a large customer base. So if you're getting money coming in from different banks and you want to know, where are my customers banking? What are the institutions that they use? Do we have a concentration of money is coming in with any particular organization, and this is useful for resilience and crisis planning. You can use open data sets. Often you get information that comes in that just tells you what the, bank code is. And it's different. Again, details around bank codes in Australia, you have what you call a BSB in the UK. You have something that they call a sorting code. What that can help you identify is based on the branch number, bank combination. This is usually open data that you can get. You can identify whether you have a concentration with any particular bank.
Conor:
9:41
You've talked there about open banking information or open banking data, and there's certainly a move at least some jurisdictions to make that more available. And you've just given a couple examples, how that's useful. We talked about census data, demographic, statistical data that most public sector statistical. Collect and make publicly available, which is really useful. Some of that you do have to pay for, we've encountered a few times. One of the other data sources, that I think is really useful is that held around property, household data, that sort of thing. There are, for example, peak bodies that. That sort of data routinely. And that can be , , useful in particular to look at things like household consumption, trends of consumption, patterns, that sort of stuff, to be able to profile activity in that, area. Again, as we always come back to in many of our discussions is the issue around quality. How do we make sure that we maintain the quality of what we're getting in and the integrity of the data that we're bringing in?
Yusuf:
10:36
There's three key things that we need to look at to ensure that the level of quality is maintained. The first of those is are you using the right timeframe? So the open data that you're using, does the timeframe match the proprietary data that you're joining it. So
quite often, you're not gonna be using open data by itself. Right. So you're gonna be using it with some other proprietary data that you have. when you're joining it to that proprietary data? Are you joining it based on the correct timeframe? The second thing is, are there any missing items in the sequence of data that you have? So if you have daily data, let's say for whatever reason, Are there any days that I'm missing, if you have monthly data, are there any months missing? And if they are, you need to work out well, how you're going to treat that. So are you going to fill in the gaps because you can use , extrapolation and other techniques to fill gaps, particularly where the gaps are fairly granular. I mean, if you have a whole year that's missing, you may need to. Take a different approach. You may need to exclude that here from your analysis. So that's where you have gaps in the data. And then the third thing is, , making sure that the data is correct. So sometimes you get data in and the data is duplicated for whatever reason. So, you know, you may have a data set that has, , individual items and totals. And so that means that your data is actually larger. You will be double what it should be. The other trap that you often encounter is where the data set that you download for some or other reason. And this happens quite often with open data and you may have a data set for 10 years and one of those years for whatever reason is duplicated. So the information in there will always appear to be higher. So wherever you see these sorts of anomalies, Higher than you expect it to be. What you want to do is go in and have a look at the detail and make sure that you're not bringing something in that doesn't have the level of integrity that you want. And then don't just chuck it away because you can't see any integrity in it. You fix those items so that you can actually use it. And those are the three key things to ensure when. Managing quality within your open data set so that you can join it properly.
Conor:
12:35
The key takeaway is that even though we're proponents of using open data, where possible, I guess you need to make sure that you make a good assessment against those three criteria early on to make sure that you're not overinvesting time and its use.
Yusuf:
12:49
There's a couple of things. One is that there are some organizations that produce open data. And by that, I mean, there's quite a few states and territories and countries and provinces that make open data available. Some of them actually have an audit step as part of the creation of their open data. In Australia, we're not that sophisticated yet. But in other jurisdictions they actually have an audit step and they publish the results of those audits. So they'll, we've audited, the data that exists in our open data portal, and this is the result we found this percentage of inaccuracies or this percentage of missing data or whatever. If you have that situation, you
can actually go and look at what the result of that audit is, and then work out how much you need to do to check the quality of the data that you're using.
Conor:
13:37
Which goes back to how much reliance we can place on it.
Yusuf:
13:39
Yeah. And just common sense audit. So if we are using the data and we are changing it up and checking it for quality and these things quite often, there's a need then to share the data that we've pre-processed, how would we go about doing that?
Conor:
13:53
Pretty straightforward from a performance audit perspective, some of the progressive performance audit auditors these days in various jurisdictions are making some of that processed data available via their website. Some might consider it a byproduct of the report , but it's actually a really important product in and of itself. And that's the linking joining, analyzing of those data sets because it can tell some fantastic stories. So, that's one way in which those data are being shared and that's via websites, and interactive and clickable and that sort of thing. Some of the other ways are providing some of those cleanse and analyzed data sets back to the entities that they have actually audited and said. This is what we've seen. This is what we've observed. These are our findings, your data. Wasn't that great. In the first instance, we've taken time as part of our audit work to do all this cleaning analysis. It's only fair that we give that back to you. And that's another very useful. Unvaluable byproduct back into those agencies themselves. So two things there. One is for the public, is it making the analyzed open data and especially where it may have been linked with other open data or proprietary data in so much as it's not private available on the websites. And the second thing is given some of that open data linked. An entity zone data back to the entity to have a look at.
Yusuf:
15:13
So in the first instance there, that's creating more open data . Conor: Yeah, absolutely. The challenge with sharing the data directly is that you don't want to be in a situation where somebody uses your work directly and doesn't check it themselves. Cause quite often you can process the data, pass it on, and the risk you have there is that they just rely on what you've done and go ahead and use it. And then you find out later, that you made a mistake and that your mistake is now promulgated through the organization and so what you do is you say, these are the steps that we followed, have a look at the steps, have a look at the data and see if there's anything we are missing and then use. That's a bit safer because it means that people have to actually start thinking about what they're doing, as opposed to just taking some data and using it blindly.
Conor:
15:55
It comes back to the reasonableness test. Was the process these guys followed to do this analysis and come up with this outcome. Is it something that we think is reasonable and
we should or could rely on?
Yusuf:
16:07
Yep. So the business will need to work out, can they actually rely on that or do they need to do something else? Yeah, absolutely.
Conor:
16:12
Sure.
Yusuf:
16:12
For performance audit, where would you usually go to find open data.
Conor:
16:16
Government websites in the first instance. So a lot of jurisdictions these days, have websites dedicated to open data. They require their agencies or their entities to provide, data to a central source. And then that gets published online. So if we speak about Australia and the UK and the USA, for example, many of the states and jurisdictions, or even county councils produce a lot of data online. So that might be from procurement data as to. Tendered and won contracts to performance data about these are our main objectives for this organization. And this, this is how we're performing against these objectives to other types of really operational data about this is how many people are visiting our facilities. So the answer is that a lot of this data is actually already been pushed out there by some of these more forward thinking organizations. Two things worth noting about that. The first thing is that demonstrates a commitment to transparency. But the second thing is it's also a bit about accountability in that, The general public can actually query and interrogate and ask questions about where their money is actually going.
Yusuf:
17:28
that is within the domain of the public sector organizations that are looking to promote that transparency and promote accountability in the work that they do. What is it that an order that can do to help improve that level of transparency or help improve that level of accountability?
Conor:
17:44
It's up to that public sector, entity or department or agency as to what they want to publish online in most respects, the important role of a performance auditor is to call out where there are deficiencies in the publication of open data. So if I'm doing a performance audit, for example, in the department of. And one of the main things we're interested in is public sector waiting times for operations and so forth. And the health department does not have that information online. Although many other departments in our jurisdiction have waiting times for various of their main services online, it may be useful for performance auditors, given their independent and somewhat helicopter review to say, well, We don't think it's reasonable for this department, not to publish that information. So there's a very important role there for performance auditors and calling out situations where data is not being openly made available when it reasonably should be.
Yusuf:
18:45
There's a number of places that you can go. So go to your state or territory or country, open data portal. If you don't know what that is, or you can't find what you're looking for. Another way to find, , open data. And we use this quite regularly is Google's open dataset search, , pretty straightforward. You just type in datasetsearch.research.google.com. You can then actually search for datasets within the end role, give you different options as well.
Conor:
19:11
In summary would say that if you're not using open data now for your assurance projects, you need to ask yourself, why not. We've explained the value of how some of these open data sets can really contribute to your projects. The last thing to do is to basically start exploring.