Another Certification Story
Dave recently shared a story about his own experience with a government certification program. Here is a semi-related story of my own (really, my wife’s, but I don’t think she’ll mind me sharing it):
A few months back, Zazzy applied for a professional certification offered by a non-governmental (i.e., private) credentialing center. She had all her affairs in order — met all the requirements and properly documented such per the organization’s standards — and assumed she’d receive her certification shortly. Instead, she received a letter indicating her application was being audited. She was not pleased. Some Googling indicated that the organization flags approximately 20% of applications for auditing. The audit is a fairly simple process: basically, Zazzy had to offer evidence that she took the requisite number of training hours and had worked the requisite number of hours in her specialty. On the initial application, she simply had to state that she met both requirements. Fortunately, Zazzy is a meticulous record keeper and within an evening’s time she had all of the necessary evidence ready to be mailed out.
Still, she was incensed. She alternately wished that they had simply asked for the necessary documentation from the onset of the application process or didn’t bother with the audit process at all.
While I understood her emotional response and the frustration she felt, I explained why the organization did what they did thusly:
1. As a private organization, the strength of their certification is derived from the quality of the people receiving it.
2. As such, it behooves them to take steps to ensure that those who receive it meet a certain standard.
3. One method of doing so is a careful review of candidate applications.
4. However, a very careful review of every candidate’s application is a timely and costly process.
5. Presumably, the organization determined that carefully reviewing approximately 20% of applications is sufficient to ensure the quality of those they bestow certification upon while also managing costs and keeping the certification accessible.
6. Ultimately, it was in her best interest that the organization did what they did. If they never audited anyone, her credential wouldn’t be worth the paper it was written on since anyone could receive it. IF they audited everyone, the credential risks being prohibitively expensive, under utilized, and ultimately of little weight in the field.
Now, there are certainly some assumptions baked into this explanation. But I’m pretty confident it is the right way to look at the situation. And it gave me a better understanding of and appreciation for how a private certification/credentialing organization might function than I had back when Professor Hanley discussed it with regards to his proposed Anti Rent-Seeking Amendment.
Of course, they could ask for documentation from every applicant but only process it for 20% of them. Then the unlucky ones (like Zazzy) wouldn’t know they’d been singled out. It would also allow them to do retrospective audits if any questions arose. And it might discourage people from making shit up, since the consequences go from “You said something you can’t document” to “You were caught telling a bald-faced lie”.
Actually, I don’t see any downside. Why wouldn’t they do that?Report
Good insight, Mike.
I could guess at a reason. Perhaps the certification is considered optional by a lot of people. If they represented getting certified as a process that required a lot of documentation from the get-go, then a lot of professionals wouldn’t bother in the first place.
I think the more likely reason though is that they might not have thought of doing it your way.Report
Good point, @mike-schilling .
@vikram-bath
The field is a relatively new one (it is related to electronic health records) so it certainly is possible that the credentialing organization — similarly new but a subsidiary of a most establishing organization — is trying to establish itself by encouraging application.
I could also see logistical issues. Documenting what they asked Zazzy to document requires 20-40 pages of paperwork in most cases. Having to file and store all that in an organized way might be prohibitive.
That said, Mike’s suggestion does certainly seem to be superior on most accounts.
However, I think I’m right about auditing some-but-not-all applications being annoying but ultimately in Zazzy’s best interests as a legitimate applicant.Report
@kazzy
Yes, but if the certification is new, wouldn’t the organization offering it want to ensure it’s good name by doing 100% auditing to start off with? How would it look in the press if this new cert organization was reported to have been essentially a “diploma mill” and wasn’t checking their applicats particulars? Seems the downside would be more harmfull than the upside of doing less audits/full documentations.
As to the cost to the organization providing the certifiction, I think the real cost in the people wanting the cert, in their time and such to document what’s requred to be certified. Wouldn’t the person processing the applicate just have to review the paper work and confirm it’s legit and correct? Doesn’t seem like a lot of work.
Net net, I think kazzy’s point is on target, but I also think that Mike’s thoughts are probably the “preferred method”. It’s always good to have data on members…just in case there is some question by “da authorities”.Report
@damon
Zazzy was asked to provide documentation that she accrued the necessary training hours (I believe it was 30) and a letter from a supervisor certifying that she has worked the requisite hours in her speciality. Training hours are usually gained 3/4 to 1.5 at a time, meaning 30 training hours might require 20-40 individual certificates noting completion of a given training. A thorough audit would seek to verify each of these certificates since they aren’t all that hard to dummy up. Training is offered through a myriad of organizations which themselves must be accredited by one of a number of other groups. So Zazzy might sent in a certificate that says she completed a one hour course offered by the XYZ Corp which is itself accredited by the ABC Org. And even if the group she is seeking certification from knows that course is real and knows that XYZ Corp is legit and approves ABC Org for accrediting XYZ, they might still want to verify that Zazzy’s certificate is real by contacting XYZ and making sure her certificate number belongs to her and isn’t faked. Repeating that process 20-40 times can be time consuming.
Or maybe they just say, “Okay, she sent in a pile of paperwork. She’s probably fine.”
I really don’t know. There is too much unknown to me about the mechanics of the whole process to comment on the effectiveness and efficiency of this particular process. But I do think auditing = good for Zazzy, as frustrating as it might have been to get that letter. As someone who rightfully earned her certificate, it benefits her that the organization takes steps to ensure the certificate is worthwhile.Report
I could also see logistical issues. Documenting what they asked Zazzy to document requires 20-40 pages of paperwork in most cases. Having to file and store all that in an organized way might be prohibitive.
Not to be rude, but… we have these things called scanners and computers these days? The last state organization whose collective shoulder I looked over to see their process was required by statute to accept paper applications (and allowed to accept online ones — statute is always behind). The first stop for every paper application and supporting documentation was the scanning station. No one ever looked at the paper copy again.Report
@michael-cain
For whatever reason, they requested hard copies. Maybe they immediately scanned them and destroyed the hard copies, I dunno.Report
“Or maybe they just say, “Okay, she sent in a pile of paperwork. She’s probably fine.” ”
This is exactly what happens.
The point of the audit is not to fulfill the credentialing center’s requirements; they can do that just fine merely by looking at job history and post-hiring performance. The point of the audit is to comply with the FDA’s requirement that audits be performed and records be retained.Report
Along with Mike’s point above, I also wonder if the savings realized by auditing 20% of the applications are really that great. Maybe if they audited only 5%, but 20%, it seems to me, could be almost as costly as doing all of them.
I’m spitballing here and don’t really know what I’m talking about. It’s just that 20% seems so high one might as well do them all or do Mike’s suggestion, especially if certification is supposed to be a big deal (I’m not sure how big a deal it is for Zazzy’s profession).Report
Well, the 20% figure is what she found Googling online, so it might be something other than that in reality. I don’t know what the ideal number is. I’d venture to guess that balancing cost/efficiency against effectiveness would land you somewhere between auditing none and auditing all but where that might be would depend on a number of things I don’t know enough about. I’d trust the organization to make the best determination.Report
Presumably, auditing 100% would be ~5 times as costly as auditing 20%.Report
It could be more, as a second-order effect; prospective certification-holders hear “oh, so-and-so agency is going to 100% audit”, and so they choose to use a different agency (or forego the certification altogether.) Meanwhile, the groups that require the certification find that it takes longer to get people certified; too long, in fact, and so they start doing the certification themselves (or use different agencies, or accept “provisional certifications” that somehow end up being permanent.)Report
There’s no point making these wild assertions if you’re not going to back them up.Report
@james-hanley
Oh, you and your “math.” I don’t do it like that. I just intuit the numbers and hope they all add up.
Peace, man!Report
Historians!Report
Empiricists!Report
I have my concerns, too. Certify means verify a set of facts for this specific person/business/land. To be of any value, each and every certified entity should be verified; not a statistical model.
We’re often told by free-market advocates that voluntary certification is better then government regulation. But if voluntary certification means random spot checks only, then there ought to be a confidence interval associated with it. I also think that if the certifying agency isn’t collecting the documenting information the certification requires, they’re certifying air.
As a consumer, if some company claims that it’s certified for some sustainable purpose, given the nebulousness of the word ‘sustainable,’ I actually want that company to be individually reviewed, their management practices verified. Woodlot certification would be my example here; woodlot owners get tax incentives to certify their land, wood products from certified lands can generate a higher selling price (rather like ‘organic’). But to be of any value, the specific land has to actually meet the standards; not some percentage of certified land, but all land certified, on a parcel-by-parcel basis.
Statistical accuracy is one thing; it has a margin of error associated with it. But the very nature of certification is that this specific entity has been certified, not some % of similar entities have had their certification credentials verified.Report
I’m friends with a good number of farmers up here.
I’ve never thought to ask, but I would be surprised if any of them had ever even seen an FDA inspector before.Report
I honestly don’t know what to make of this, @vikram-bath
We’re not talking regulatory process, we’re discussing voluntary certification, and often, this is a way of either meeting or exceeding regulatory standards, and is used to justify less regulatory oversight by government.
As to regulatory standards for food from farming, there’s a complex web of things that happens. A lot of it is not under the FDA’s jurisdiction (unless there is a documented problem). Manure management, for instance, is typically regulated at the state level. It has to comply with the Clean Water Act, so in egregious cases, there’s some chance the federal EPA agent might be called in, but mostly, whomever has that responsibility in state government (in mine, it’s the state’s Dept. of Environmental Protection (DEP)), and their role isn’t just in enforcing, it’s helping farmers craft best management practices that are in line with both state and federal law. They help farmer’s solve their manure management problems, since they have a lot of expertise; they don’t just tell farmer’s that they’re breaking the law.Report
Manure Management is such a bullshit regulation.Report
zic,
Voluntary certification is just as prone to “watering down” and subversion by forces that want weaker regulation, if not moreso than government regulation.Report
Obviously; it depends on the certifying body and rules.Report
Statistical accuracy is one thing; it has a margin of error associated with it.
This is true regardless, though, zic. If you have a rigorous enough verification procedure, you have a high audit cost, which will be passed on to the person acquiring the certification, which makes the certification more valuable, which increases the incentive to suborn the process.
Funny thing about audit and security, you hit diminishing returns pretty quickly.Report