NAPLAN in Kafkaland

Burkard Polster and Marty Ross

The Age, 12 May 2014

Dear reader, are you ready for a story? Pour yourself a nice hot cup of Milo, grab a Tim Tam or two and ... No, wait. Grab the whole packet of Tim Tams: it's a long story.

Once Upon a Time

Your Maths Masters were pondering NAPLAN. And they were puzzled. Plenty of pundits were arguing the merits and demerits of NAPLAN testing. And there was plenty of data: how Wombatville High outperformed West Wombatville Grammar; the stats on rural Tasmanian left-handed women. However no one ever seemed to discuss the actual tests. There was never mention of individual test questions, nor any indication of how students overall performed on any given question.

It was all very strange. What were the pundits actually discussing? What types of questions were students answering well or poorly? Were the tests even worthwhile? Or, should Mr. and Mrs. Fredfellow simply take little Alice to the zoo on Test Day? How could they decide without a clear understanding of what the tests were testing? Your Maths Masters resolved to make amends.

In April 2013 one of your Maths Masters contacted ACARA, the administrators of NAPLAN (and of the stunning Australian Curriculum). A friendly employee, let's call him Karen, directed us to put our request in an email. We did so, requesting copies of the NAPLAN numeracy tests, along with the nationwide results of those tests: for each test question, the percentage of students who answered that question correctly.

We received no reply. So, we emailed Karen again. Again, no reply. So we called Karen. She assured us that she would look into our request and get back to us. She did not.

Soon after we emailed a second person at ACARA, "George". George promptly replied, indicating that we should submit a formal application for the materials. Hmm: not exactly a process encouraging public scrutiny. We emailed George and Karen, indicating that we were willing to apply for the tests and the data but that we wished to first discuss the nature of the application. That wish was not granted.

We made a dozen attempts to discuss matters with ACARA staff, but no one would return our calls or emails. We received one joint phone message from George and Karen. They claimed that the tests could be found floating on the web, a suggestion that was false, irrelevant and plain weird.

Going to the Formal

It began to dawn on us, dear reader, that professionalism and common courtesy may not be ACARA's strong suits. Reluctantly, we gave up hope of making human contact and in May 2013 we submitted our application. We received a prompt acknowledgment from another ACARA employee, "Lucy", who cordially suggested that any further questions be directed to her. Alas, she made no promise to answer any such questions.

We asked Lucy if our application was sufficiently clear and she implied that our language was too colloquial, that we hadn't employed the appropriate and expected technical terms for the requested data. We suggested that our application was very simple, asked Lucy whether the information we were requesting was clear to her, and if so would she indicate the relevant technical language to incorporate. She refused to say. We asked again. Five times in total. Nothing.

It was no great shock when your Maths Master's application was rejected. Was it the lack of technical language? No, we were denied the data because we lacked an "ethics clearance".

An ethics clearance? For nation-level statistics? Yes, dear reader, an ethics clearance.

Applications such as ours are evaluated according to ACARA's Protocols. They are written in such a protective, only-the-right-people manner that even data without a hint of ethical implication requires an ethics clearance. What it means is that Professor Hogglesthworp from the Institute of Educational Truisms will be granted the data but that Mr. and Mrs. Fredfellow will not.

Were your Maths Masters disheartened to be denied the data? No, dear reader, not a bit. The data we had requested was and is freely available.

Each year the Queensland Studies Authority publishes a Test Reporting Handbook. Along with tons of other information the Handbook provides a short description of each question on every NAPLAN test, along with the (approximate) percentage of students nationwide who answered that question correctly.

We have no idea whether the QSA Handbooks violate ACARA's Protocols, however we urge caution. Dear reader, if you intend to download the Handbooks we suggest that you first chant "I am as ethical as Professor Hogglesthworp". Hopefully that will satisfy ACARA's gatekeepers.

And the NAPLAN tests themselves? No, your Maths Masters didn't receive them either. The reason? None was proffered. Apparently we had already been "provided with advice" in that regard. We suspect that the actual reason was our failure to have Professor Hogglesthworp sign on to our application.

Dear reader, you will probably not be surprised to learn that your Maths Masters already had copies of the NAPLAN numeracy tests. The tests may not be floating on the web, but they are floating around, in schools here and there. As with the data, the apposite question was whether ACARA was willing to provide the tests to Mr. and Mrs. Fredfellow, to the public at large.

So, is that the end of the story? Time for the Maths Masters and the Fredfellows to admit defeat? No, dear reader, it's time to open another packet of Tim Tams.

Information Wants to be Free(ish)

In October 2013 your Maths Master lodged a complaint with ACARA. We received a polite apology, along with a bucket of disdain and a swimming pool of denial. At that point we could have, and with hindsight should have, written to the Commonwealth Ombudsman. We chose a different route.

Your Maths Master filed a Freedom of Information application, requesting all previous NAPLAN numeracy tests and the national performance data. The process started off rather well. There were some silly excuses and some Laurel and Hardy moments, but in general our application appeared to be treated with proper respect.

In January this year ACARA sent your Maths Master the NAPLAN numeracy tests from 2008 to 2011, together with the performance data. (ACARA had objected to releasing the 2012 and 2013 tests. Their reasons appeared to be pretty flimsy but we chose to defer that battle; those tests will be re-requested in a second FOI application.)

So that was it? Silly reader! Of course not.

Just as the tests were about to be handed over ACARA mentioned a teensy weensy problem: some of the images used in some of the tests were subject to third party copyright. The implication was that any use your Maths Masters made of the NAPLAN tests would have to respect the rights of those copyright holders.

It seemed a small problem since a few new images could easily be substituted without changing the character of the test questions. So, ACARA just had to supply the copyright information. And you'll never guess, dear reader: ACARA didn't have it.

ACARA was not sure of the copyright status of many of the images and their lawyers were in the process of trying to sort it all out. That's fine: your Maths Masters could wait the "few weeks" it would supposedly take.

Finally, months later, ACARA got back to your Maths Master with a confidentiality agreement to sign. That seemed odd, since it was difficult to imagine any reason to want the copyright information kept secret. In any case, ACARA's letter contained a much bigger surprise.

Months after settlement of the FOI application, ACARA had decided to enforce their own copyright over the tests. In effect, they had substantially reneged on their FOI decision.

Ethical questions aside, is ACARA permitted to change course like that, and does it carry any legal weight? Your Maths Masters don't know, though we will certainly endeavour to find out. But for the time being at least, ACARA's decision has complicated matters.

Your Maths Masters had intended for the past NAPLAN numeracy tests to be posted, to have them freely available so that Mr. and Mrs. Fredfellow, everyone, could see what was contained in little Alice's numeracy tests. The council of the Mathematical Association of Victoria had agreed that this was worthwhile and had kindly agreed to host the tests. Those plans are on hold.

The numeracy tests are now in a strange FOI limbo-land. Your Maths Masters and the MAV cannot safely publish the tests, however ACARA's FOI Disclosure Log lists the tests and performance data as "publicly available". The Log implies that anyone can request copies of the tests, and indicates how to do so. Will ACARA honour such a request? Will they change their mind again? Your guess, dear reader, is as good as ours.

So, finally our story ends (for now). All that's left is the moral.

Goodies and Baddies

Maybe your Maths Masters are the bad guys here. Why are we so concerned to publish the numeracy tests? And who are we to do so if ACARA in its wisdom has decided otherwise? Well, let's consider that wisdom.

ACARA, and many others, are concerned about "teaching to the test", that teachers are spending too much class time practising for the NAPLAN tests. Whether or not that is a reasonable concern, would publishing previous NAPLAN tests make any tangible difference? Not a chance.

Any teacher who really wants the past NAPLAN tests can easily hunt them up, just as your Maths Masters did. Indeed, teachers don't even need to hunt: zillions of past test questions are available on Improve, a Government-run teaching resources website. So it's only Mr. and Mrs. Fredfellow who can't access past NAPLAN test questions: all they've got to work with are the hundreds of assorted NAPLAN-like tests for sale at their local newsagent.

For better or worse, any teacher or parent who wants to drill NAPLAN questions can already do so perfectly well. But why might it be worse? What's so wrong with teaching to the test?

Despite some ridiculous suggestions otherwise, teaching to the test is not cheating and it will improve test scores. The question is, is the test worth worrying about, or is it a distraction from more valuable pursuits? The answer depends upon the test.

Any VCE teacher who didn't teach to the test would likely be sacked; VCE results are that important. NAPLAN is not VCE, but NAPLAN results are more important than ACARA cares to admit. Still, NAPLAN obviously distracts schools from non-NAPLAN activities such as music and art, which is a genuine, huge concern.

However, the CEO of ACARA has claimed that NAPLAN is a test of "reading, writing and mathematics" (emphasis added), which would definitely be valuable. Except, it isn't true.

Your Maths Masters are so, so tired of the presumption and the pretence that "numeracy" is synonymous with "mathematics". It is not, not even close. Numeracy doesn't even rise to the level of basic arithmetic. You don't believe us, dear reader? Then grab a NAPLAN numeracy test and check for yourself what is and what is not being tested. Oh, wait ...

The reason your Maths Masters are determined to publish the NAPLAN numeracy tests is so there can be a proper, informed debate on what numeracy is, and whether the Australian public is really happy about numeracy's supplanting of mathematics. As it stands, we doubt the Australian public is even aware of what is happening, that it is not just the study of the arts that is being harmed by NAPLAN, but the study of mathematics as well. The only real effect of ACARA's passion for secrecy is to make that debate so much more difficult.

Dear reader, your Maths Masters cannot show you the NAPLAN numeracy tests. We cannot critique them in any reasonable manner. Attacking a few silly questions is very easy but it doesn't begin to indicate the scope of the problem. All we can really do is offer you our opinion.

We strongly believe that the Fredfellows should be considering that trip to the zoo. We suspect that little Alice may well be happier and better educated for it. At least for a day, and maybe for ever after.

THE END

Burkard Polster teaches mathematics at Monash and is the university's resident mathemagician, mathematical juggler, origami expert, bubble-master, shoelace charmer, and Count von Count impersonator.

Marty Ross is a mathematical nomad. His hobby is helping Barbie smash calculators and iPads with a hammer.