Guardian Cryptic solvers – this may interest you.
Posted on behalf of Dr Chris Wheadon, Founder & CEO of No More Marking Ltd
I plan to conduct a study analysing the difficulty of the Guardian Cryptic crossword over time. This research will be modeled on the study I co-authored, “Fifty Years of A-level Mathematics” (Jones, I., Wheadon, C., Humphries, S., and Inglis, M.), which was published in BERJ (Volume 42, Issue 4, pp. 543-560) and won the BERJ Editors’ Choice Award in 2017.
The methodology is simple. I have chosen a sample of 5 clues from every puzzle in the book “A Clue to our Lives” and included one puzzle per year from the point where the book concludes to give me a sample spanning 1930 to 2026. I will present judges with a series of pairs of clues along with their solutions and ask them to choose: “The harder clue?”
Ideally I hope to recruit around 50 judges to complete 93 comparisons each. The analysis will reveal any broad trends in difficulty over time.
The study was stimulated by simple curiosity after completing the puzzles from “A Clue to our Lives”, and having been a regular solver since the early 1990s. I don’t have a strong directional hypothesis, but my feeling is that, contrary to our finding on A-level mathematics, the puzzle has not become easier over time!
To take part, solvers simply need to click on this link, which will ask them for an email address before they begin: https://au.
Great idea for a study! I’d just like to report that several clues in the collection appear to be concise-style non-cryptic clues (single definition only), which may mess with the data collected. I’ve been unsure whether to grade clues which don’t work as cryptic clues as “harder than any legal cryptic clue” or to judge the obscurity of the single straight definitions on their own independent merits.
The single definition clues are from early Guardian cryptic crosswords c1930 to 1950. I agree that they are probably the hardest to judge!
No problem for me Bluejacket @1 and Chris @2. Quick clues are always more difficult as they don’t have all the extra stuff that enables you to work out a unique answer.
Can I ask why the site requires me to record my voice before participating Chris @2? It’s definitely discouraging me from contributing.
I would like to do this but I never click on links and I do not have an email address .
Since the 1990s I would say on average the Guardian crosswords have got slightly harder but the range of difficulty is much narrower . We now get very few seriously hard puzzles and not enough friendly puzzles for newer solvers . John Perkin had the right balance .
When I took the printed paper every day (mid-1980s to late 1990s ?) there seemed to me to be a gradual increase in difficulty through the week. I could often polish off Monday’s in a lunch break; whereas Saturday’s needed a lot of puzzling & wrestling over a few days. Now, I still always get a Saturday paper and wrestle with the Prize crossword; if I happen to get a weekday paper, it’s not much easier !
(I still like the crossword as a printed puzzle – online is just more screen time).
Comment #6
I also agree with Tim C, comment #3. I could do the survey whilst sat with family, but not if it needs the microphone. I’ll have to squirrel myself away somewhere.
1) definition-only clues can be devilishly difficult when solved stand-alone, i.e. with no checkers. Set (4) for example would be nigh-on impossible.
2) The Times likes to increase the difficulty level of its puzzles as the week progresses, as do other papers. However I’m not absolutely sure that The Guardian does this in any hard-and-fast way. Better ask Alan.
Roz#4 Surely you have to give an email address to post on here?
Not my own , I borrow a Chromebook that has been set up for me with just three sites , maybe it has an email address but it is not mine . KenMac knows about it , as did Gaufrid .
I have never known the Guardian to get harder throughout the week . Traditionally an easier Monday and harder Saturday but not always the case these days . John Perkin as editor aimed for two easy , medium , hard each week . Usually the second hard puzzle was a Wednesday , Bunthorne or Fidelio etc . Most weeks now have 5 or even 6 medium puzzles .
I have turned the audio comments off now, that was an oversight on my part. They were only optional in any case. Thank you so much for everyone who has done some judging – hopefully I can encourage a few more people to participate so we can get some reliable results!
Looking forward to this paper. My hypothesis is that they have got harder over time with the bulk of solvers being aficionados of the puzzles rather than the man on the Clapham omnibus.
I do wonder if it’s a bit like when you pick up an old trivial pursuit where references that were straightforward 20 or 30 years ago have become forgotten with time. I rarely see actor=tree anymore probably because the majority of people now don’t know who he is.
I will sign up and contribute.
Fabulous, much appreciated!
I have a few thoughts on this:
– the number of clues is a lot. I got half way through but stopped paying as much focus potentially invalidating answers
– some of the earlier clues are likely harder now for people who don’t have the same current affairs knowledge (e.g. of stuff that was present time for earlier clues) which could mean a clue which would have been easy then, is harder now
– rather than asking people who is harder, what about testing people, getting them to complete the answers instead
I tended to mark the non-cryptic, definition-only clues as harder, as others have commented.
I don’t think comparison between the non-cryptic and cryptic clues is meaningful.
Some of the clues — I am guessing earlier ones — seem to me unsound, or poorly constructed in some other way. In some sense these are more difficult than well-constructed clues, because it’s harder to be sure you’ve found the right answer, but it’s not at all the same sort of “more difficult” as a clue that’s more intricately constructed or involves more obscure words. It wouldn’t be very surprising if the clues had become sounded but more intricate over time, and those two effects may be hard to disentangle.
I also question what will actually be able to be deduced from this study, other than something about the willingness of people on this site to take part! My thoughts:
* Definitely too many questions. I left it a while between sessions but got bored and went quicker as time went on so later ones are not as considered as earlier ones
* I made a different judgement on the straight clues and treated them as de facto simpler than anything cryptic
*I generally thought clues using older knowledge or terms no longer in use were harder, but they would not have been at the time – is a “Whig history” approach really fair? I don’t think so.
*There is a timer which kept saying things like “18s” – made me worry I was being timed so was reluctant to take a break in case it invalidated something when I spent 2 hours away from it apparently cogitating but actually cooking dinner!
* “harder” is definitely subjective – I don’t get on with double definitions so rate them harder than other types of clue. There is a risk of similar bimodal distributions of personal opinion which means an actual ranking is impossible, or reflects the biases of the sample.
* How did others treat the clues which involved anagrams with no anagrind?
Whilst I understand the principles of comparative judgement, and have had to use it in marking (and found it helpful), I am not convinced the clue selection was ideal.
I attempted to evaluate old clues needing old knowledge on the assumption that old and new knowledge were on an equal footing even if I personally have more of the latter than the former. I can’t think what the timer could usefully do (it can’t say “this clue took longer to solve than that one” because it’s a single timer for a pair of clues being compared; it can’t say “these clues were harder to decide between than those” because a long time could mean a difficult decision or one difficult clue) so I’m guessing it’s just a thing that the site provides for other reasons and not directly relevant here. I think things like an individual solver’s dislike of double definitions will average out OK across all the solvers.
I assumed that the unindicated anagrams were par for the course back when they were set and generally felt that they were somewhat but not hugely more difficult than they would have been with proper anagrinds.
I had the feeling that all the clues were quite straightforward ones, but it’s hard to be sure how much of that was just because I had all the solutions in front of me alongside the clues. In particular it didn’t feel as if there was all that much variation in difficulty, aside from problematic things like cryptic versus non-cryptic.
I definitely expect the results to be difficult to interpret with much confidence.
Hello Chris,
I’ve noticed several mistakes in the suggested answers:
Aneriod, rather than aneroid
Marshamallow rather than marshmallow.
Am I missing something, or are there typos?
Best wishes,
Neil
Ah, I transcribed the clues so I apologise for any errors. Thank you for noting them.
The data is looking really promising! Ideally I need some more participants – the more data we get the more reliable the results. Thank you all so much for your contributions!
I can’t see the acknowledgement email you sent when I signed up. The message is “Failed to retrieve message content”. Doesn’t stop me participating but does it contain something I need to know?
i am not clear whether the experiment is asking which clues I can’t readily solve or which I think are objectively harder. A quotation or a simple anagram doesn’t require any knowledge of the conventions of cryptic crosswords, so is solvable by anybody. I think that must make them “easier”. clues. On the other hand, while I have solved a lot of cryptic crosswords, and will rapidly get the answer to many clues that will totally bewilder a first-time solver, if I don’t know a quote or can’t think of a synonym, there’s no way to work it out. I am sure that clues have got more complicated, but that often makes them easier for the experienced solver who knows the rules.
Chris, I don’t know whether you’ve been checking annotations added by users but I noted several mistakes including some not yet mentioned above. I doubt I’m the only participant to have done so.
I’ve done this, marking all non-cryptic clues as harder regardless of what the other clue was like. I imagine they would have been easier to solve in situ with crossers to help.
Looking at some of the earlier comments about difficulty progression throughout the week for the Guardian cryptics – perhaps if you’re a more experienced solver, they’re all equally straightforward. But for less clued-up folks like my partner Evilrooster and myself, we definitely see the difficulty gradient. Mondays and Tuesdays are notably faster for us to solve than Thursdays and Fridays. We often find the Saturday prize crossword a little easier than the Friday.
One item that is emerging as very challenging is this one from Paul in 2003, crossword 23023:
God, Lord of the Flies’s William Ing? (4).
The solution is THOR.
Can anyone parse this, or could it be an error in the book?
God = Thor
Author minus gold(au) = Thor
William (gold) Ing = Author minus gold .
I remember solving this a long time ago .
It looks like William Golding, the author, missing gold (au)
I completed this and fairly enjoyed it, but I think you should have made the effort to tidy the data set (e.g. remove non-cryptic clues, check spellings etc) before asking us to commit so much time to your study.
To be honest, I thought about doing this, but the somewhat negative comments on here have completely put me off
Huge thanks to Roz @28 for the parsing!
I was another who marked the single-definitions as harder, because I find those harder in puzzles. Also tended to mark the more British ones more difficult (a lot of overlap with the single definitions.) After a couple I started making allowances for the unindicated anagrams (or indicated only with ?) as I figured those were conventional at the time. And like everyone else found “Thor” the hardest!
Noticed another typo in the given answers, NEMISIS for NEMESIS
I haven’t yet attempted this exercise, having just come across it, but I wonder if the data being gathered can’t be combined with something I started a few years ago, but is now on the back burner, to provide an automatic puzzle difficulty number for any cryptic.
While I was still working (in AI) I wrote a program to solve cryptic clues. Internal to the program is a recipe, if you like, for the solution. Extractable from this are so-called “features”, i.e. devices used (anagram, hidden, charade etc) and also the distance apart of “synonyms” in a semantic metric space. If you apply machine learning to the features in a clue and a human-provided difficulty, for hundreds of clues, you get a model. From this you can generate a difficulty score by analyzing any new, unseen clue. Average these in a puzzle and you get the puzzle’s expected difficulty. All automatically.
Would this be of interest?
An interesting endeavour and I’ve completed the survey. I share the views of Roz about the convergence of puzzles towards the “medium” difficulty level. It’s happened because media outlets (such as the Guardian) use feedback from readers to inform what they choose to publish in future. Easy puzzles attract lots of negative user comment (“harrumph, it’s supposed to be a challenge”) as do difficult puzzles (“wilfully obscure and pretentious, how can anyone enjoy this torture”) so there’s a bias towards commissioning and publishing stuff in the middle. It’s regrettable but, I suppose, understandable.
Mandarin@35 , I sometimes wonder if this site is part of the reason . Instant feedback for setters , they must read it and see the praise given to puzzles of medium standard .
Firstly, I completed the survey on my phone and at no point was I asked to record my voice. On mobile, you will probably need to rotate your screen to horizontal.
And oh my goodness, there seems to be a lot of overthinking in these comments. We’re not being asked to quibble about parsing, or rate our enjoyment or judgment of the clues; simply which is harder, A or B.
If you’re taking longer than 10-15 seconds to answer each one, you’re probably analysing it beyond what the researcher wants to know.
I came to the conclusion that we are comparing apples and pears.
Solvability v cultural literacy
Quotations / General Knowledge
(Arch of Titus ; all our yesterdays) are part of cultural literacy and cannot be solved without that knowledge. They may be guessed from crossers etc.
Even though I know most of these I marked them harder than clues where the answer is derivable from the clue.
I’ve recently been doing NYT Connections. It’s a salutary lesson in cultural literacy: I am frequently stumped by NBA managers; Radio Hall of Fame Comedians; brands of dish soap; and US slang for things you put on a barbecue (etc) which prove unfathomable to me but would be simple across the pond.
However, I don’t think those sets can be classed as ‘harder.’ Maybe ‘harder for non-US players…’
So I think we need to distinguish between solvability and easiness.
The Golding/Thor clue is very hard but solvable. The Arch of Titus was easy for me, but is not derivable from the clue. Hard? Or hard for solvers without classics in their arsenal.
It’s like rating pub quiz questions. Too hard? Too obscure? For whom? Is The Arch of Titus now obscure… and was it when it was set? I suspect it was no more obscure when set than ‘Hotel-based Sitcom starring John Cleese (6,6)’. I maintain this is underivable. However ‘Hotel-based sitcom involving flowery twats (6,6)’ would be solvable.
Anyway, I enjoyed the process.