Are Google & Web Scale Discovery services — Low skill cap, Low performance cap tools?
Where I speculate about how the concepts of skill caps and performance caps and the views that librarians implictly hold on them in these two dimensions, affect how they engage with them.
I recently started playing Hearthstones a digital collectable card game and like most games in this genre, certain combinations of cards are quickly identified as “metagame defining” and dominate the game. These half a dozen decks are commonly referred to as “Tier 1” or “Tier 2” decks and are quickly copied and played by all players regardless of skill level.
The interesting thing is you will quickly notice in the data that not all these top tier 1 or 2 decks are equally effective (in term of win rates) at different skill levels of the game.
For example as I write this the straight forward to play ‘Token Druid” deck which has pretty much only a few lines of optimal play regardless of who you face has roughly the same win rate regardless of whether it is played at Legend ranks(Highest ranked players — roughly top 1% ), Rank 1–4 (next strongest players), Rank 5–10 (intermediate), Rank10–20 etc.
This implies this deck is pretty easy to play such that even not very skilled players can get maximum value from the deck.
As such Hearthstone players refer to this deck as a deck that has a “low skill cap”. In other words, it does not take much skill to pilot the deck to it’s full effectiveness.
On the other hand, “Conjuring Mage” decks which requires many skill testing choices at every turn and plays differently depending on who you face has a horrible (below 50%) winrate for lower ranked players but the win rate increases as you look at the higher ranked players until at Legend (the top elite players), it is actually one of the three strongest decks.
Conjuring mages deck are of course the opposite to the earlier Token druid deck and here the player’s skill makes a big difference to playing the deck with success. In other words this is a high skill cap deck and the range of skill displayed makes a big difference.
While skills may or may not affect how well a deck can be played depending on the skill cap, there is a maximum performance a deck can command. So for example “Token Druid” deck has roughly the same win rate across all the skill levels of player, but at the highest Legend level,the win rate isn’t that high compared to other decks (it is probably just within the top 10 decks), so that deck actually has what I call a relatively low performance cap.
In other words, a deck with a low performance cap means that even when played with the maximum level of skill required (ie up to the skill gap) it still can’t perform that well, while a deck with a high performance cap has the potential to perform very well, if played with maximum skill.
Applying skill caps and performance caps to search tools
So what this have to do with Web Scale Discovery services (e.g. Summon), Google or library databases?
Here’s my thesis. Search tools are similar to Hearthstone Decks and have skill caps (does the searcher skill with the search tool make a big difference?) as well as performance caps (even at the maximum level of searcher skill, does the search tool perform well?).
Knowing which categories they fall into, explains a lot about how librarians have historically responded to Google, Web Scale Discovery and has implications for Information Literacy.
Let’s use the 2x2 Grid matrix — a favourite in many consultant’s toolkit and consider the 4 extremes.
Ideally, we want to recommend and use search tools that are on the top left of the quadrant — Tools that have low skill cap (easy to use), yet potentially perform very well and avoid tools in the bottom right quadarant, crappy tools that take a ton of skill to use and yet don’t really perform well. In reality the top left quadrant of tools probably doesn’t exist hence I call them Unicorn tools and I’m sure any librarian can recall some crappy databases they would happily slot in the bottom right.
Conventional thinking would suggest there is a trade-off. There are some search tools that work pretty much at the same level over a pretty wide range of skills, though tend to have a pretty low performance gap though.
On the other hand, there are more powerful and complicated search tools that are fiddly to work with but if you master them, they can do amazingly.
How I see our search tools
I am of the view that Google/Google Scholar and our Web Scale Discovery tools e.g. Summon are low skill cap but low performance cap tools as well. (Note: Some may disagree above the low performance cap bit if it meets their needs, so to a degree this is subjective)
In other words, it doesn’t take a lot of training to get the maximium performance out of something like Summon but the best you can get out of it isn’t a lot.
On the other hand, some of our Library databases or things like Pubmed are much more difficult to use, but if you spend a lot of time mastering the tricks of the trade you can have very precise results (high performance cap).
I suspect there are quite a few academic librarians who would agree with me on this classification based on their behavior.
Surveys have shown academic librarians tend not to teach Web Scale Discovery in information literacy classes but prefer to spend more time teaching subject databases, A&I etc.
Of course some do this because they simply hate those tools, or think that there is no need to teach because it is so prominent on the library webpage that users will definitely will use it.
But I think many implictly reason that Web Scale Discovery service are by default easy to learn but there is just so much performance you can squeeze out of it so they don’t need to teach it much. Subject databases are the opposite of course.
If you agree with me, you are probably thinking , this is “Captain obvious” stuff. But I supect though there are some librarians who act like they do not agree. So let’s move Web Scale Discovery tools into the other 3 quadarants and see if any of the alternative positions in the 2x2 grid makes any sense.
Web Scale Library Discovery is wonderful!
I don’t really think many librarians would class Google or Web Scale discovery tools like Summon as low skill cap, high performance. The only people would be our discovery vendors and perhaps some users who think it fits their needs completely. I can see a meme about Discovery services that go “How our vendors see themselves…” . But let’s move on.
Web Scale Discovery tools are rubbish!
Under “how some angry librarians see Web Scale discovery”, we have this lower right quadrant.
I highly suspect judging from reactions of librarians to Summon recorded when it was still brand new (see for example “Culture Shock: Librarians’ Response to Web Scale Search”), a significant number of librarians then (or maybe even now?) thought web scale discovery tools were total rubbish.
As I chronicled in the blog post “Why nested boolean may not work as well as they used to” and in others posts, librarians were finding the more they tried to transfer the skills and experiences they brought over from other databases (aka nested boolean), the worse results they got.
Add issues like uncertain coverage, relevancy issues for known item search (mostly due to library catalogue records with little metadata , short titles, no full text getting drowned out by full text articles) and it is no wonder there was so much resistance and it seemed the absolute performance was dismal.
Even today where things have improved quite a bit on the relevancy front, you can find academic librarians (sometimes but not always IL librarians) still gripping about how poor such tools are on Twitter…..
But let’s end with the last possibility where I may say something a bit radical.
Web Scale Discovery tools have the potential to be very good if you just know how/more
This is a school of thought I’ve sometimes encountered which argues that librarians were getting not that good results with Summon not because it was fundmentally low performance cap or low skill cap but rather because the librarians did not have high enough skill.
For example in response to my arguement that Web Scale Discovery actually performs worse with nested boolean, they would argue this only proves the librarians were low skilled to even try using it and not that the tool is fundementally low skill cap.
Such librarians argue that if they knew more about Web Scale Discovery tools like Summon (and maybe Google) — though I’m not sure what exactly they want to know, perhaps if they were told exactly how the relevancy algothrim works , they could improve their search performance with the tool to amazing levels.
In other words they think these tools are high skill cap, high performance cap, but we just don’t have the skill to see that.
I admit to being very doubtful of such a claim and have debated this with some Information literacy librarians on Twitter.
I can only share my personal experience as someone who has implemented Summon & Primo in two different institutions and spent a big part of his librarianship career from 2011–2015 seriously tweaking, thinking and writing about Web Scale Discovery and perhaps gained some reputation for that (e.g. I almost fell off my chair when I saw that Marshall Breeding once cited me twice in a NISO whitepaper on discovery) .
So I would immodestly consider myself above average in understanding on how such tools work compared to most librarians yet I have not noticed that I’m that much better at searching than a typical librarian or researcher. Sure, I know some tricks like it is a good idea to login before searching because some results only appear when signed in, or that for finding catalogue records with minimal metadata you should really restrict to library catalogue only (often labelled “Books & more”), but these are tips I could share in 10 minutes really or you could figure out yourself with some use.
In fact, I know quite a bit more e.g. Why some links break, how long it takes to update when you activate a new database, how fast typically a new indexed item from a database is added, odd quirks due to the use Ebsco API in Primo but not a lot of it actually impacts searching performance IMHO.
Heck in Primo I can even see some of the weights for field boosting, synonyms etc and adjust them, but it has never really helped me. I highly suspect search tools are just so complicated their workings are effectively opaque to even the people who build-it, much less librarians.
But perhaps as I said the things I do know aren’t exactly useful aka I’m not high skilled enough? Maybe only someone who knew the exact relevancy algo and not just weights and do the math in his head could show the true power of Summon/Primo? I jest of course.
Can we measure this?
While I argued using logic why Boolean was no longer has effective in this day of full text search (as opposed to purely title+abstract days, someone actually did a study that compared simple boolean with natural language search in a variety of databases such as JSTOR, Google Scholar, Pubmed, Scopus, Web of Science, Lexis Nexis, Academic Search premier and showed that there was no difference in search relevancy results!
In the same vein, can we test empirically if a tool is low skill or high skill cap? Let’s say we define performance as precision and recall. Search skill is measured by time using the search tool and/or length of time taught by someone knowledgable in the tool.
We take someone and let them gain some reasonable level of skill after trying out for say 30 minutes (take this as the baseline low skill) and measure their performance.
We then either give them intense training in the tool, or let them try it for months and come back after that to measure peformance again. If their performance is about the same do we then say “ yes the tool is low skill cap” ?
Would that suffice to show a tool is low skill cap? Or would someone still insist it is because the person still wasn’t high skilled enough?
But what about comparing relative performance on Summon vs say Pubmed? If after training we find search performance doesn’t increase as much in the former while it does a lot in the later, surely we can say Pubmed is relatively higher skill cap.
Conclusion
I have a suspicion, many librarians just do not like the idea of a low skill cap tool in principle, because how are we going to show value of our hard earned professional skill? If it took only a little time or effort for the base searcher to learn to perform as well as anyone what need do we have to exist?
Could we be in denial and teach in Information Literacy classes tons of things that don’t really help that much because we think knowing more will always help? (e.g. Teaching all the google operators when in actuality just a few like site or intitle operator are really useful).
In other words, might we be biased to see high skill cap , high performance cap tools everywhere?
Unfortunately I don’t think there is a logical reason why low skill cap search caps tools can’t exist. Arguably the existence of such tools are a net good to society as we want to make things as easy as possible right even at the cost of eroding our value.
As of now, we can comfort ourselves with the thought that even if low skill cap tools like Google exist, they are low performance (though even that is an assumption….)
But will we be in denial if the great hope of our age, Machine learning brings us both low skill cap yet high performance cap tools? Or are we going to insist such tools can’t really exist and that they aren’t really high performance….
