Using Lens.org to kick start your literature review — a search strategy for finding review papers, systematic reviews and meta-analysis
TL;DR — This articles shows you a reliable search strategy that you can use in Lens.org to find meta-analysis, systematic reviews, review articles of topics you are interested in regardless of your discipline to help kick start your research. I also do a comparison of this search strategy in Lens.org against Pubmed and show that this search strategy yields similar results, giving some assurance on the reliabilty of this method.
One of the techniques I teach new research students when starting a deep dive into a research area they are not familar with is to look out for review articles, systematic reviews, meta-analysis which provide an expert’s view of a given topic as well as providing a rich source of references to mine.
Many such reviews are also very well cited and as such one can also “go forward in time” and look at citations by newer papers to the reviews. If you are really into it, you can even do literature review/bibliometric mapping of references collected using tools like Dimensions, VOSviewer, Citespace or Citation Gecko.
But how do you find such reviews in the first place?
One obvious source is Google Scholar and you can search like this
<your topic> systematic review etc, but the lack of precision in search features in Google Scholar (e.g. you can only restrict matches to titles, but not by other specific fields like abstract, subject) which means beyond the first few results you might start getting irrelevant results.
As such I actually recommend you use Lens.org instead, if you do not have a discipline specific database to use.
In the example below I’m going to use the topic of creativity as an example.
Why Lens.org?
Firstly you would want to use something that covers as much ground as possible and Lens definitely fits the bill.
It is really huge, as it pools together scholarly data from sources including
a) Microsoft Academic (roughly 170 million records)
b) Crossref (roughly 100 million records)
c) Pubmed/PMC (roughly 29 million records)
d) Core (7 million records)
All in all after deduping there are close to 200 million records, while a lot of them may not be traditional Scholarly publications (book chapters, books, journal articles, conference proceedings) but you can safely say you will include most traditional published items and even some grey literature.
Most people including me have suspicions Google Scholar is still a bit bigger (bibliographic analysis studies in the past few years show Microsoft Academic is still smaller but close), but at this level it may not matter.
Secondly, Lens.org provides one of the most powerful search features you can find on the market and it’s totally free! I’ve talked about some of them here and here.
For the purposes of our exercise, the main thing we will need that Lens.org provides is not just the ability to do field searching but to construct long complicated Boolean strings in a text query editor.
In particular, Lens.org unlike Google Scholar allows us to search for our topic keywords in the Title/abstract/Subject, which greatly increases the precision of our search and avoids getting a flood of irrelevant results because the word was matched in the last page of full text for example.
While full text searching has it’s place , when we are searching for reviews of a general topic such as “creativity” or “innovation”, in general we only need to search in the most important fields namely title, abstract, subject. Searching within full-text is likely to lead to more false hits.
Lastly, string and text matching even restricted to specific fields is all well and good, but it can sometimes lead to misses (because the item uses a different string from what you searched for e.g. you searched for “Cars” but the paper used “Automobiles”) or false hits due to ambiguity in the term.
Unfortunately most subject fields you find in databases, journal sites are “author assigned keywords” , in other words authors can choose what subject term they want to assign without following any fixed subject system.
This obviously leads to a lot of inconsistency. Take the simple subject “meta-analysis” , while this concept is quite straight forward and there is generally no ambiguity of what it means, the fact that the subject terms are not controlled in author assigned subjects leads to authors populating the keyword field in databases like Lens, with terms from “meta-analysis”, “meta analysis”(without hyphen) or “metaanalysis”(one word). And that is assuming the author borthered to even assign a keyword for this!
Ideally, subjects should be controlled, where everyone agrees on rules on how and what subjects to assign. So for example in controlled vocabulory schemas, you might all decide to use the preferred form of “Meta-analysis” rather than all the other variant forms.
Assigning controlled subject heading is a lot of work (mostly by human indexers) of course, and while discipline specific ones like MESH (medical/life sciences), ERIC (education) exist, there is no one agreed schema across all subjects.
Given that there are easily 100 million articles out there, the chances of one cross discipline subject heading to rule them all seems bleak, but today one actually exists.
Microsoft Academic Graphs’s Field of Study is a ambitious attempt to index all scholarly material regardless of discipline. Microsoft being microsoft solves the problem by using ML/AI to automatically extract meaning from the text and assign subject headings.
As mentioned before these subject headings are controlled and beyond that the Machine is also able to construct a 6 level deep Hierachy of subject terms!
As you shall see the search query we construct will use this.
The search query in Lens.org
Once you are in Lens.org, you can do a simple search, then click on “Edit search” on the top right, then click on “Query Text editor”.
Copy and paste the canned search query below.
(title:(“creativity”)
OR abstract:(“creativity”)
OR keyword:(“creativity”)
OR field_of_study:(“creativity”)
OR mesh_term.mesh_heading:(“Creativity”))
AND (field_of_study:(“Systematic review”)
OR field_of_study:(“Review article”)
OR field_of_study:(“Meta-analysis”)
OR field_of_study:(“Bibliography”))
You dear astute reader probably see what I am doing with this. The first part of the query (in bold) simply searches for items where the topic keyword in the title/abstract,keyword and field of study. I even throw in MESH header.
The second part of the query (in italitics) from AND…. , combines this with a search that looks for items that are labelled by Microsoft’s ML/AI as systematic reviews, review article, Meta-analysis and bibliography
A warning about using controlled subject fields like MESH and Field of Research
As both field of study and MESH are controlled terms, you should check in advance you are using the right terms, otherwise you will get no results.
For Microsoft academic you can search in the topic browser while you can use the MESH browser for MESH.
I have done this in advance and verified the terms — “Systematic review”, “Review article”, “Meta-analysis”, “Bibliography” etc. are the right terms for Microsoft’s Field of study, so when you duplicate your search you just need to check for your topic’s terms (in this case creativity is both a term used in MESH and Microsoft academic’s field of study).
Do note that when looking up the controlled term for your topic, when searching in Lens.org, it only searches items that are matched to that level of topics and not the child topics.
Take “Library science” in Microsoft’s field of study.
Below that topic, there are a bunch of child topics such as “publishing”, “collection development”, “interlibrary loan” etc. But if you do a search for “Library Science” in Field of study you will not automatically get items tagged only with those child topics.
Currently there is no way I know using Lens.org or Microsoft academic’s interface to state that you want not just that topic but also any child topics, so you may consider entering the relevant child topics instead.
A more complicated/complete search
In fact, if you are up to it, you can even do a more complicated search that looks like this
(title:(“creativity”)
OR abstract:(“creativity”)
OR keyword:(“creativity”)
OR field_of_study:(“creativity”)
OR mesh_term.mesh_heading:(“Creativity”))
AND (field_of_study:(“Systematic review”)
OR field_of_study:(“Review article”)
OR field_of_study:(“Meta-analysis”)
OR field_of_study:(“Bibliography”)
OR keyword:(“systematic review”)
OR keyword:(“review”)
OR keyword:(“literature review”)
OR keyword:(“Meta-analysis”)
OR keyword:(“Meta analysis”)
OR keyword:(“Metaanalysis”)
OR keyword:(“bibliography”))
The search query is pretty much the same as the last one, except for the part in bold, which will allow you to match using the keyword field. Because the keyword field is uncontrolled, I try to catch multiple variants of the concept meta-analysis (a check in Lens shows these variants are actually used sometimes)
NOTE: If you are striving to maximise recall you can even go beyond relying on Field of study and Keywords and also look for string matches in title or Abstract
OR TITLE:(“systematic review”) OR ABSTRACT:(“systematic review”) …
but this makes the search string even longer which can cause trouble if your browser hits the URL character limit in which case the search query sent might be cut off and resulting in odd returns.
How good is this search? Comparing with Pubmed
If you are from the medical or life science fields, you are probably thinking this is a solved problem in your discipline.
Indeed it is, Pubmed now has publication types for meta-analysis, Systematic review, reviews etc. For some reason this isn’t currently imported into Lens.org though.
In any case, I decided to do a search using Microsoft academic fields and see the number of items it would come up with and compare it with the equalvant pubmed filter.
Below is what I used for in Lens.org
It’s a stripped down search that just uses Microsoft academic field of study fields to search for systematic reviews, review articles and meta-analysis.
This initially yield 149 results, but when I filtered it to Pubmed and PMC only I got 106 results.
In Pubmed. I ran the PubMed Clinical Queries, which yielded 111 results.
In case you are wondering the Pubmed Clinical Query uses a strategy of not just matching pub type but also doing matches with title to expand the recall for systematic reviews, meta-analysis and reviews.
Do the results in Lens.org hold up?
If this was a research paper, I would probably export both lists of hits and do a comparison (to see how much overlap) but as it is the closeness in the numbers hints to me that the strategy I constructed is not too far off from the Pubmed search strategy which is a pretty well refined strategy.
I eyeballed the 106 hits found by Lens (looking at titles and abstracts) that it states are in Pubmed and PMC and they all look reasonably relevant (but note I am not a medical librarian).
More interesting was to look at the 43 hits that Lens found but it claimed were not in Pubmed or PMC. Does this 43 hits give additional value?
Looking closely at some entries, some were actually in Pubmed, but just not marked in Lens.org as such (possibly due to some time lag in indexing) but there were some that were definitely not in Pubmed (checked by doing a title search).
Of the 43, Lens tells us most of the authors were from Institutions in China.
The author affiliations of the additional 43 items found by Lens.org
They are span a few years (so not all the differences were due to indexing time lag) and quite a few are not tagged as journal articles.
Interestingly enough, I find papers in top journals like Lancet (e.g this one) , Journal of the American College of Cardiology, Heart that are not indexed in Pubmed much less picked up by the search filter/screen.
Granted a lot look like this, which are posters at conferences?
Others seems to be theses (I can tell because the source is a http://etheses.lse.ac.uk/)
There are some that really look like outright journal articles, and even one in Chinese.
All in all, I’m not saying Lens.org is better than Pubmed for this work, or even you can replace Google Scholar with Lens.org for finding gray literature. A ton more serious research needs to be done on the potential of Lens.org for such work.
Rather the argument I’m trying to make is, if Lens.org can perform almost as well as Pubmed a specialised tool for this discipline, this gives us some assurance it probably it’s not too bad when we apply it to other disciplines…
Conclusion
All in all the number of results were surprisingly close using my search strategy in Lens v the search strategy in Pubmed (106 vs 111).
In fact, since the search strategy I tried relies heavily on field of study to detect reviews rather than using matches in titles, it shows Microsoft AI/ML is good enough to correctly tag a paper as such, almost as well as a more complicated strategy matching different strings in the title.
Using this strategy you should be able to quickly narrow down to the relevant overview papers. Throw in a search to cover dissertations or theses in your topic, and you will have a great starting point….
Next up science mapping with Dimensions/VOSViewer and Lens/Citespace,
