According to Microsoft, 50% of online searches could be carried out via a conversational agent (Chat BOT) based on generative AI. But what if Internet users are not interested in this value proposition? Everything depends on Internet users’ willingness to change how they search for information online. But the user experience proposed by Google has been ingrained in us for almost 20 years. So, I wonder if Internet users will change their habits and massively adopt the new Bing or ChatGPT. In this article, I develop my position and bring some scientific elements to calm the buzz around ChatGPT.
If you only have 30 seconds
Conversational agents using generative AI (ChatGPT, new Bing) do not represent a credible alternative to the hegemony of classical search engines because they:
- require a natural language formulation that today represents only 10.58% of queries and 1.73% of search volume.
- do not deliver results that are necessarily better than those delivered by a classic search engine (96% of queries are satisfied with the first link clicked)
- require more time to formulate the question and read the answer
Finally, the massive use of conversational agents instead of traditional search presupposes that Internet users will change their habits. However, these habits have been in place for more than 20 years and are a major obstacle to adoption.
2 very different search experiences
The new Bing is equipped with a conversational interface that embeds GPT-3. This allows the user to formulate complex queries. Having already access to the beta version, here is an example of a command line that allows one to exploit the power of this artificial intelligence:
Propose 3 destinations from Brussels on a city trip for 1 weekend and a price lower than 100€ per person.
As you can see, the request is quite elaborate. The more detailed it is, the more precise your results will be.
Did you know that of the 100 most used queries in the United States, 58 contain only one word? In contrast, a query on a classic search engine (Google) is usually very short. The genius of Google’s algorithm is to interpret these queries and transform them into something usable.
Of the top 100 Google queries in the US, 58 contain only one word.
A query on ChatGPT, therefore, requires a special effort from the user:
- first, an effort of reflection to structure the query
- a writing effort to explain in natural language the expected result
The “return on investment” of the effort to use GPT-3 will only be positive in certain contexts. In others, Google will continue to dominate.
There are different ways to search for answers
The way we search for information has been researched by Marcos et al. (2012). They showed that there are broadly 5 ways to search for information. Two approaches have particularly high success rates:
- the user writes his query, consults the 1st result, and does not return to the search engine (96% success rate)
- the user writes his query, consults the first result, returns to the search engine, and consults a second result (81% success rate)
To find information, Internet users use combinations of keywords. Formulating a question in natural language remains an exception.
Formulating questions remains the exception
I analyzed 178.04 million queries in the United States, corresponding to a volume of 5.47 billion searches on Google. Questions represent 10.58% of queries but only 1.73% of search volume.
What motivates users to massively adopt generative AI at the expense of traditional keyword research? However, research shows there isn’t one since the success rate is high from the first link consulted (96%). The breakthrough of conversational agents is conditioned (beyond the current buzz effect) to dissatisfaction (a “customer plain”).
Questions represent 10.58% of queries but only 1.73% of search volume.
Finding information is an iterative exercise
There is another barrier that conversational agents will have to overcome to compete with traditional search engines. This barrier is our natural way of searching for information. In this domain, eye tracking is useful for understanding human behavior. Strzelecki (2020) notes that 56 such research studies have been conducted since the beginning of the century.
For example, here are the results published in the research by Papoutsaki et al. (2017). You can see that the internet user’s gaze is focused on the first organic results. We also find the typical inverted “L” shape. The Internet user spends more time on the 1st result and reads fewer and fewer meta descriptions as his gaze moves down the page. Note in passing the little attention dedicated to sponsored results.
As I explained in this article, searching for information on the internet is complex to understand. While scanning the page, the brain can pick up weak signals and snippets of information. They contribute in an unconscious way to the understanding of the subject. However, a “heatmap” does not allow you to understand the sequence in which the user looks at the answers provided by the search engine.
Lorigo et al. (2008) researched such sequences (see illustration above). They showed that our eyes go back and forth between rank 1 and 2 results and even to rank 5. This is visible on the graph they published (see below). The letters A, B, … J represent the rank of the different search results (the SERP). A is rank 1, B is rank 2, and so on … Notice the different sizes of the circles corresponding to the different letters. The bigger the circle, the more time spent staring at it. This means that in this experiment, the eyes are more fixed on the result of rank 2 (B) than on rank 1 (A).
No, ChatGPT and the new Bing will not replace Google
Everyone got a little excited about ChatGPT first and the new Bing second. Microsoft’s predictions seem unrealistic to me. It seems unlikely that generative AI will replace 50% of the searches currently done mostly on Google.
There are several reasons for this.
No “customer pain”
Generative AI does not solve a user problem (“customer pain”) in finding information. Google provides excellent service (96% success with the first result) with minimal user effort (so much so that 56 of the first 100 searches contain only one word).
More user effort to use generative AI
Writing your query in natural language and reading the answers is much more tedious. Eye-tracking research shows that the user scans the first 5 results and follows an iterative process. With generative AI, moving from one result to another is impossible to choose the best match for the search objective. Therefore, I have already highlighted a risk of bias in this article where ChatGPT returned incomplete or even false data.
Since the birth of search engines, users have been used to entering combinations of keywords. These habits are ingrained in us, and changing them will be difficult for most Internet users. It is likely that users with very specific needs will concede a higher search effort to obtain results that Google cannot provide. In the end, this could be the biggest obstacle to the use of ChatGPT-type conversational agents.
In conclusion, I see several obstacles to the massive adoption of ChatGPT-type conversational agents. They are not close to replacing Google. However, they offer interesting possibilities to generate ideas. They should not be neglected but used for what they are. They are tools that juxtapose words from which meaning, sometimes surprising, can emerge.
Posted in Data & IT.