Tuesday, December 5, 2023

Large Language Model Adoption Metrics Will be Difficult

Large language model adoption and use might very well surpass all prior examples of internet app/site usage and engagement, in large part because virtually all existing sites and apps will embed LLM functionality into the fabric of their operations--and possibly into the core of their operations in some cases. 


In other words, the possible fundamental difference between LLM adoption and all other major successful apps and sites is that LLM can be immediately deployed into the operations of all sites, almost immediately. 


People might be using LLMs and not be aware of it. For example, we might estimate that natural language interfaces powered by AI already range between 20 percent and 40 percent of smartphone users. That is a proxy for daily active use, in all likelihood. 


So the issue will be how fast LLM is likewise incorporated into natural language interfaces for smartphone apps, for example. 


All of which points out the likely difference in metrics we might have to develop for use of LLMs. It has been relatively simple to track daily- or monthly active users for specific apps or sites. It will be harder to track engagement and usage with LLMs that are embedded into the fabric of other experiences. 


Some suppliers will likely use metrics such as application program interface calls or event logging as ways of illustrating usage or engagement. But most of the measurements are likely to be rather indirect. 


Examples might be changes (ncreases) in specific actions, including number of searches, completed forms, voice interactions) or time spent on relevant pages. 


All that noted, it might still take some time for any single LLM (there will be multiple contestants) to reach 10-percent adoption or usage levels. 


Taking nothing away from the breathtaking eruption of ChatGPT-3, not even ChatGPT-3 really emerged from “nowhere.” about three years elapsed between ChatGPT-1 and the popularization of ChatGPT-3. 


And as important as large language models might be, we likely are quite some years away from a point that even 10 percent of internet users avail themselves of an LLM on a daily basis. In fact, based on history, it could take five to 10 years for any single LLM to reach the level of 10-percent daily active users. 


The caveat is that most of the successful early apps or websites provided one main value: search, e-commerce, social networking or entertainment. LLMs are likely to be embedded into multiple functions for any business or consumer, and might happen “in the background, so users might have no idea they are “using” features of an LLM. 


So the adoption curve, and the time to reach 10-percent usage, might be shortened, as the cumulative use of LLMs will occur across a potentially large array of use cases, apps and website interactions, including customer service, search, e-commerce, natural language queries of any sort, any recommendations or queries. 


So all past experience with successful apps and sites might not be predictive. To the extent that LLMs underpin interactions and usage with virtually all major apps and sites, “adoption” might not be the relevant metric. 


Instead, some measure of indirect usage will probably be more important. 


source: Tooltester


That is not an unusual time frame, even if many make much of the initial ChatGPT attainment of one million users, total. The most-popular apps and sites launched since 2000 have generally required three to five years to reach 10 percent usage by the internet population. 


Ignoring “sampling” or “novelty” behavior, all large language models have some ways to go to reach 10-percent levels of regular use by internet users. There are 5.3 billion internet users globally. One million users barely registers. 


Any large language model would have to hit a level of about 53 million regular users to reach one-percent adoption. 


App/Website

Category

Launch Date

10% Users Reached

Time to Reach 10% (Years)

Facebook

Social Media

Feb 2004

Sept 2009

5.75

WhatsApp

Messaging

Jan 2009

Feb 2014

5

Messenger

Messaging

Aug 2011

Apr 2016

4.67

Instagram

Social Media

Oct 2010

Jun 2013

2.83

TikTok

Social Media

Sept 2016

Nov 2020

4.17

Amazon

E-commerce

July 1995

Jun 2008

12.92

eBay

E-commerce

Sept 1995

Dec 2000

5.33

Alibaba

E-commerce

Apr 1999

Nov 2013

14.67

Google Search

Search

Jan 1998

Dec 2002

4.83

Bing

Search

Jun 2009

Jul 2012

3.17

Baidu

Search

Jan 2000

Dec 2008

8.83

YouTube

Video Streaming

Feb 2005

Jul 2011

6.5

Netflix

Video Streaming

Jan 1997

Sep 2014

17.67

Disney+

Video Streaming

Nov 2019

Nov 2022

3

Spotify

Music Streaming

Oct 2008

May 2015

6.67

Apple Music

Music Streaming

Jun 2015

May 2020

5


Reaching a level of 10-percent adoption could take a while. Consider that it took Amazon nearly 25 years to reach a level of 10-percent DAUs. It took Amazon 13 years to rach a level of 10-percent monthly active users. 


Granted, that might be considered an outlier. But many other popular and successful apps still required eight to nine years to reach a 10-percent DAU level of usage, and a decade often was required to reach a level of 10-percent DAU. 


App/Website

Category

Launch Date

10% DAU Reached

Time to Reach 10% DAU (Years)

10% MAU Reached

Time to Reach 10% MAU (Years)

Facebook

Social Media

Feb 2004

Apr 2012

8.17

Sept 2009

5.75

WhatsApp

Messaging

Jan 2009

Feb 2020

11

Feb 2014

5

Messenger

Messaging

Aug 2011

Jul 2023

11.83

Apr 2016

4.67

Instagram

Social Media

Oct 2010

May 2018

7.5

Jun 2013

2.83

TikTok

Social Media

Sept 2016

Nov 2020

4.17

Nov 2020

4.17

Amazon

E-commerce

July 1995

Jun 2020

24.83

Jun 2008

12.92

eBay

E-commerce

Sept 1995

Dec 2016

21

Dec 2000

5.33

Alibaba

E-commerce

Apr 1999

Dec 2021

22.67

Nov 2013

14.67

Google Search

Search

Jan 1998

Dec 2007

9.83

Dec 2002

4.83

Bing

Search

Jun 2009

Aug 2018

9.17

Jul 2012

3.17

Baidu

Search

Jan 2000

Dec 2011

11.83

Dec 2008

8.83


Likewise, there is a difference between “daily active users ” and other measurements of usage, including:


  • New users

  • Retained users

  • Returning users / Resurrected users

  • Churned users

  • Cohort (useful for churn analysis)

  • Monthly Active users 

  • Daily Active users 


All these are ways of measuring engagement or usage. It is too early to cite accurate “daily active users” for any large language model such as ChatGPT, Bard or others. 


Metric

Description

Value (Example)

User Acquisition

Measures how users find and install the app or visit the website.

10,000 downloads per month, 500 organic website visits per day.

Active Users

Measures the number of users who interact with the app or website within a given timeframe.

5,000 daily active users (DAUs), 10,000 monthly active users (MAUs).

Engagement

Measures how users interact with the app or website.

30 minutes average session duration, 5 pages viewed per session, 20% app retention rate (returning users).

Traffic Sources

Measures where users come from to find the app or website.

40% organic search, 30% social media, 20% direct traffic (bookmarks, typed URL).

Conversion Rate

Measures the percentage of users who complete a desired action (e.g., purchase, sign-up).

2% conversion rate for free trial sign-ups, 5% conversion rate for product purchases.

User Flow

Measures the path users take through the app or website.

70% of users land on the homepage, 20% go directly to a product page, 10% bounce from the landing page.

Device Usage

Measures the types of devices users access the app or website from.

60% mobile, 30% desktop, 10% tablet.

Location Data

Measures where users are located when accessing the app or website.

50% users from the United States, 20% from Europe, 15% from Asia.

User Feedback

Measures user sentiment and satisfaction through surveys, reviews, and support tickets.

4.5-star average rating on app store, 80% positive sentiment in survey responses.


source: Tooltester


Monday, December 4, 2023

AI Wearables: Will Any Emerge Eventually as Leaders?

Where AI goes next, and where AI devices could go next, is a huge issue at the moment. Sometimes such devices are described as “wearables” that some believe could be a replacement for the smartphone. Almost all rely on natural language processing for user input. 


Tab is a wearable device that "ingests the context" of your daily life by listening in to all of your conversations and using artificial intelligence to function as an assistant. By actively monitoring user conversations, Tab intends to provide  instant access to a vast reservoir of person-specific knowledge and provide concise, relevant summaries to user inquiries.


Humane is developing an “AI Pin” that features a projector to allow its simple user interface to appear on a hand or other nearby surface.


Rewind.ai has developed a neck-worn pendant that's designed to record conversations and transfer them securely to a smartphone. Its AI software sorts through and gleans insights from that mass of audio info, creating a sort of searchable database.


Meta smart glasses now can use an AI chatbot to interact. 


And then there are efforts by Jony Ive, former lead Apple designer, said to be working with  SoftBank and OpenAI for an AI device of some sort. 


Those of you familiar with the development of computing appliances and devices know that the “early leaders” often are not the ultimate winners of big device markets. That is likely to happen for wearable AI devices as well.


ITU Releases Framework Document for 6G

The International Telecommunications Union has published some details of its framework for 6G  networks, and many of the objectives are what you’d expect. Compared to 5G, 6G will support higher speeds, lower latency, be more spectrally efficient, energy efficient, feature artificial intelligence and sensing. 


  • Peak data rates of 50 Gbps, 100 Gbps or 200 Gbps

  • User experienced data rates of 300 Mbps and 500 Mbps

  • Spectrum efficiency 1.5 and 3 times greater than that of IMT-2020 (5G)

  • Area traffic capacity of 30 Mbps/m2 and 50 Mbit/s/m2

  • Connection Density could be 106 to 108 devices/km2

  •  Mobility Maximum speed, at which a defined QoS and seamless transfer between radio nodes could be 500 – 1 000 km/h

  • latency (over the air interface) could be 0.1 – 1 ms.


In addition to those quantitative metrics, there are the expected qualitative benefits. The framework document includes talk of ubiquitous computing, ubiquitous intelligence, immersive multimedia, digital twins, virtual worlds, smart industrial apps, digital health, ubiquitous connectivity and sensing integration.  


As with prior generations (3G, 4G, 5G), many of those qualitative outcomes might be delayed or available only in rudimentary form. 


There are good reasons why mobile operators are much more concerned about “application revenue” than home broadband providers. Mobile operators always are in the “applications” business, where home broadband providers are only in the “internet access” business. 


In other words, mobile operators derive significant revenue from their own voice and messaging applications. ISPs providing home broadband mostly make money from subscriptions providing the internet access function. Most of their other revenue is related to the access function, such as equipment rentals or install fees. 


Revenue Source

Home Broadband Network

Mobile Operator Revenue

Subscription Fees

70.00%

40.00%

Voice Services

0.00%

25.00%

Data Services

0.00%

20.00%

Equipment Rental Fees

10.00%

0.00%

Installation Fees

5.00%

0.00%

Roaming Fees

0.00%

5.00%

Other Revenue

15.00%

10.00%


Also, mobile operators are dependent on government regulators to authorize additional spectrum, so there is a political underpinning to arguing that additional spectrum will lead to public advantages beyond “faster speeds.”


Still, in large part, the success of  built-in “app capabilities” will likely be hard to predict. Since mobile network value includes a mix of “apps” (largely voice and messaging) and “internet access” (dumb pipe access), much--if not most--of the value comes from the “internet access” function. 


As for home broadband networks, faster speeds are a continual requirement, as are support for carrier voice and messaging. Beyond those essential functions, it always is difficult to say how much other value can be reaped directly by mobile operators in the “apps” space and beyond connectivity itself. 


6G might not be so different from earlier generations in that regard.


Open Source Now is Ubiquitous in Enterprise Computing

Though there was a time, decades ago, when open source might not have been widely trusted and used in enterprise settings, that time has gone, as virtually all enterprises use open source code. 


Black Duck Software in 2020, found that open source software is used in 98 percent of enterprises. The study also found that open source software accounts for an average of 60 percent of the code base in enterprise applications.


The Linux Foundation in 2021 argued that open source software generates $6 trillion in economic value each year, and supported 16 million jobs worldwide.


Most studies do suggest substantial use of open source by enterprises. 


Study

Usage

Date of Publication

Publisher

Black Duck Software

70%

2022

Black Duck Software

North Bridge

66%

2022

North Bridge

Linux Foundation

80%

2022

Linux Foundation

Opensource.com

50%

2022

Opensource.com

OSS Impact Report 2023

65%

2023

The Linux Foundation

Open Source Software: Economic Impact and Benefits

60%

2022

BSA

The Value of Open Source: A Study by the Open Source Initiative

55%

2021

OSI

Open Source Software: The Real Revolution

50%

2020

O'Reilly Media


There might always be some tension between a community-driven open source software initiative and vendor-led or vendor-organized open source projects, but there arguably also are advantages, including less risk of sudden abandonment, greater resource commitments and more direction.  


On the other hand, concerns might be raised about vendor influence or control, sustainability if the vendor loses interest (though that can be an issue for any open source initiative), community engagement or transparency. 


Consider Android (operating system), Kubernetes (container orchestration platform), Azure DevOps (DevOps services), TensorFlow (machine learning platform), .NET (cross-platform development framework), GitHub (hosting service), Cloudant (NoSQL database, OpenShift (container platform), Hyperledger Fabric (open-source blockchain), Red Hat Enterprise Linux, OpenShift (container platform) or JBoss EAP (application server), Prometheus (monitoring and alerting toolkit), OpenTelemetry (instrumentation framework), Jaeger (distributed tracing system), Apache Spark (distributed data processing framework), PyTorch (machine learning library), React Native (JavaScript library for building user interfaces for mobile appsGrafana (data visualization platform) or Apache Kafka (distributed streaming platform).


Will AI Fuel a Huge "Services into Products" Shift?

As content streaming has disrupted music, is disrupting video and television, so might AI potentially disrupt industry leaders ranging from ...