glass.ai expands its global coverage, enabling deep sector research across countries.

The Challenge of Sector Research

Sector research is hard. The task to identify all companies in a particular sector is difficult as companies describe themselves in different ways, the boundaries of sectors are unclear, and the data is scattered across many sources — company websites, social media, official business registers, news articles — all containing different parts of a puzzle that researchers need to piece together. Moreover, sometimes sector research projects must identify organisations in the wider ecosystem (e.g., government, universities, research centres), and companies that are active in the market but headquartered elsewhere. All this adds complexity to the task.

Sector research becomes even more difficult when researchers need to compare geographies and gather a deeper understanding of the activities, interactions and networks of the businesses. These may include data about employees and ‘signals’ like launching new products, receiving investment, winning customers, adopting new technologies, hiring for new skills, opening offices, partnering with suppliers, etc. All this information might be publicly available on the web, but it’s a challenge to collate it and it is constantly changing.

Until now governments, consultancies, trade associations, regulators, universities and corporates have typically conducted sector research in a manual way, using teams of researchers that spend a lot of time on search engines, company databases, social media, and other sources. It’s a painful, expensive and long manual effort that can still miss many relevant data.

At glass.ai we have developed a new capability that transforms how sector research is done, driven by proprietary AI that deep reads the web to understand companies and sectors. Today we are pleased to announce that we can do this on a global scale across countries and languages.

A New Global Sector Research Capability

The coverage of our research capability has expanded over the past few months and now we gather data and insights on approximately 75% of all businesses globally. This means we regularly deep crawl the websites and social media presence of millions of businesses, reading billions of web pages every month. In some geographies like the UK, Europe and North America, our coverage is around 95%, which means we’ve almost mapped these entire economies based on their web presence. For other countries like China, Japan and South Korea, the coverage is lower, but we are making great progress, reading deep web content about businesses in many countries and in different languages. Our objective is to cover 90% of all businesses in the global economy in 2023. This will make it an unparalleled capability for anyone who needs to do sector research.

We are achieving this with AI technologies that automate the research of sectors by machine-reading the websites, news and social media of millions of businesses globally. The glass.ai system can conduct deep sector mapping research at scale and at unprecedented speed with a deep understanding of the context in which words/phrases appear. Its power is akin to employing hundreds of researchers for a given sector study. Also, by continually scanning the web and updating our rich view of the businesses, the insights remain valuable over time.

Below we highlight some examples of recent sector projects that have benefited from the scale and depth of our AI research capability.

Case Study 1: Emerging Technology Sectors in the Middle East

An example of our research capability being applied across languages and countries is the recent work we’ve done for a global consulting firm in Dubai. The client wanted to map several technology sectors in different countries of the Middle East and benchmark the results against selected European, North American and Asian countries. For each company identified, they also wanted to gain a deeper understanding of the products and services, growth, innovation, and human capital. It was also important to distinguish between companies developing technologies vs. companies adopting emerging technologies.

We followed our tested methodology that reads the web at scale to identify the companies in each sector and country. Through various iterations and a combination of seed keywords/phrases in different languages, our AI capability read millions of web sources and produced a large database of companies. Once the baseline was identified, the next step was to understand the products and services offered by the companies. To do this, the glass.ai engine deep crawled the company websites in English, Arabic, and other languages to automatically gather details about the types of products and services. The system also read press articles and social media posts that mentioned the company’s offerings — doing all this with a deep understanding of the context in which different sources mentioned the companies and their technologies. This approach allowed us to go deeper into the products and services offered and we uncovered many examples of companies adopting these technologies across the Middle East.

At this stage, the system also categorised the companies based on whether the companies were local or from overseas but active in the market. To do this, the AI automatically identified the locations of offices (including head office) and locations of the personnel — this information was then triangulated to determine whether the company was local or an overseas company with a local presence. Additionally, companies that are active in the market but don’t have an office in the country were also identified by reading press releases, social media posts and people data. The last step for the glass.ai engine was to automatically discover a wide variety of growth signals: for example, any announcements of new products/services, increases in personnel count, number of patents filed/awarded, awards received, customer wins, new partnerships and investments received.

Once all the above insights were collated for the different sectors in the Middle East, the exercise was repeated for selected European, North American and Asian countries. This produced unprecedented, nuanced insight into Middle Eastern companies developing and adopting emerging technologies and comparisons with leading markets. This sector study would have been very difficult to complete without our AI research capability that deep reads web content about businesses globally.

Case Study 2: Knowledge Sectors in North America

In another recent sector project, we worked for a government agency in Canada that delivers economic development programs and initiatives for inward investment. The agency wanted to map all companies operating in their region and gain a deeper understanding of selected knowledge sectors. The research also had to understand the human capital part of the businesses and benchmark the region against other regions in North America and Europe. The insights would be used to position the region to foreign investors and corporates that wanted to expand in North America.

Like in most sector projects, we started the research by developing a taxonomy (in consultation with the client) to identify all the companies across sectors, with a particular interest in the knowledge sectors. This taxonomy detailed many technical terms (and nuances) that companies use to describe their activities, products and services. The glass.ai engine then used this taxonomy to crawl millions of Canadian company websites (both in English and French), social media sources and news, to identify text that may suggest a company may be in a particular sector. The system automatically classified thousands of companies in the region. Also, by machine-reading the company websites and social media and by cross-referencing the data with Canada’s official data registry, we validated the locations of the companies.

The first part of this project was essentially building a picture of the entire economy. The initial web crawl identified and categorised several thousand companies. The sheer number highlighted the need for an AI-driven approach and the limitations of manual sector mapping research.

Once the above baseline of all companies in the region was established, attention was focused to map the details of the Knowledge Economy. The aim was to identify and categorise companies into seven knowledge-intensive sectors. For each firm in the sector, we also needed details about the people employed (roles, skills) and other data points.

The client not only wanted to identify ‘core’ companies but also companies in the ‘wider ecosystem’. Core companies develop or provide solutions (products or services) in the key sectors and the ‘wider ecosystem’ refers to organisations actively involved in the sectors but not developing solutions e.g. higher education institutions and research sites.

glass.ai then ran a deeper crawl of the company websites and its social media profiles to find the data and insights related to human capital. The aim was to understand the total number of people employed in each sector, the roles they occupy, and the skills they possess for these roles. The number of employees in a company can be ascertained by reading company website pages, social media and press releases. The role of an employee in each company, say, a laboratory technician at a life sciences company, or the head of R&D, can also be gleaned from company websites, press releases and entries on social media sites. The skills of each employee are also typically available at these information sources e.g., PhD in Chemistry, 7 years as a chemical engineer. The engine collated all these pieces of intelligence about the businesses.

The final step was to compare the regions’ knowledge sectors with other technology clusters in the US, Canada and Europe from a human capital perspective. To answer this, fifty knowledge-sector roles ranging from Aerospace Engineer to Gaming Developer were collated from the glass.ai data and we calculated the prevalence of these roles per 1,000 employees. This allowed our client to identify roles and sectors where the region could be world-leading and roles that may require further interventions (e.g., incentives to attract technical talent).

Conclusion

With our expanded global coverage and the continuous monitoring of millions of companies, we are transforming the difficult task of doing sector research across countries. Our AI research capability can not only identify companies in any sector, but, it can also track the sector over time and go deeper into the activities, interactions and networks of the companies. This is a unique capability that we have developed at glass.ai. Exciting times ahead!