Today's newsletter is about meshing data sources, scraping technologies & natural language processing experiments.
🙌 Thanks for reading
In my spare time (i.e. when the kids go to sleep) I have been experimenting with tools to further my understanding of the digital, social and political environment we live in. The project works as follows:
1.Collect Info → 2.Blend/Mesh Info → 3.Identify Insights
Here are my projects from the past few weeks.
Meshing Data Sources
What is it? This is about blending multiple sources of information together to reveal new insights. For example, using the weather forecast, moon calendar and gardening guide to decide when is a good time in the week to plant certain crops; or using trending keyword volume from Twitter, news article clustering trends and crypto-currency prices to determine when to sell stocks.
What I found: I wanted to step away from the major aggregation platforms, such as segment.io, customer.io and look at more at start-up/developer tier stacks. The challenge was to develop a simple insights dashboard using JAMstack. I ran two set-up experiments on API-Platform and Takeshape. I preferred Takeshape, API-Platform was installed via a docker with many utilities & resources (too many).
How does it work? Simply, the data is fed via API and is output is a single endpoint that can be accessed using GraphQL queries. Takeshape have provided ready-made 'patterns' for data-models and a very easy to use interface.
Why does it matter? Traditional approaches to combining/meshing/blending data into one centralised destination required significant design & engineering resources at the outset. The Takeshape approach, while highly opinionated, offers significant time efficiencies for projects that need to move very quickly in obtaining actionable insights. With a team of 2 consisting of a front-end dev, data modeller you could have a dashboard up in probably 2 hours and then meaningful insights within 2/3 days.
Scraping Technologies
What is it? Scraping is about harvesting or collecting data from conversation networks and publishing platforms. Conversation platforms include: Twitter, Instagram, Facebook, WhatsApp. Publishing platforms include news sites.
What I found: Each information platform/network seeks to protect some of its information to varying degrees, with both the way information is made available and the way it is re-publishable. Most have privacy agreements that protect the IP of the contributors (authors, users) but almost all allow external access it via API, RSS, email newsletters, embed, affiliates, developer or syndication agreements. Each site has in place varying levels of data-protection measures such as authentication, anti-bot scripts, rotating IP addresses, human time-to-action restrictions or token/cookie/refresh checking scripts.
How does it work? API access is provided by the platform(s) and requires a developer account. Scraping is provided either by external services or using tools such as python and searching the online millions of posts or communities for 'how to'. Get a start on scraping here.
Why it matters? Drawing an analogy between an individual farmer harvesting a crop using a sickle and a tractor harvesting fields using an automated cutter, the machine would collect far more information than the individual. Therefore the machine operators are equipped with more potential insights.
Natural Language Processing (NLP)
What is it? Following on from an earlier article, my interest in NLP is taking a big collection of news articles and identifying the attitude in the article. Is it positive, neutral or negative? This week I tested two services: NewsAPI (provided by Google) and GoogleCloud NLP and published the results to a Google Sheet using GoogleAppsScript.
What I found: I found that certain services are orientated to English and don't service other languages very well. I posted this:
The ʻokina in Hawaiian is an integral part of the language, its character is ʻ .
— Native Zone (@native_zone) October 5, 2021
The code below is pulled from an off-the-shelf artificial intelligence natural language processing system that intentionally removes the ʻokina making Hawaiian invisible. @MaoriDSov @Te_Kotahi pic.twitter.com/lV6i7NW7RF
I also found that GoogleCloud NLP does not support analysis of languages outside the larger language sets:
Rangahau: Māori language is not accepted for sentiment analysis in the Google NLP system, need to figure out how to work-around that - big job. pic.twitter.com/T01bejKuKF
— Native Zone (@native_zone) October 5, 2021
Why it matters? For small minority groups wanting to understand how to obtain insights from NLP it's hard! They have to develop the tools and approaches themselves or augmenting their tech with the current providers. This could be an asymmetrical relationship.
BTW - here are the results of my sentiment analysis test.
⚡️ Sentiment Analysis. Using 'Maori' instead of 'Māori'. international media have picked-up on the latest COVID19 news. The top article since 1 October 2021 in English language global news media is
— Native Zone (@native_zone) October 5, 2021
"New Zealand abandons its goal of eliminating the coronavirus." pic.twitter.com/ukR478aQfG
Have a great week! The traditional knowledge from the Māori moon calendar says we are heading into 'Tamatea' nights phase which is building positive energy, check it out here.
