StateOfOpenCon24: Connecting the Dots

Description:

Presented by: Tim Armstrong

Data nerds, like the presenter of this talk, love Stock Market data. Stock market data is simultaneously fascinating and infuriating, as it combines what should be predictable data (company does well = price goes up) with human nature (CEO gets on stage and puts foot in mouth = price goes down). The first part of any stock market data project - be it training an AI, tracking your own portfolio, or analyzing the fall of a company from IPO to Delisted - Is collecting the data.

Stock market data is big! With thousands of data points per hour per ticker, simply collecting the data is a challenge in and of itself. Thankfully, CloudQuery is very good at collecting data, so all you need is a source plugin. Fortunately for you, creating plugins to pull data from pretty much any source is easy, thanks to the CloudQuery Source Plugin SDKs and our super simple YouTube Tutorials. But what’s better than writing code while watching a tutorial? - Having a dedicated tutor write the code for you while teaching you how to write similar code in real time.

In this talk, the audience will learn how to write a CloudQuery Plugin in Python that pulls actual data directly from the London Stock Exchange and uses CloudQuery to push it to Postgres for later analysis. This talk will cover best practices, security concerns and solutions, publishing and selling plugins on CloudQuery Hub, and answer questions from the audience about CloudQuery, software development, and data engineering.

The goal of the talk is to empower the audience with the ability to go out and make Plugins for any API they desire so that they can make data-driven decisions.

Videos overview