Data Science App Empowering Venture Capitalists to Find High-Value Startups

Earlier evaluating a startup’s potential involved a mix of leadership assessment, market realities, and often, gut feeling. However, data science and analytics are now revolutionizing this process.

As a result, our client wanted an objective assessment of a startup's potential by programmatically analyzing data metrics, financials, and growth rates. Through such assessments, our client wanted to find and invest in high-value startups or founders.

The client already had an analytical app to do the job, but it was built on an old codebase. The security mechanism had multiple issues. The database operations were slow. The unoptimized APIs were not driving quality data from external sources. Such problems led the client to consider rebuilding the application from scratch.

At this stage, the client contacted Capital Numbers. The client requested that we build a data science-powered web app to process billions of startup data from external sources. They wanted the app to serve as the single source of truth for the following operations:

• Data collection
• Data storage
• Data analysis
• Data monitoring
• Data reporting

We needed a robust cloud data warehousing solution to perform the above operations. Our focus was to ensure super-fast processing of advanced analytical queries to support quick decisions.

Our team began the work by studying the existing analytical app. We found the data sets badly scattered in the existing setup. For example, S3 buckets contained raw data, PostgreSQL contained SQL data, and DynamoDB contained NoSQL data. So, we prioritized organizing all that data in one place using Snowflake.

We chose Snowflake because this cloud data warehousing solution is excellent for storing large workloads of near-unlimited scale. It allows firms to store overwhelming amounts of data economically while dynamically scaling compute resources to meet the demands of active querying processes.

We ran CRON jobs and custom APIs within Snowflake to automate data extraction from external sources. Additionally, we implemented ETL processes to centralize all data in desirable formats for structured analysis.

Next, we had to show the collected data on the frontend. So, we implemented Sigma because it reflects data on a spreadsheet-like analytics interface. Although Sigma resembles a traditional spreadsheet, unlike traditional spreadsheets, it pushes far more complex queries to the data warehouse to return accurate results against billions of data.

Moreover, traditional spreadsheets have a hard limit on the number of rows. However, Sigma can visualize billions of rows of datasets in seconds, putting no limits on analytics. Sigma also allows calculations to be performed on columns instead of rows, allowing users to work on large datasets without worry.

In addition, once built, Sigma’s reports directly query live data sets to the data warehouse and update them in real-time. So, we didn’t have to manually ingest data pipelines to connect the backend and frontend, which saved us time. We also developed custom React plugins and JavaScript APIs in the Sigma package to power up the frontend.

Once we fully prepared the application for data collection, storage, monitoring, analysis, and reporting, we hosted it on the AWS server. The collaboration required us to push multiple statistical formulas to present the most granular data out of billions of data values.

Capital Numbers’ work led to the delivery of a data science powered web app that is an order of magnitude larger than regular data-intensive apps. Here’s how the app can benefit its users.