Scaling Products, the Right Way

We worked with a leading mobile advertising and ad-optimization platform that creates and delivers advertisements to mobile devices.

The company needed a solution to generate real-time reporting for Key Business Metrics and Campaign Pacing Engine, insight report with 50+ dimensions to help publishers and advertisers with ad-requests traffic pattern and an Audience Management Platform (AMP) to profile users’ interests and facilitate user targeting.

• Needed to handle the velocity of 5mn ad requests/sec.
• User segmentation should support storage and realtime-querying of about one Billion users (device Ids) across 30K+ user characteristics.
• Both Reporting and user-segmentation solutions should scale beyond 50Billion requestsand 1Billion users if the traffic grows further.

• Developed a Lambda architecture with Amazon Cloud Platform with a processing layer,storage layer, and messaging system.
• Used Apache Storm based real-time Stream Processing Engine to crunch Key Business Metrics in real-time.
• Used EMR Hadoop/ Spark based batch processing engine to crunch 5-6TB data daily.
• Used MySQL DB for transactional data, Amazon Redshift Data warehouse for analyticaldata, Aerospike DB and Elastic Search for storing and querying user-profile and Amazon S3 for storing raw data.
• Implemented Apache Kafka to collect events data from over 200 ad-server instances.

• Managed over 10mn ad requests/sec when traffic peaked, with 2bn+ active users segmented across 80k+ dimensions.
• Developed the solution within an opti mal infrastructure cost.

The client introduced enterprise-grade Wi-Fi, BLE and IoT together to deliver personalized, location-based wireless services without requiring battery-powered beacons. Its AI-driven wireless platform makes Wi-Fi predictable, reliable and measurable.
The company needed a solution to generate data for Analytics dashboard for WiFi users and real-time alerts for anomaly events. It also wanted to develop a Virtual Network Assistant to help troubleshoot anomaly events.

• The velocity of IoT data was at the scale of 10mn/sec.
• The anomaly condition should be deduced and needed to be broadcasted in Real-time(within few seconds)

We designed the solution using following Big-Data tech stack

• Processing Layer
  • 80+ Nodes Storm-Cluster to process data from over 80 topologies in Real-time to intercept anomaly conditions and crunch data for Analytics dashboard.
• Storage System
  • 40+ Nodes of Cassandra cluster with capacity to hold 5+Terabytes data for Analytics dashboard and Anomaly alerts.
  • 10+ Nodes of Elastic Search cluster to store detailed data (50+ dimensions) for Virtual Network assistant to facilitate troubleshooting.
• Messaging System
  • 10+ Node Kafka-Cluster housing over 200+ Kafka topics, average data transfer of 100+MB/sec. Each message having from 50 to few hundred fields.
• Cluster Manager
  • Apache Mesos on AWS using Docker.

The solution we built has been robust and handles over 10mn IoT data/sec and the alerts go out within 30sec of any anomaly condition. The solution worked in various cloud environments – Amazon, Google & Azure.

We worked with a supply chain management software platform that has operations in 20 countries with 10K buyers and 200K suppliers on the platform. Expansion of the business made it clear that modernization is necessary to handle 10K rows and 600 columns concurrently for 100 users.

• The existing solution was using an online excel to manage Bill of Material (BOM) from buyers and responses from sellers. But it could handle only 500 rows and 20 columns concurrently.
• The company had to process larger BOMs manually. It used to take 5-6 months to reward single BOMs, and they wanted to reduce it to 5-6 weeks.

• We built a frontend, like excel, with responsive backend APIs, which could handle a concurrency of 3K requests/second. It allowed 100 users with different roles to make changes/format the sheet simultaneously.
• To support 10K rows and 600 columns for every BOM, we migrated existing SQL to scalable NoSQL (MongoDB) and migrated .NET monolith to node.js based microservices.
• For awarding, we created an online excel supporting over 10 million with search, filtering and sorting options using our own database on Lucene search.

Scaled a platform to handle more than 10K rows and 600+ columns for 100+ concurrent users.

The company built project flow management software to improve the on-time delivery ofprojects. Their existing solution was using a thick client granting edit access to only a single user. It had a limitation of handling X tasks that impacted handling long-duration projects.

• Allowing concurrent access to multiple users.
• Scaling the existing system to handle 100K tasks for a single project.
• Migrating monolith application to decoupled subsystem.
• Upgrading technology to ease development and improve skill availability.

Phase-wise implementation

• First phase – Built basic Gantt UI and exposed scheduling algorithms.
• Second phase- Added complex features and scaling to handle bigger projects.

Layer-wise breakdown

• Exposed API Layer for processing patented algorithms as a stateless webservice.
• Implemented a separate API layer to handle database/storage.
• Segregated the UI layer and user interactions.

Refactored into decoupled subsystems

• Improved the software maintainability and upgrades.
• Covered end-to-end business with unit, integration and regression test suits.

• Enhanced end-user collaboration with notifications for conflicts, merge and resolve mechanism.
• Made integrations easier with other applications using web services.
• Built a powerful query engine to analyze project data and enable custom reports.

Our customer was a media company that helped creators, brands, and traditional media firms reach the right viewers. In order to understand the user sentiments, they wanted to analyse entire data on YouTube every 12 hours.

• Build a data processing platform that can efficiently acquire, ingest and process entire data from YouTube every 12 hours.
• Understand the mood and sentiment of the users on a daily basis based on comments, description.
• Recognize the audience demographics their likes and dislikes.

• Built a custom crawlers to pull data from various sources and collect data.
• Amazon S3 as a data lake where all the fetched data is stored.
• Snowflake is an analytic data warehouse provided as Software-as-a-Service (SaaS) for data processing. It pulls data from S3 processes and stored data in relational form in Snowflake as well as loads it to postgres where reporting
  application connects

Analysed the sentiments and mood of users across YouTube videos every 12 hours.

The company has a financial search engine platform for business insights and market intelligence powered by AI and NLP technology. It empowers leading corporations and financial institutions with data-driven decisions and a competitive edge.

It came to us with its initial product version using publicly available data and a few reputable third-party sources. They wanted us to scale the integrations of industry- specific data from third parties and include private organizational data in search results.

• Enabling search engines to use organization-specific private data stored on software like Evernote, OneDrive, and SharePoint to improve search results.
• Adding high-value curated content from more than 1000 third-party sources in search results for credible search results.

Implemented integration framework

• To handle unstructured private data from data management software like Evernote, OneDrive, and SharePoint.
• To manage bulk data spikes and refresh the data at set intervals using file sync.

Built data pipeline

• To add high-value curated content from more than 1000 industry-specific credible third-party sources in search results.
• To allow users access to paid content with various payment options like pay per report, per page, etc.

Made data AI-ready

• Implemented data annotations for private as well as third-party data to provide improved search results for organizations.

• Accelerated customer acquisition as the inclusion of private and third-party data increased the efficiency of search engine.
• Helped in raising Series B round funding of $50M.