Social media has undoubtedly changed the world. It has reshaped how people live, work, and communicate, and also brought people around the world closer together. With social networking comes a veritable smorgasbord of information types, from peer and family posts to videos to Instant Messaging chat history.

People are also now 2.5X more likely to turn to social media for news as reported by The Reuters Institute. With the proliferation of social media networks and their accessibility on smartphones, everyone has at least one social media channel as their source of daily newsfeed, such as Facebook.

Organisations are also seeing the benefits of leveraging social media to connect with their communities and target audience, engage with their customers, and grow their businesses.

However, managing the endless streams of structured and unstructured information and data coming from multiple sources and all too often in real-time can be overwhelming. Hence, organisations need to ensure that they have robust and multifaceted databases and systems to support them. And the platform enabling the accurate capturing, archiving, and processing of these data needs to be well structured to accommodate various streams, easy to manage and upgrade, and responsive enough to support multiple interoperable applications.

For Brolly, Australia’s and Southeast Asia’s preeminent social media archiving and insights solution, securely capturing and preserving social media conversations in real-time is critical for its customer base. But just as important is providing the analytics framework to make sense of data for its social clientele and maintaining digital records that comply with the various data and privacy regulations in different regions.

Developed with regulated industries in mind, Brolly’s attention to compliance means it is the ideal solution for all levels of government and any regulated industry, including banking, insurance, health, financial services, and education. It has many such clients in Australia, New Zealand, North America, and Europe, where tight regulatory landscapes monitor data usage on digital platforms.

“In those regions, there are some regulations in place that advise organizations to be aligned with those regulations. Because there, activity on social media will be considered public records,” said Ali Nazemian, the Chief Technology Officer who leads Brolly’s engineering and technology team.

To meet these diverse needs agilely and with minimal operational hassle, Brolly uses an open-source cloud data platform provider, Aiven. “With the use of open-source technologies in the form of managed services from Aiven, we can have the best of both worlds by keeping our operations cost low, having the ease of mind for commercial support, and using a reliable solution with the support of open-source communities.”

This platform support is vital for a dynamic outfit like Brolly, Ali says, as it allows Brolly to “focus on what is important to us, which is bringing a delightful experience to our customers.”

Aiven’s cloud-based infrastructure is suited to support the growing scale of Brolly’s intensive social media archiving needs, such as the rate of data ingestion changing frequently: a solution that can keep up with millions of messages over a short time is needed.

“Messages can go viral, so it’s important to have a scalable solution that can keep up with social media records no matter what the rate of ingestion is,” he elaborated. “To overcome this, we have used Apache Kafka and its ecosystem to build an event-driven data pipeline that can be a highly distributed and scalable medium for social media records in transit. Aiven helped us to reduce our total cost of ownership while using Apache Kafka.”

Because of Brolly’s work with enterprises from different regulated sectors, the need to accurately ingest, store, search and export social media data with the highest level of accuracy and coverage is top-of-mind. “Not being able to collect a social media record is simply not acceptable,” Ali stated emphatically.

“We cannot afford to lose a record because we never know what record gets important in the future, so we have to treat every single record as it’s critical for the business,” Ali stated. “With the help of Apache Kafka [functions] from Aiven, we have been able to provide the highest level of accuracy and coverage. We have built various monitoring and analytical tools with the help of open-source technologies to get notified of any failures and automatically recover from errors.”

Accuracy is paramount, but the twin competencies of speed and reliable data volume handling are equally important. “When a social media account gets connected to Brolly, all the corresponding social media records for that account get collected in just a few hours in addition to serving real-time messages immediately,” the CTO said. “Our customers are able to extract and search millions of records in real-time once their historical data is collected, thanks to the underlying open-source technologies and Apache Kafka.”

Apache Kafka’s flexible architecture makes moving data a breeze, tapping its open-source advantages to pull in various APIs, management interfaces and applications to give diverse options. As an open-source platform, Aiven connects all the components in the value chain, from brokers to producers to consumers.

“The use of open-source technologies from Aiven has helped us to focus on making social media a better place for our customers, communities, and citizens,” summarized Brolly’s Ali. “This will allow us to be the best social media archiving and insights platform and yet give back to the community.”