BSides London 2021 BSides London 2021
Join event to build your agenda.

Making Big Datasets Searchable

- GMT
Rookie Track

Calum is a Senior Security Consultant and Lead Security Engineer at onsecurity.io. His work is split between pentesting and building an automated security platform for clients. In addition to his work at OnSecurity, he also develops and maintains some open-source tools for the community.


This talk covers the process of making Rapid7's Project Sonar dataset searchable. Starting with the initial naive approach taken to achieving this goal, and then moving onto the next iteration which took a more sophisticated approach in order to achieve microsecond lookup times for entries in a 180GB dataset with O(1) time complexity.