This talk covers the process of making Rapid7's Project Sonar dataset searchable. Starting with the initial naive approach taken to achieving this goal, and then moving onto the next iteration which took a more sophisticated approach in order to achieve microsecond lookup times for entries in a 180GB dataset with O(1) time complexity.
Making Big Datasets Searchable
Calum is a Senior Security Consultant and Lead Security Engineer at onsecurity.io. His work is split between pentesting and building an automated security platform for clients. In addition to his work at OnSecurity, he also develops and maintains some open-source tools for the community.