Why I Created Swirl Metasearch

I’ve been fortunate to work in search a few times now. And I believe it continues to be the most human way of interacting with data. But search changes constantly; Brin and Page were chasing the Star Trek computer; they ended up with an ad engine. Search is tricky, and not just because of the massive costs associated with indexing, classification and all the other things associated with it. Like large language models. 

In 2022, I noticed something else had changed. 

• The CIO of a global publisher told me that M365 had saved their business during the pandemic. They had been forced to the cloud by office closures. 
• A CRO told me about the effort to combine their various CRM systems into Salesforce had cost a fortune… but it was worth it. Because they had better visibility across accounts. 
• A senior PM told me about how they were finally decommissioning knowledge bases they had moved to ServiceNow.

Federated search, also known as Metasearch, has always been interesting. If every silo is getting smarter, as the search vendors surely tell us they are – it should work, and work efficiently. The trick has been connectivity. It was not that long ago that network plumbing changes were needed to access each silo. It was equally impossible to compare disparate result sets, different schemas – and resolve entitlements as well. 

Most of these things are not true today.  Most every enterprise application uses a Single Sign-on standard such as SAML or OIDC which securely handles both authentication and authorization as the access and identity model. Almost any developer can read the documentation and figure out how to connect to it. More importantly the rise of AI has made it possible for metasearch engines to contextually compare responses. And this is just the beginning; generative AI can even be applied to user queries and results… 

Search will never be the same. 

No massive indexing. Ubiquitous connectivity and large language models unlock unstructured data like nothing else! You can get results in a fraction of the effort. 

So, why did I create Swirl? Having worked in enterprise search for many years, I am shocked to see so many of the same problems still plaguing the industry. But I have also realized that there is a better way. 

I appreciate everyone’s’ support for the project so far and am excited about the future. I encourage you to learn more about the Swirl Metasearch project by visiting the project page on GitHub where you can: 

• Download the latest version of Swirl Metasearch and try out the software (it’s open source and free to download). 
• Review the technical documentation explaining metasearch in more detail and learn about the architecture and features. 
• Join the project and become a contributor.
• Join the discussion and participate in the development roadmap. 
• And if you like what you see, star the project to help us raise awareness.
• You can also check out some of our recorded Developer Webinars here.

Cheers! 

-Sid