It’s Friday 1st February, 2023. And Swirl releases version 3.2.0. With this, we continue our march towards a secure, tight, air-gapped enterprise-grade unified search and data access platform, which operates within your enterprise firewall, and with your private enterprise large language model.
Swirl is available in Azure with a 30-day free Trial. Check out this page to know more.


This release includes the first version of the AutomaticPayloadMapperResultProcessor,  RequireQueryStringInTitleResultProcessor,  DropIrrelevantPostResultProcessor, and NoModQueryProcessor, along with new SearchProviders for OpenSanctions.org and LittleSis.org. We’ve also added support for running OpenAI in Azure and storing query templates in JSON, plus updated Python, Django, and OpenAI API versions and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to star/favorite results above a particular relevancy score.


We’re seeking anyone interested in testing new Connectors to Collibra, Thoughtspot, and Pinecone DB. Please contact Swirl via email or Slack for access.

New Features

🔹 New AutomaticPayloadMapperResultProcessor profiles response data to find good strings for Swirl’s title, body, and date_published fields. It is intended for SearchProviders with few (or no) good result_mappings options. It should be placed after the MappingResultProcessor, and the result_mappings field should be blank.

🔹 New NoModQueryProcessor that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).

🔹 New RequireQueryStringInTitleResultProcessor that drops results that do not contain the query_string_to_provider in the result title field. It should be added after the MappingResultProcessor and is now included by default in the “LinkedIn – Google PSE” SearchProvider.

🔹 New DropIrrelevantPostResultProcessor drops results with swirl_score < settings.MIN_SWIRL_SCORE (set to 500 by default) and results with no swirl_score.
⚠️ The Galaxy UI will not display the correct number of results if this ResultProcessor is deployed. This will be addressed in a future release.

🔹 Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI’s ChatGPT models.

🔹 The Galaxy UI now highlights results with a swirl_score above a configurable threshold with a star in the results list. The swirl_score configuration is available in theminimumSwirlScore entry of static/api/config/default, and the default is 100.

🔹 New SearchProvider.query_template_json field which stores the query_template as JSON. The MongoDB SearchProvider has been updated to use this feature.

🔹 Support for lowercasing of URLs is now available via a new LC_URL option in SearchProvider result_mappings.

🔹 Added support for list-of-list responses from source APIs, where the first list element is the column names.

New Search Providers

🔹 Search the LittleSis free database of “who-knows-who at the heights of business and government” with the new LittleSis.org SearchProvider.

🔹 Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.

🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.

Improvements

🔹 Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.

🔹 When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.

🔹 Swirl was validated on Python 3.12.1.

🔹 Swirl now runs on the new Django 5.0.x series Python packages.

🔹 Swirl now uses the latest 1.6.x OpenAI Python API.

🔹 The QueryProcessors base classes were updated.

🔹 The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI. For example http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000

🔹 The Snowflake SearchProvider query_template was revised.

🔹 The BigQuery SearchProvider no longer dedupes results.

🔹 Improved zero results feedback and logfile messages.

Resolved Issues

🔹 Removed duplicate static/api/config/default configuration file.

🔹 Removed unused timeout settings (SWIRL_Q_WAIT and SWIRL_RERUN_WAIT).

🔹 Fixed Elasticsearch authorization and addressed a certificate error.

🔹 Updated the index_email_elastic.py script and related documentation.

🔹 Stopwords are no longer highlighted in search results or AI Summaries.

🔹 Various small fixes to Galaxy UI behavior.

For More Information

Please check out this release from our GitHub Page.

Don’t forget to join our Swirl Enterprise Search Community Slack.