Boomi Unlocks the Cloud with Solr

Customer Overview Founded in 2000, Boomi provides integration between on-premise applications -- such as SAP or Oracle -- and off-premise SaaS applications like Salesforce.com, Taleo, or NetSuite, which are often running on public cloud services (like Amazon EC2, or on platform-as-a-service providers such as force.com and Intuit Partner Platform). Boomi helps their customers tie together these applications to create integrated business processes that span multiple applications in and out of the cloud, tracking the flow of data inside and outside of the corporate firewall. It manages complex work flows across multiple applications and locations, adhering to strict constraints for data control, ownership, and compliance.

To handle these requirements, and deliver data visibility and governance, Boomi developed an innovative multi-tenant cloud integration platform built on Apache Solr using services and expertise. The resulting implementation puts Solr's data-source virtualization capabilities at the heart of Boomi's business process model. Challenges A detailed view of business execution information is essential for application and data integration - any time data is moved from one application to another, key information must be captured, such as the time it takes to move the data, the amount of data processed, and verification of transaction success.

With key new applications moving out into the cloud, IT organizations must be able to manage the information inside the company and inside the cloud simultaneously, as though they were in a single environment. In order to achieve this kind of centralized control and governance for its customers, Boomi needed to have all its customers on a single instance of its platform. Were monitoring to be decentralized, users would be forced to remember which integration was running in which environment, and have to switch from one to the other, propagating changes manually and fragmenting the integrated view they had turned to Boomi to create in the first place.

Boomi achieves its pinpoint accurate visibility into transactions across applications by processing and synthesizing integration data from log files. As a result, their platform had to be able to scale to billions of log files while delivering mission-critical reliability for customers. Its use as a tool to monitor mission critical business process through the cloud could not tolerate slowdowns.

Speed was a concern on two fronts. First, the logging-and-searching service is characterized by frequent commits to make the data available for search; every 5 seconds or 10,000 transaction messages. Second, customer queries had to meet an aggressive performance threshold. Finally, customer queries about application events could not be slow Boomi decided against using a database approach due to scalability requirements to billions of documents, high cost of ownership for databases, and the requirement for dynamic customer queries and metadata.

To build a single integrated multitenant platform, Boomi decided to use Apache Solr to monitor and search transaction logs, and deliver the information for a timely, integrated view of business process execution for applications in and out of the cloud. Boomi asked Lucid Imagination to review its Solr implementation for current best practices, to provide recommendations on performance, prepare headroom for future scalability, and ensure fault resilience for business continuity.

Solution Solr is integrated deep into the platform that Boomi provides for customers to achieve an integrated view across applications in and out of the cloud. They now have hundreds of customers who use this platform to monitor their business process applications, by harvesting, parsing, and indexing log data across many applications, and then presenting a consolidated business process view to each individual customer for each customer's individual process.

Today, Boomi customers rely on Solr searches to monitor and verify cloud application activity and workflow events via a web interface, through RSS feeds.