Abuse detection is a crucial component of maintaining the integrity and performance of services that interact with LLMs. Layerup Security has developed a custom algorithm that monitors and prevents abuse by setting thresholds on the number of requests per minute (RPM) at various levels. To utilize abuse detection, metadata must be specified for each request. The abuse_config object allows you to specify an abuse configuration to detect abuse across the project. The configuration can be set at four different levels:
  1. Project Level (project_rpm): Sets a threshold for all requests to the project. If the RPM exceeds this threshold, the LLM call will be marked as unsafe.
  2. Customer Level (customer_rpm): Sets a threshold for all requests made by a customer for the project. If the customer’s RPM exceeds this threshold, the LLM call will be marked as unsafe.
  3. Scope Level (scope_rpm): Sets a threshold for all requests made within a scope for the project. If the scope’s RPM exceeds this threshold, the LLM call will be marked as unsafe, regardless of the customer.
  4. Customer-Scope Level (customer_scope_rpm): Sets a threshold for requests made by a customer within a given scope for the project. If the customer’s RPM for the scope exceeds this threshold, the LLM call will be marked as unsafe.
To activate abuse detection, invoke the layerup.abuse guardrail with the appropriate abuse_config metadata.