Spike in Pipedream Internal Errors for workflows
Incident Report for Pipedream
All systems are operational. Incoming events should be processed successfully. We've also processed the backlog of events that arrived during the first part of the incident, from ~1:00am UTC to 3:51 UTC.

From 3:51 AM UTC to 6:04 AM, we had to disable incoming events due to some of the load issues we were experiencing. Events sent during this time may be retried by the source services. At 6:04am, we re-enabled the collection of incoming events again.

We'll follow up with a detailed retrospective of this incident as soon as possible.
Posted Feb 14, 2023 - 07:06 UTC
AWS has shipped a fix for the issue. We're restarting our services and bringing workflows back online. We'll send another update as soon as that's done.
Posted Feb 14, 2023 - 06:02 UTC
AWS is still working on a patch for the issue. We're still in active communication with them.
Posted Feb 14, 2023 - 05:56 UTC
The AWS Lambda team confirmed the issue was due to the scale of volume Pipedream is running on Lambda. They're working a fix now. We'll update this incident again soon.
Posted Feb 14, 2023 - 04:49 UTC
We are continuing to work on a fix for this issue.
Posted Feb 14, 2023 - 04:09 UTC
AWS has escalated to more teams internally. The issue is still ongoing.
Posted Feb 14, 2023 - 03:33 UTC
We're continuing to discuss this with the AWS team. The issue is still ongoing.
Posted Feb 14, 2023 - 03:01 UTC
AWS Lambda — part of the service we use to run workflows — has identified a service issue. They're working on it, and we'll communicate updates here.
Posted Feb 14, 2023 - 02:27 UTC
We're seeing a spike in Pipedream Internal Errors across workflows, and we're investigating.
Posted Feb 14, 2023 - 02:04 UTC
This incident affected: Frontend (https://pipedream.com) and Backend (Workflows - Timer, Workflows - HTTP, Workflows - Email).