All systems are operational. Incoming events should be processed successfully. We've also processed the backlog of events that arrived during the first part of the incident, from ~1:00am UTC to 3:51 UTC.
From 3:51 AM UTC to 6:04 AM, we had to disable incoming events due to some of the load issues we were experiencing. Events sent during this time may be retried by the source services. At 6:04am, we re-enabled the collection of incoming events again.
We'll follow up with a detailed retrospective of this incident as soon as possible.
Posted Feb 14, 2023 - 07:06 UTC
Monitoring
AWS has shipped a fix for the issue. We're restarting our services and bringing workflows back online. We'll send another update as soon as that's done.
Posted Feb 14, 2023 - 06:02 UTC
Update
AWS is still working on a patch for the issue. We're still in active communication with them.
Posted Feb 14, 2023 - 05:56 UTC
Update
The AWS Lambda team confirmed the issue was due to the scale of volume Pipedream is running on Lambda. They're working a fix now. We'll update this incident again soon.
Posted Feb 14, 2023 - 04:49 UTC
Update
We are continuing to work on a fix for this issue.
Posted Feb 14, 2023 - 04:09 UTC
Update
AWS has escalated to more teams internally. The issue is still ongoing.
Posted Feb 14, 2023 - 03:33 UTC
Update
We're continuing to discuss this with the AWS team. The issue is still ongoing.
Posted Feb 14, 2023 - 03:01 UTC
Identified
AWS Lambda — part of the service we use to run workflows — has identified a service issue. They're working on it, and we'll communicate updates here.
Posted Feb 14, 2023 - 02:27 UTC
Investigating
We're seeing a spike in Pipedream Internal Errors across workflows, and we're investigating.
Posted Feb 14, 2023 - 02:04 UTC
This incident affected: Frontend (https://pipedream.com) and Backend (Workflows - Timer, Workflows - HTTP, Workflows - Email).