MIP Reference Guide
Welcome to the comprehensive operational guide for the DFR, developer first responder, managing the MIP, monthly-invoicing process. Details include monitoring, remediation, and communication procedures throughout the monthly invoice generation cycle.
The following tasks don’t have to be individually completed by
the DFR; consider reporting and or delegating tasks in Slack
channel #invoicing-dev. Tuning individual resources may not be
necessary, but required manual processes include redriving messages
from the dead letter queues and retrying failed pending invoice
requests.
MIP Workflow Overview
flowchart TD
Prep[Pre-MIP Prep
Internal Tools Access, Deployment Scheduling] Prep --> Start([MIP Begins]) Start --> Monitor[Continuously Monitor
Datadog Dashboards, DLQs] Monitor --> Check{Issues?} Check -->|Yes| Investigate[Investigate
Database Queries, Error Logs] Check -->|No| Verify[Verify Zuora Invoices
Check ZIP dashboard] Investigate --> Remediate[Remediate
Redrive DLQs, Retry Requests] Remediate --> Monitor Verify --> Complete{Invoice
creation
complete?} Complete -->|No| Monitor Complete -->|Yes| Communicate[Post-MIP Communication
Notify Stakeholders] Communicate --> End([MIP Complete]) linkStyle default stroke:#7dd3fc,stroke-width:2px classDef decision fill:#2d2d2d,stroke:#7dd3fc,stroke-width:3px,color:#fff class Check,Complete decision
Internal Tools Access, Deployment Scheduling] Prep --> Start([MIP Begins]) Start --> Monitor[Continuously Monitor
Datadog Dashboards, DLQs] Monitor --> Check{Issues?} Check -->|Yes| Investigate[Investigate
Database Queries, Error Logs] Check -->|No| Verify[Verify Zuora Invoices
Check ZIP dashboard] Investigate --> Remediate[Remediate
Redrive DLQs, Retry Requests] Remediate --> Monitor Verify --> Complete{Invoice
creation
complete?} Complete -->|No| Monitor Complete -->|Yes| Communicate[Post-MIP Communication
Notify Stakeholders] Communicate --> End([MIP Complete]) linkStyle default stroke:#7dd3fc,stroke-width:2px classDef decision fill:#2d2d2d,stroke:#7dd3fc,stroke-width:3px,color:#fff class Check,Complete decision
Quick Reference
| Scenario | Action |
|---|---|
| MIP starting soon? | Review Pre-MIP Preparation → Verify database access → Discuss deployment schedule |
| Invoice creation running slow? | Check Continuous Monitoring → Review Datadog Dashboards → Verify creation speed >10 inv/s |
| Messages stuck in DLQ? | See Remediation Steps → Use AWS CLI to redrive messages → Monitor completion |
| Failed invoice requests? | Check Investigation Techniques → Query failed requests → Use retry endpoint |
| Zuora-sync issues? | Follow Zuora Invoice Verification → Check ZIP dashboard → Redrive ZIP DLQ |
| MIP complete? | Use Communication Protocol → Post in #ops-mip-supports → Include invoice count |