Workflow Overflow

It seemed like a simple request.

“If a technician closes a ticket without attaching a Configuration (device), reopen it, send them an email, and change the ticket’s status so they know they need to remedy the lack of Configuration before closing it again.”

Cool. I can do that.

ConnectWise, aka ConnectWise Manage, aka ConnectWise PSA (apparently that’s the new new name) has a solution for that, the Workflow Rules system. I gave it three simple parameters:

  1. Ticket is in a Closed state. There are several “statuses” that make a ticket Closed, we don’t care which one was used, just that the state is that of being Closed.
  2. Ticket has no Configuration attached. Even if there wasn’t a specific computer or other physical device, we still want something, even if it’s the “Generic Client Device” entry we created for exactly this purpose. (The internal politics behind this decision are a lot more “inside baseball” than anyone needs to know, honestly.)
  3. Ticket’s last status change cannot have happened more than 1 (one) hour prior to the workflow rule check (scheduled to happen every quarter hour). The goal here is not to raise any tickets from the dead. Necromancy is strictly forbidden.

Three simple parameters. And somehow, ConnectWise Manage PSA Thingy got one of them very, very wrong. It is, in fact, still getting it wrong as I write this. What actually happened is that starting from Ticket #1 in the system, every ticket in the selected Service Board that is Closed (which is, as you might guess, nearly all of them) and has no Configuration attached (which is, as you could guess if you knew that this requirement is a relatively new company policy, also nearly all of them) had the changes applied (reopened & reassigned), ignoring the fact that their last status changes were months, years, or even over a decade ago. The rule set was quite clear, we double and triple checked it, and ConnectWise ignored it anyway.

All of the reopened tickets, all of them, were set to the “please update this ticket” status and, if the technician who worked the ticket previously is still in the system, that technician was reassigned and sent an email about their need to update said ticket.

Necromancy was forbidden, and ConnectWise went ahead and unleashed the undead anyway. “When thou misspoke the words…”

Now, not many of us have been with the company for more than a decade, but some of us actually have. Imagine our inboxes right now. But that’s not the fun part. Oh, no. Select All and Delete are a fine solution to that part.

However.

We enabled the workflow rules at around 10 o’clock this morning, after announcing them in the bi-weekly service team meeting. Nearly an hour later someone noticed that we had a lot of reopened tickets. A lot a lot. Oh, no. So I chatted up ConnectWise Support, who basically said, “Have you tried turning the rule off and back on again?” Off, yes. Back on again, hell no. (Besides… by this point the last status change will meet the third requirement of the rule! Not much good this is going to do anyone, now is it?)

So. The rules are disabled, and the cleanup begins.

That number went UP (to just shy of 3k) between the time I started writing this post and got most of the way finished with it. And yes, I’m making sure there’s a re-closing batch running at all times. I’m a professional, dammit.

I have, as of this writing (2:30pm or so), been playing whack-a-mole with re-closing tickets since about quarter past 11. The problem… okay, an additional problem… is that the database query which reopened all those tickets? It’s still running. And it’s faster than I am. It started at Ticket #1 and is working forward in time from there, and I’m chugging along behind it, using the clunky GUI to say “hey, no, re-close those all please” and waiting for it to process each batch.

(Before you ask, tech nerds: No, we don’t have direct access to the database. Believe me, I wish we did right now.)

Oh did I mention that there are actually two of these workflow rules that kicked off at the same time? Right. (The other one is for missing Type & Subtype values. Trust me, you don’t want to know. It’s boring tech support stuff.)

I had other plans for today’s workday, but this? This is my life now. And for the foreseeable future. What fun.