seperis: (Default)
seperis ([personal profile] seperis) wrote2019-02-19 05:45 am
Entry tags:

so this is all mostly theory, which is the entire problem

In honor of only two days of work this week as Holiday!Monday and Escapade Starts Thursday, I've been contemplating the more esoteric parts of my career, or more specifically, my least favorite part.

For those who don't know, I'm a Quality Control/Quality Analyst; my formal title is System Analyst IV, my job description is program testing, primarily, UAT, aka 'User Acceptance Testing' but have done and will do everything from unit testing to testing live in production literally during and after deployment. UAT is the last line of defense before a program is released in production, and our job is to break it with only the tools and general knowledge available to the average user of this program aka Everyman.

And when I say 'the program', that refers not just to 'one single program' but 'an entire program ecosystem that all work together to do shit'. We call the latter Integration Testing, which combines 'so breathtakingly boring even death avoids you when you have to do it' with 'astronomically high stakes'. For System Integration is literally repeating all your tests on those same damn programs (sometimes you're on your fifth repetition and resent key parts of the alphabet) but now while all programs are connected to each other.

In general, if there are problems, they're tiny; earlier testing of the individual parts of a program and then the program itself should have and generally does catch everything with a realistic chance of happening, quite a bit that realistically won't but possibly could, and some that is technically impossible but when you were on repetition three of the same set of ten to twenty goddamn tests, dev was naturally the target best suited to share your suffering. At that point, they were so goddamn tired of seeing your name on defects they didn't care if it was possible this situation would ever occur, they'd code as if it would happen every day just to avoid how rejection at end of business day inevitably meant that the first thing they'd see in their inbox the next morning would be a gratingly cheerful email that included an essay (and references) on why the defect was not only very possible but could cause the apocalypse if not fixed like right now please, sometimes with malice aforethought in thirteen point Comic Sans.

But I digress.



Let me go back a bit: user testing is basically 'use program like a user would', which on the surface seems less 'a job' than 'doing what everyone does on a computer every day' but salaried with health insurance, a generous leave policy, and a pension on retirement along with paid health insurance until I die. Yeah, I'm totally living the dream here, and also, no.

What you don't know is the average user using a program is an idiot savant with an IQ ranging from 'potato' to 'requires the use of exponents to represent accurately' at the exact same time. Keeping people from deliberately fucking with a program is the testing equivalent of taking crayons from toddlers, comparatively speaking; there are some fairly strict limits for a hacker to work with and very little room for error, and that's before security specialists exercise their professional paranoia on it by assuming any attempt to hack is actually literally going to kill them dead in their beds if they don't stop it and that's not entirely hyperbole.

The average user doesn't recognize limits as a thing or if they notice them, think don't really care or, on occasion, are insulted by them for reasons vague. In horrific wtf example, you can set a text box to integers only for phone number and the end user will somehow paste in their email address and crash oracle. How the fuck is that even possible, we tested for pasting. We didn't test for entering your phone number, saving the page, hitting 'back' to reopen the saved page and for reasons very likely not even malicious control-v'ed their email in there and hit save again.

The answer of 'how' is actually not 'witchcraft' (or at least not only) but does involve 'cache', which is not dissimilar, but yeah, that's one example of how users are the nightmare offspring of chaos theory and honey badgers with fingers that may or may not be individually possessed when they touch a keyboard, you cannot predict this shit.

Now, with that context of user ability to crash oracle with back and control-v, Integration Testing.

As stated, this is not testing one program, but it's also not just testing three and how they interact or even sometimes all the programs. This involves bringing up the entire system with all functionality working and extremely accurate mockups of every single possible program/system that it interacts with or gets data from that we do not own, and not a few belonging to the Federal government. As in, these mockups aren't just supergood at pretending to be terrifyingly secure federal systems that have a start value of twenty years in prison for fucking up, some could pass for copies because fear is really motivating and no one wants to wake up to being responsible for accidentally crashing the United States with a random control-v.

And sometimes--much worse--we get access to the live system when data is read only, and now we get to where 'more boring than anything ever' would be a relief.

Specifically, on those thankfully rare days that Data Broker is involved, because as we all know, the one thing all of us lack in our lives, work or not, is active terror. Data Broker, for those playing the home game, is a database (possibly a literal 'base' I really don't know) where something not unlike your entire life history from SSN/DOB/marriages/children/job history to every address you have ever had and every neighbor you have had at every address and that's just to start. It's super secure, access is incredibly limited, and even looking at the button on the screen too hard creates a log. It can only be used to look up info absolutely required and looking up anyone for any reason outside the specific person you're supposed to be looking up is a firing offense plus felony and the attention of possibly multiple Federal agencies and more acronyms that anyone sane should deal with sober or even drunk. And for the most part, because it's possible your blood pressure isn't critical quite yet nad that must be fixed, 'accident' is not something that is recognized as separate from 'deliberate attempt to identity steal/stalk/blackmail', which is equal to 'deliberately breaking into a federal system'.

Now to be fair, they're not entirely wrong; it is, actually, incredibly hard to do anything accidentally and to get any (criminally useful) data, intent is required. The average authorized user just doing their job has nothign to worry about, and I say that as a former authorized user who accessed it several times a day in the course of my job. In an honestly surprising display of logic, they made it super easy to be ethical even before you had coffee and even should you abruptly become catatonic. Doing evil took effort, and for that matter, more effort than any state employee would bother expending without a consummate increase in salary with or without a higher classification and a blood contract stating two days we could work from home into perpetuity.

(Before anyone even thinks anything close to 'totally overpaid' I am for all intents and purposes an expert, I am legitimately good at my job, and I can and do act as lead and quasi supervisor and check the work of private contractors hired by the state whose salaries at start value are twice mine. If the next statement is 'you could work for them, then', that's possible but problem: they will hire me for the same exact job I am doing now--including leading--but my pay will be equal to or less than now, almost no PTO, higher medical premiums, and no pension, because I don't have a four year degree. Yeah no.)

The problem is I am not the average authorized user accessing the system in production and only know it works for reasons that may or may not involve wizards and elves or not, they never even wonder because, much like gravity, it never doesn't work. I am the one testing brand new code for production that is sometimes interacting with other programs for the first time ever and some of those programs possess code that is also brand new, and combining that with database access where accidents are synonymous with felonies. My job is to not just make sure the code works, but to the average authorized user it is mentally classified as gravity; it works, don't care.

In a very real and not at all logical sense, my job is to do my best to get an accidental felony so the end user doesn't. If it is possible to do it accidentally, I am ethically and morally obligated to mkae it happen. You might even say that the day I am convicted of [you have no idea how many felonies this shit falls under] and sent to a minimum security federal prison, I will have reached the epitome of my career and set the bar so high when it comes to the ethical standards of my profession no one will be able to match, much less surpass me, without the addition of lethal injection and/or quite a bit of electricity. The rush for that hypothetical person would be goddamn amazing which hopefully isn't lost under the entire 'impending death' thing and also I'd be pissed because how the fuck do I top that? Asshole.

On the scale of drama from 'paint drying' to 'lady gaga wearing a meat suit to a PETA meeting with a furry paleo bodyguard' how much am I milking this if not outright lying? Not lying but definitely much higher than paint, but having said that, the principle of uncertainty assures it is not actually ever out of the realm of possibility when it comes to legality. The penalties are explained to authorized users in enough detail to assure PTSD is not out of the realm of possibility, but user testing, funnily enough, was not in mind (or technically existed as such or at least in this form) when the legalities were hashed out and as of yet, no one is really feeling this should change. Technically, a user tester could simultaneously be both an authorized user with all privileges and responsibilities inherent and hackers doing evil, who can really say but a Federal judge, which sure, does have the advantage of some legally defined clarity but also, you know, prison. So uncertainty, not all that bad, really.

And to add to this; Data Broker isn't the only program with some grey area, just the most serious. There are many that have very strict legally-binding access rules that lack clear guidelines on the status of testers, and some with read-write access.

The fear is real, is what I'm saying, and sure, somewhat theoretical, and taken individually, not much. When combined, however, that is a lot of goddamn theory to have just hanging around. Pure chance says one of these totally theoretical concerns will be tested, and while it's unlikely it will be me, 'chance' is now getting a little too big an area as well. Yes, the odds are good I am wasting very valuable worrying time on this which could have been spent on increasing the amount devoted to alien abduction and my rabbits eating me if I fall down in the cage and knock myself out (rabbits? NOT ACTUALLY OBLIGATE VEGAN AND HAVE EATEN MY LASAGNA), but would it be such an imposition to de-grey some areas without duress being involved? Get that out of the way and I could get some serious worry-traction started on the likelihood of Alexa telling me I really want to buy Frye lace-up boots that are on sale to wear for my murder spree while I'm asleep and worse, that I'd wear new leather boots on a murder spree and risk bloodstains on cognac leather. That shit would never come out.



...yes, I am doing integration testing this week. How'd you guess?
wychwood: Lt Welsh facepalming (due South - Welsh facepalm)

[personal profile] wychwood 2019-02-19 11:57 am (UTC)(link)
See, I would laugh harder at this if I weren't currently engaged in UAT on a system provided by a supplier who has Serious Form for sending us broken shit

(one of the bits of feedback we got back from them was "we had a lot less trouble with error reports after we worked through the test scripts you sent us!" like, ok, that's nice, but why does it take your customer to put together a basic set of tests to make sure that you didn't irretrievably break core functionality before releasing new versions???)
wychwood: chess queen against a runestone (Default)

[personal profile] wychwood 2019-02-19 06:57 pm (UTC)(link)
I mean, they're using the scripts now? And the more recent versions have been substantially less broken during my testing... that's good, right... :D

(they're very nice people, just.)
alexseanchai: Katsuki Yuuri wearing a blue jacket and his glasses and holding a poodle, in front of the asexual pride flag with a rainbow heart inset. (Default)

[personal profile] alexseanchai 2019-02-19 12:02 pm (UTC)(link)
ooooh that sounds like so much, uh, fun
j00j: rainbow over east berlin plattenbau apartments (Default)

[personal profile] j00j 2019-02-19 12:22 pm (UTC)(link)
I'm an IT project manager who did more testing in a previous life and have seen what testing for electronic medical record systems looks like so... yeah. I feel your pain.
The logging on medical records is intense but the penalties less so. You'll just get fired for looking up people that are none of your business, probably.
out_there: B-Day Present '05 (Default)

[personal profile] out_there 2019-02-19 12:48 pm (UTC)(link)
From my brief brushes with UAT from changing systems / installing upgrades, you have my deepest sympathies. It's tedious and important and so very annoying.
brainwane: My smiling face, including a small gold bindi (Default)

[personal profile] brainwane 2019-02-19 01:01 pm (UTC)(link)
Where is the lie, as the saying goes.

By the way, may I publicly link to this and to https://seperis.dreamwidth.org/1019270.html ? I am writing a post about missing stories in fiction about programming (because of the art festival I'm running).
brainwane: My smiling face, including a small gold bindi (Default)

[personal profile] brainwane 2019-02-25 04:59 pm (UTC)(link)
Thank you! I added it to a list in this post. And you're welcome to propose a piece of art for the festival!
archaeologist_d: (Default)

[personal profile] archaeologist_d 2019-02-19 02:57 pm (UTC)(link)
Sounds hard. I would be crossing my eyes at all that but then a lot of people found chemistry confusing/boring/never again and I liked it.
Have fun with the integration testing.
azurelunatic: Teddybear that contains ethernet switch.  (teddyborg)

[personal profile] azurelunatic 2019-02-19 03:06 pm (UTC)(link)
You are the hero who keeps me from terrorizing Tier 2 tech support by going "did you know that when you enter an ampersand twice in the domain purchase flow it puts 'amp' into the name you were trying to buy; yes it is user error but also we should not have let the escaping escape" (the user got their refund) and harassing Tier 1 with "So did you know that my manager keeps filing tickets that escape all the queues and lie on the floor for weeks before I follow up on them, thereby violating all sorts of SLAs? No, it isn't actually a user education moment, thanks. I'm on a first-name basis with the fixing-this-shit manager."

Tedium plus terror: what a fantastic combination.
azurelunatic: Vivid pink Alaskan wild rose. (Default)

[personal profile] azurelunatic 2019-02-20 06:20 pm (UTC)(link)
...That sounds incredibly epic and I am so glad I am nowhere near it.
thornsilver: (candles)

[personal profile] thornsilver 2019-02-19 03:08 pm (UTC)(link)
So the only thing I got out of this, is that we are one extremely stressed out employee from computermageddon every day of our lives.
azurelunatic: Vivid pink Alaskan wild rose. (Default)

[personal profile] azurelunatic 2019-02-20 01:38 am (UTC)(link)
At my daughter's workplace, all of their code has to hit the emergency brakes in case "the bucket" overflows. Some engineer in another group was being incredibly dismissive of this. My daughter's team lead casually asked whether he was concerned about molten aluminum escaping the machine.

"Surely you have some method to prevent that!" he blustered.

"That's why we have a bucket."
alexseanchai: Katsuki Yuuri wearing a blue jacket and his glasses and holding a poodle, in front of the asexual pride flag with a rainbow heart inset. (Default)

[personal profile] alexseanchai 2019-02-20 01:43 am (UTC)(link)
lollll
olanthanide: (Default)

[personal profile] olanthanide 2019-02-19 03:49 pm (UTC)(link)
On that note, I must say you're the only programmer I know who seems to embrace technology, while everyone else seems to be "no oh god NO don't use paypal just send me a check, disable wifi on EVERYTHING".
kore: (Default)

[personal profile] kore 2019-02-19 07:38 pm (UTC)(link)
I love this comment so much I want to eat it.
anoyo: Made for me! Amy leaning against Spartan and smiling. (Default)

[personal profile] anoyo 2019-02-20 09:47 pm (UTC)(link)
+1
mishaday: (Default)

[personal profile] mishaday 2019-02-19 10:43 pm (UTC)(link)
As someone who has had to investigate people's browsing history at work, this kind of zany search history would be a fucking DELIGHT. It would fucking make my day and be the source of the best kind of 'once a user did THIS' stories that I always have to carefully anonymize and bland up for public consumption.
olanthanide: (Default)

[personal profile] olanthanide 2019-02-21 03:19 am (UTC)(link)
To be fair, I think some people have more specific concerns in our current post 2k16 hellscape -- for instance, if your entire ethnicity has been declared effectively an enemy of the state by a politician, and you're not sure whether or not you're already on some kind of government watch list, do you really want to give them more cameras, metaphorical or otherwise?

My own (somewhat depressing) take is that if they really want to send us to camps (as that man the bus suggested; I kid you not -- California the liberal state still includes the experience of sharing public transit with people will say out loud that all Chinese and Mexicans should be sent to camps), having a squeaky clean record under voluntary surveillance is STILL not going to help.
olanthanide: (Default)

[personal profile] olanthanide 2019-02-22 03:09 am (UTC)(link)
Alas, I just moved out of LA last year for job reasons(and will probably move again within the next six month because research funding is like that). Have a boba in my honor if you feel so inclined, and if you're into sweet things LA has some of the best Chinese bakeries outside of bay area. (Though maybe decide on a budget before you go in. I've had friends who ended up buying WAY too much because the smell of freshly baked goods is like -- someone could probably do a whole neurosci dissertation on the effect.)
azurelunatic: "Offices are why big people get GRUMPY and say BAD WORDS" (offices are why)

[personal profile] azurelunatic 2019-02-20 06:34 pm (UTC)(link)
My household's day-to-day practical worries about Smart Everything are approximately:

Is the Smart Refrigerator going to get compromised and start sending spam? (we don't have a smart refrigerator, partly because expensive and the current one is fine, partly because this)

Is the Smart Lightbulb going to DoS the network? (Some unfortunate person traced the flood on their home network to a rogue lightbulb, as linked by the Internet of Shit Twitter account)

What happens when the Smart Lock manufacturer goes out of business? What happens when the internet goes out during the ice storm?


At my old apartment I had a beautiful suite of moderately smart switches controlling lights, but it's somewhat harder to orchestrate the light schedules for 3 people than for 1. Plus the wiring in this house is faintly terrifying.
grammarwoman: (Default)

[personal profile] grammarwoman 2019-02-19 06:02 pm (UTC)(link)
You make me very sad that I'm not attending Escapade so that we could meet up and buy each other drinks whilst exchanging horror stories about DEV, QA, HIPAA, and testing systems. (Starting with baby programmers who only code for happy paths and don't put a thought into basic error handling. ARGH.)
kore: (Default)

[personal profile] kore 2019-02-19 07:41 pm (UTC)(link)
In horrific wtf example, you can set a text box to integers only for phone number and the end user will somehow paste in their email address and crash oracle. How the fuck is that even possible, we tested for pasting. We didn't test for entering your phone number, saving the page, hitting 'back' to reopen the saved page and for reasons very likely not even malicious control-v'ed their email in there and hit save again.

ohhh my God

(this whole post is gold especially the parenthetical about the non-vegan rabbits)
jenna_thorn: hazard warning sign of falling, with jazz hands replacing text. (jazzhands)

[personal profile] jenna_thorn 2019-02-19 08:37 pm (UTC)(link)
I volunteer as a user tester for UAT whenever it comes up, which confused my team lead no end, because I have publicly wished for monks in scriptoriums and messenger pigeons, usually when fighting with the secured safe. But my thought is that if I can't break it in testing, it won't fail in use.

Note: to date,I have fallen into full stop issues on every upgrade I've tested. The higher ups hate me, but the people actually doing the work now request me by name.
niciasus: (Default)

[personal profile] niciasus 2019-02-20 03:52 pm (UTC)(link)
This entire post sent me back 19 years in which I managed a computerized accounting system with $2.0 billions in assets. I spent years testing, coding, writing manuals, sitting on project teams as the Corporate company I worked for switched from mainframe (don’t laugh) to enterprise systems such as SAP and Oracle, or specialized systems I was introduced to at a non-profit company. I feel you.

I have been on a binge reading fan fic. Read “Three Impossible Things”, three times, Lol. I’m looking to read more of your ClarkLex fic. Do I miss those days.
complicat: (Clark_OMGNo: fandom_girl)

[personal profile] complicat 2019-02-20 05:36 pm (UTC)(link)
I'm a developer and when we're in the final stages of our periodic scrambles to meet our UAT release deadline and get Safety to sign the damn thing off, each day I'm holding out for around 2.30 pm when I know that the offshore testing team will have knocked off for the night and I can relax and get some peace to work through the fixes without another "Well, if you follow this sequence of events which no sane person could have predicted then the whole thing gets screwed up, pls fix asap" defect arriving in my inbox. But then of course the offshore team get a 3 hour headstart on me in the mornings, so I log in to a stack of "Yep that one is fixed but then there's this mad scenario..."
jesse_the_k: text: Be kinder than need be: everyone is fighting some kind of battle (alanna is amazed)

Thanks for this

[personal profile] jesse_the_k 2019-02-21 12:49 am (UTC)(link)
epic and delightful screed.