Dear Users... (A thread for Sysadmin, Technical Support, and Help Desk people)

[KAJ]

In which case they can do that on their side. I'm providing raw data, not doing analysis for them. I'm putting properly numeric data into that field, and dumping the rest into a "miscellaneous" field. If they want to parse through all the possible combinations of garbage text users enter in there to figure out if they can math something, they'll have to do it themselves.

And I think you missed where the entry wasn't ">12" it was the words "greater than" 12.

Calm down dear! I was just pointing out that the response "That's numeric!" is correct in some circumstances. In those circumstances the value ">12" is "properly numeric". The value is distinct from its representation so "greater than 12" is the same value as ">12" or "greater than twelve" or "more than a dozen". Of course, the implementation will likely restrict acceptable representations.

If censored data is at all frequent in the application domain, then perhaps the user requirement should have included it, and the implementation might then have allowed (and validated) such entries in a numeric field - I have implemented such data entry, it isn't very difficult.

tl;dr Looks like inadequate specification (communication?) rather than stupid user or bad programmer.

 

[xterra]

If a spreadsheet is adding numbers:


34
45
12

How is it going to add


"greater than 12" when the words are written in letters?

 

[novaphile]

If a spreadsheet is adding numbers:


34
45
12

How is it going to add


"greater than 12" when the words are written in letters?

@val(string)

 

[alfaniner]

If a spreadsheet is adding numbers:


34
45
12

How is it going to add


"greater than 12" when the words are written in letters?

How could it add ">12" even when parsed as numeric? It's not a number, it's a comparison.

Perhaps instead of "add" the word was intended to be "process" or some such.

 

[TragicMonkey]

In any event, in this particular context the values don't even matter: the point is that there is something there at all. It's a data set of medical lab results, and the field/s in question are the particular results of each lab. What matters is that there is a result, because that's how this system differentiates between a lab that was complete ("resulted" is the term used) and a lab that was ordered and not yet done. So the client wanting the result values separated out by datatype, even if I could do that neatly, is pointless for this endeavor. If they got a "yes/no" on results would work just as well, but there's no point in not sending them the actual results if we got 'em. On their side all they need to do their processing is determine if the field/s are null or not.

eta: and now that I've thought about it more, it strikes me that they might be trying to use this data for purposes other than what it's intended for. I may just have a word with my higher ups and see if they think we need to shut that down and send only the absolute bare minimum to achieve the task in hand. There's contract obligations and money involved, so it could really go either way--they might be sneakily trying to get extra stuff from us beyond what's been agreed, when they ought to be paying more.

 

[arthwollipot]

Are you crazy? A client would never do something like that!

 

[TragicMonkey]

Are you crazy? A client would never do something like that!

I slept on it, and on waking realized they actually already have access to the lab values from another source. The "specs" for this project (which while fulfilling an industry standard is new to this particular client) were probably just copied from another extract process, and they weren't familiar enough with the data to realize they're asking for unnecessary things and redundant fields. In any case, they are entitled to the information...exactly as it appears in the database on our side--I needn't do any cleanup or alteration. In fact, I shouldn't--my obligation is to ensure the data I send accurately reflects what's stored, whether it's correct in itself or not. (That's not just a CYA thing when it comes to medical record data, it's a legit standard because as a nonclinician I'm not qualified to change data, and even if I were it would have to be done at the source in the records themselves.) The most I can do is reformat dates, etc. So I'm sending what we have, divided as they requested into three fields: numeric, positive/negative strings, and everything else.

 

[KAJ]

If a spreadsheet is adding numbers:


34
45
12

How is it going to add


"greater than 12" when the words are written in letters?

Most spreadsheets cannot handle censored data. Some (e.g. based on R) can.

As I said earlier, the value and its representation are distinct, so whether or not it is written in letters is by the way. Incidentally, when I type "twelve + four" into Google's search box the first result is 16.

sum (34, 45, 12, >12) = >103; censored arguments, censored result.

 

[KAJ]

How could it add ">12" even when parsed as numeric? It's not a number, it's a comparison.

Perhaps instead of "add" the word was intended to be "process" or some such.

That depends on the parser, which in turn depends on the syntax of the language being parsed. If the system is to handle censored data then clearly the syntax must distinguish between censoring indicators and relational operators.

 

[KAJ]

I slept on it, and on waking realized they actually already have access to the lab values from another source. The "specs" for this project (which while fulfilling an industry standard is new to this particular client) were probably just copied from another extract process, and they weren't familiar enough with the data to realize they're asking for unnecessary things and redundant fields. In any case, they are entitled to the information...exactly as it appears in the database on our side--I needn't do any cleanup or alteration. In fact, I shouldn't--my obligation is to ensure the data I send accurately reflects what's stored, whether it's correct in itself or not. (That's not just a CYA thing when it comes to medical record data, it's a legit standard because as a nonclinician I'm not qualified to change data, and even if I were it would have to be done at the source in the records themselves.) The most I can do is reformat dates, etc. So I'm sending what we have, divided as they requested into three fields: numeric, positive/negative strings, and everything else.

Emphasis added. Point taken. You have met the spec.

 

[malbui]

Testing stuff properly seems to be anathema to almost the whole industry

I have major philosophical differences with the rest of my organisation in respect of testing. Both IT and user teams believe it's a question of seeing what happens if you input a few sample transactions. I actively try to break the software.

But Malbui, they say to me, nobody would ever submit a transaction or a batch with a control character in a numeric field. I laugh bitterly.

 

[quadraginta]

I have major philosophical differences with the rest of my organisation in respect of testing. Both IT and user teams believe it's a question of seeing what happens if you input a few sample transactions. I actively try to break the software.

But Malbui, they say to me, nobody would ever submit a transaction or a batch with a control character in a numeric field. I laugh bitterly.

And you referred them to this xkcd cartoon, of course.

 

[Blue Mountain]

I have major philosophical differences with the rest of my organisation in respect of testing. Both IT and user teams believe it's a question of seeing what happens if you input a few sample transactions. I actively try to break the software.

But Malbui, they say to me, nobody would ever submit a transaction or a batch with a control character in a numeric field. I laugh bitterly.

Yes! Unfortunately developers tend to coddle their software. "I know that doing x, y, and z together will break the program, so I won't do it." I can get away with that thinking if I'm writing a program for my own use, but not if anyone else can supply it with input. A good tester should try to break the software.

In some programs, 80% of the code is simply making sure the inputs are good before doing anything with them.

"If builders built buildings the same way programmers wrote prrograms, the first woodpecker to come along would destroy civilization."

 

[TragicMonkey]

I have major philosophical differences with the rest of my organisation in respect of testing. Both IT and user teams believe it's a question of seeing what happens if you input a few sample transactions. I actively try to break the software.

But Malbui, they say to me, nobody would ever submit a transaction or a batch with a control character in a numeric field. I laugh bitterly.

2015
TM: Shouldn't this huge notes field have some limitations on it, or strip out some unwelcome characters and whitespace before it stores it in the reporting database?
Expert: That's not necessary.

2016
Expert: Why are people putting pipes and so many tabs and enters in the notes field?! It's messing up all the exports!!!
TM: What a pity nobody thought of-- oh, wait, here's the email chain where I did, and you said it wasn't necessary. I saved it. I save all the emails.

Turns out a lot of people actually do use pipes when typing out large screeds of text, they think it looks neat.

 

[Wudang]

I have major philosophical differences with the rest of my organisation in respect of testing. Both IT and user teams believe it's a question of seeing what happens if you input a few sample transactions. I actively try to break the software.

But Malbui, they say to me, nobody would ever submit a transaction or a batch with a control character in a numeric field. I laugh bitterly.

They're amatuers. I wouldn't give them a job. Seriously.

 

[CORed]

In any event, in this particular context the values don't even matter: the point is that there is something there at all. It's a data set of medical lab results, and the field/s in question are the particular results of each lab. What matters is that there is a result, because that's how this system differentiates between a lab that was complete ("resulted" is the term used) and a lab that was ordered and not yet done. So the client wanting the result values separated out by datatype, even if I could do that neatly, is pointless for this endeavor. If they got a "yes/no" on results would work just as well, but there's no point in not sending them the actual results if we got 'em. On their side all they need to do their processing is determine if the field/s are null or not.

eta: and now that I've thought about it more, it strikes me that they might be trying to use this data for purposes other than what it's intended for. I may just have a word with my higher ups and see if they think we need to shut that down and send only the absolute bare minimum to achieve the task in hand. There's contract obligations and money involved, so it could really go either way--they might be sneakily trying to get extra stuff from us beyond what's been agreed, when they ought to be paying more.

It seems to me that you (or your employer) need better input validation on whatever they are using to enter the data. I realize that may not be your responsibility. IME, users will put any damn thing you let them into data fields, in any damn format they can dream up, and then expect your software to figure out what they really meant and process it appropriately, even though in all likelihood, they don't remember what they meant when they entered it.

 

[Wudang]

My job is writing software that merges medical data from various formats into a common format. link. Without naming names it's horrific the data we see in fields that are supposed to be one thing and turn out to be another.

 

[Darat]

Yes! Unfortunately developers tend to coddle their software. "I know that doing x, y, and z together will break the program, so I won't do it." I can get away with that thinking if I'm writing a program for my own use, but not if anyone else can supply it with input. A good tester should try to break the software.

In some programs, 80% of the code is simply making sure the inputs are good before doing anything with them.

"If builders built buildings the same way programmers wrote prrograms, the first woodpecker to come along would destroy civilization."

Sometimes it can too far the other way. I remember arguing with Nintendo about a bug, apparently in one particular vehicle out of about 15 if you ran the car at a wall 32 times at full speed with one particular (time bound) power up you could pass through the wall. This was classed as a critical fail, the game was in the end released with this critical fail but it took about 10 days of argument. The amazing thing is that there were a couple of places in some of the maps that you could if you approached at a particular angle always pass through a wall, they never raised those as a critical fail.

 

[Darat]

2015
TM: Shouldn't this huge notes field have some limitations on it, or strip out some unwelcome characters and whitespace before it stores it in the reporting database?
Expert: That's not necessary.

2016
Expert: Why are people putting pipes and so many tabs and enters in the notes field?! It's messing up all the exports!!!
TM: What a pity nobody thought of-- oh, wait, here's the email chain where I did, and you said it wasn't necessary. I saved it. I save all the emails.

Turns out a lot of people actually do use pipes when typing out large screeds of text, they think it looks neat.

I keep all my emails, and before that all my memos and I always record any decisions with a summary of the decisions and why and most telling who has made the decision. It’s not just a matter of CYA, it is amazing how some people will change their mind about something that is apparently very important (and of course will cost more or take longer) when their name is put against the decision.

 

[Gord_in_Toronto]

Testing stuff properly seems to be anathema to almost the whole industry

Plus, of course, if your project has TEST in the project plan just before GO LIVE, as (when) the project starts missing deadlines the time for testing gets cut shorter and shorter. I've had the misfortune to be Test Manager on some fairly large projects and I don't think I've ever been on one where the PM or senior management said, "Whatever we do we must complete the testing as we planned".