Programmer?

[anivair.livejournal.com]

So a Programmer (A PROGRAMMER!) from a regional hospital which shall remain nameless just sent me the following:

"It seems like we always have issues processing the return file from you every month. Today's issue is that the file is pipe (|) delimited instead of comma delimited. That is a major change to the file.

I cannot process the file in this format. Please fix the file, resubmit to the ftp site, and notify me when this is resolved."

okay, problem 1. I know what a pipe looks like. Problem 2. Can you not replace the pipe's with commas? Also, the file is fixed format. They shouldn't be using delimiters to begin with.

but then she sends me this jewel:

"will you please send me all your information on ftp connection to your site. Maybe the issue is with the connection."

Really? Out ftp connection might have snuck in and replaced most of the commas with pipes, but left all the other text and formatting intact? You're fired.

Wow. just wow.

Comments

[jimbojones.livejournal.com]

Just reply sed s/\|/\,/g file.txt | file.new.txt - with no other text whatsoever.

Fuck 'em. Here's a nickel, kid, get yourself a better computer.

[samldanach.livejournal.com]

As a programmer, permit me to translate:

"I'm a highly trained computer professional! Why do I have to do this monkey work every month? Now the file has pipes in it! Don't you know that will consume almost fifteen minutes of my precious time to fix? As a peon, I'm sure that you have no concept what fifteen minutes of my time is worth.

"I refuse to engage in this endeavor any further. Please do my job, and let me know when I can take credit for it."

[egearman.livejournal.com]

Newbie or idiot?

Return file every month, but today's issue???

As far as a fixed file format goes, sounds like someone decided to rewrite their import without telling the folks that write the export.

Where is that link to order a BFCB?

[snarl817.livejournal.com]

I'm going to play Devil's Advocate here, so bear with me....

Her issue isn't, "I'm too stupid to use sed, vi, or even notepad to perform a simple search and replace." It's, "The format for this production data is incorrect. I cannot modify it because it's not my data. The data came from you, so YOU need to fix it."

It's all about production controls. I NEVER modify production data that I receive for processing, even when it would take me 2 seconds to fix. The reason is two-fold: 1 - IT'S NOT MY DATA TO MODIFY. 2 - If I modify it so that it'll load and there's a data issue (ie., incorrect values), then I can be blamed because I made an unauthorized change to production data. I contact the data provider and tell them that there's a problem with their file and make them fix it...forcing THEM to be accountable for the validity of their data.

[egearman.livejournal.com]

But there is the problem that a comma could be valid data. That's one of the main reasons I've seen for a fixed format on imports.

And they could probably use the nickel to get a better programmer than pipe-boy.

Anivair, sorry you have to deal with this type of programmer.

[egearman.livejournal.com]

Good point. Hadn't considered that for my above posts, since I am a developer and rarely if ever get to play with/see production data.

[snarl817.livejournal.com]

I'm a trained monkeybatch scheduling operator. I deal with this shit on a daily basis. When my processes started spitting out files in the wrong format and downstream recipients complained, I manually fixed it (because it was my data), then beat the developers about the head and face with a newspaper until they fixed the code. :D

[ptomblin-lj.livejournal.com]

*THIS*

I was going to say almost exactly the same thing.

I run a web site where I take data from a bunch of different sources, and do things to it to make it available for others. One of the providers of that data kept changing his format every month. Some months it was CSV, sometimes it was tab delimited, sometimes it was fixed format, and the "fixed" format changed without warning. Every month I had to figure out what they'd done differently, and change my import program to accommodate it. When I finally wrote to them to ask them to provide the data in the same format every month, they stormed off in a snit and stopped providing the data.

[jimbojones.livejournal.com]

I think you're missing the whole point of "if you insist on comma-delimited, then fine, change the delimiter yourself." If the OP were being paid on spec to deliver finicky arbitrary formats and sanitize data conversions, I don't think he would have posted this rant in the first place, he would have just put on his Helpful Hat and done the job.

* edit: at least, here's hoping the OP wouldn't be griping about this if it was actually his job to provide the data in a particular format.

[jimbojones.livejournal.com]

::sniff:: ::sniff:: that smells suspiciously like real estate data aggregation... =)

[egearman.livejournal.com]

Wow... sounds like a good customer/client/partner/whatever to not have anymore. I prefer tab delimited for the obvious reasons that you can typically tell the difference between data fields and delimiters (say, vs comma, pipe or *shudder* space separated fields).

[egearman.livejournal.com]

Oh, I agree with you. I think it could also be an issue with the people doing the extract not talking with the people doing the import.

Of course, the "we always have issues processing the return file from you every month" part makes me ask "and why is this only coming up NOW?"

[superbus.livejournal.com]

Not even fifteen minutes! Shit, you could find every | and replace them with , in VI, let alone Word or Write!

[tregare.livejournal.com]

-snicker- that SO sounds like the attitudes of some of the programmers I have worked with (and PM's)

[digitalraven]

*THIS*

I used to program for a bank. There was no way in high holy fuckery that I was touching the data to make it fit the program once the program was written. Data provider makes a change? They can damn well spec it out and give us a chance to fix it and get the fixes through the harsh QA processes. Didn't matter how trivial the change. Always get someone else to sign off changes before they're made, so people know who to sue when it goes wrong.

[ptomblin-lj.livejournal.com]

navaid.com - aviation data. Much of which used to be available courtesy of the National GeoSpatial Intelligence Agency until Australia started asserting copyright over their own airspace data.

[kizayaen.livejournal.com]

Seconded. I do a lot of work importing insurance eligibility files from our client companies and generating database exports for their carriers, so this is an issue I'm well familiar with.

... having said that, the programmer is still a flaming idiot for suspecting it might be an FTP issue.

[jimbojones.livejournal.com]

Different markets, largely the same kinds of data. A fart rose by any other name... ^_^

[anivair.livejournal.com]

Nope. And honestly, they demanded commas and nothing else, as if that won't screw their format up when commas are in the fields.

this is also the third of fourth time that they're changed their format requests on us and each time they do we send them what they asked for and they get mad because it changed. I just don't get it.

[anivair.livejournal.com]

that would be an excellent point if they had not asked us to make the change in the first place (I was mostly posting about her lack of understanding of the ftp, so I left that out, but this is constant there). Honestly, if they were smarter, I'd be fine dealing with them. I don't mind doing work for people that know their jobs.

[mattcaron.livejournal.com]

See, the way I would reply is:

"I'm a highly trained computer professional! Why do I have to do this monkey work every month? Now the file has pipes in it! Don't you know that will consume almost fifteen minutes of my precious time to fix? Therefore, I will write a very small shell script which I can just run on the file and I will go read techsupport until it is finished."

[mattcaron.livejournal.com]

So, you don't right input filters at all?

Intewwesting.

I'm used to having to shoehorn crap in with perl duct tape.

[ellyfialy.livejournal.com]

What about spontaneously trying to change the way they send you the files? "I know we were using SFTP until yesterday, but now we're going to send them to you by email."

Thanks a bunch. My automation loves you. Yeah...

[jjjiii]

Mabye the file transfers are taking place over some mutant charset that is exactly like ASCII except the hex codes for | and , have been transposed? That's what I'd think if I were a dumbshit programmer:)

Check your server, there's probably a window somewhere that looks like this:

+=FTPd Config===========================_#X
|                                         ^
|  Transfer Type:                         |
|  [ ] Binary                             |
|  [ ] ASCII                              |
|  [x] ASCII, except commas are pipes     |
|                                         |
|                                         |
|  [OK] [Cancel]                          |
|                                         v
+-----------------------------------------+